hello. im using a nosql db as my primar db & my data is distributed across multiple nodes/clusters. I cant seem to get manticore to work since all the mysql related info etc. I cant setup manticore & would love to get some help on the same. Ive gone through the docs but my docker gives some or the other error everytime. Kindly help. If I could get a step by step guide on commands to execute to set this up, id be grateful. thankyou
Hello
im using a nosql db as my primar db
What exactly db?
my data is distributed across multiple nodes/clusters
So is the primary nosql db distributed?
since all the mysql related info etc
How is it related with mysql? Is mysql your primary db?
commands to execute to set this up
How are you going to use Manticore: plain indexation, real-time indexation, with replication or not?
I am using cassandra/scylla db.
So yes, the db is distributed
How is it related with mysql? Is mysql your primary db?
No, i meant all the docs about using mysql to interact with manticore using port 9306 & stuff. I know it uses sphinxQL but im getting very confused.
I am going to use manticore to enable some complex full-text search & normal queries like aggregation etc too. Im going for plain indexation with batch data since that provides better throughput compared to real-time indexation. I need this to be as low-latency as possible. The data doesnt have to be searchable in real time, there can be a slight delay since users normally dont search for data generated instantaneously rather historic data. & yes, i do wish to replicate it across 1 node so if the primary node fails I have a fallback for the search logic
the architecture of manticore I need- enables sharding (which will be handled in application side since manticore doesnt have that yet), so I need multiple nodes/clusters with replication to 1 node. I dont want to store all my data in manticore, I want to still keep cassandra as the source of truth, but send data to manticore for indexing to search texts.
I need global indexing to work on the app until a new user joins after which I can use user-specific indexing to provide user-data security via this segregation & speed up load times for searching data by new users
Here’s how you can proceed:
- Use the plain mode and plain tables. Learn about the differences between plain mode and rt mode here: Manticore Search Manual: Read this first
- Enroll in this course to understand how the plain mode, plain tables, and indexer interact with MySQL: Manticore Search - Indexing data from MySQL example
- Enroll in this course to understand how the system operates with CSV: Manticore CSV Import Example
- Read more about fetching from XML streams here: Manticore Search Manual: Data creation and modification > Adding data from external storages > Fetching from XML streams
- Write a script that reads from ScyllaDB and outputs XML in the xmlpipe2 format.
- Compose a Manticore configuration with the correct xmlpipe_command pointing to the script.
- Build the table using
indexer. - Start Manticore.
- Rebuild the table using
indexer --rotate.
That should cover it.
thankyou for the reply, But I cant seem to run the indexer because I cant find the manticore.conf file for some reason. I installed it using docker & there are folders inside docker /bin/bash, but theyre all empty. idk whats going on, this is driving me insane…any ideas?
What does this return?
ls -la /etc/manticoresearch/manticore.conf
With docker for plain tables you need to mount your custom config into the container - GitHub - manticoresoftware/docker: Official docker for Manticore Search
I tried-
root@deceb3fcb2e6:/var/lib/manticore# ls -la /etc/manticoresearch/manticore.conf
& it returned -
-rw-r--r-- 1 manticore manticore 9326 Oct 23 12:14 /etc/manticoresearch/manticore.conf
I cant find the manticore.conf file for some reason
So here you are.
thanks, ill try to update this
ok so I think i can understand the real problem. theres no way for me to open this file. logging in as a root user shows different files, etc. But as a root user I cant use editors or nano or vi to edit files. & as a normal user using sudo, this file doesnt exist. Any ideas?
How is it possible? Is that in Manticore’s official docker? How do I reproduce it then?
But as a root user I cant use editors or nano or vi to edit files
Why don’t you install vim or nano? With the official docker image you can do apt update and then install whatever you want.
vim & nano come preinstalled in my os. i also have helix. none of the commands work as root user
How is it possible? Is that in Manticore’s official docker? How do I reproduce it then?
This is the command I ran initially (didnt work)-
sudo docker run --name manticore -v ~/manticore/etc/:/etc/sphinxsearch/ -v ~/manticore/data/:/var/lib/manticore/data -v ~/manticore/logs/:/var/lib/manticore/log -p 9306:9306 -p 9312:9312 -d manticoresearch/manticore
My most recent attempt-
sudo docker run --name manticore -p 9308:9308 -p 9312:9312 -d manticoresearch/manticore
followed by-
sudo docker exec -it manticore indexer --all --rotate
gives:
Manticore 6.2.12 dc5144d35@230822
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)
using config file ‘/etc/manticoresearch/manticore.conf’…
FATAL: no tables found in config file ‘/etc/manticoresearch/manticore.conf’
i entered as root user - sudo docker exec -it manticore /bin/bash
root@deceb3fcb2e6:/var/lib/manticore# ls -la /etc/manticoresearch/manticore.conf
-rw-r–r-- 1 manticore manticore 9326 Oct 23 12:14 /etc/manticoresearch/manticore.conf
But I cant open this. Ive tried everything. Its very weird
lemme know if a screenshot of my terminal helps. I can send it
-v ~/manticore/etc/:/etc/sphinxsearch/
is wrong. There’s no /etc/sphinxsearch/ in the container. Manticore also doesn’t use it as a default dir even if it gets created by docker when you map the dir inside the container.
The proper config path is /etc/manticoresearch/manticore.conf . Please read GitHub - manticoresoftware/docker: Official docker for Manticore Search
sudo docker exec -it manticore indexer --all --rotate
Make sure to index under user “manticore”. Use gosu, the example is here GitHub - manticoresoftware/docker: Official docker for Manticore Search
yeah that was 1 of my many failed attempts. my most recent one is-
sudo docker run --name manticore -p 9308:9308 -p 9312:9312 -d manticoresearch/manticore
ill check out the links some more, & get back
thanks
OK. With this you get the default config which is not compatible with indexer as it’s running in RT mode. You can read about the modes here Manticore Search Manual: Read this first
sure. thanks.