Indexer Running Slow

General Discussions - English
1

I’m trying to build a plain index from a mariadb table using the indexer like so:

docker-compose exec manticore gosu manticore indexer --all --rotate

There are only 100K records in the table that I’m querying the data from and it seems that it’s taking longer than I remember to index the data in manticore.

Manticore 6.0.4 1a3a4ea82@230314
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

using config file '/etc/manticoresearch/manticore.conf'...
indexing table 'all_listings'...
collected 20000 docs, 4.0 MB

I was testing getting the data from mariadb into manticore to rebuild the index from scratch and attach it to an rt index. It’s taking a very long time to run the operation, I’m not sure if there is something i’m missing. I have roughly 28 columns, one of which is the full text field. the rest of the columns are most bigints.

is there a way to go straight to an rt index from the database? my knowledge might be a little out of date. i remember doing this in the past but there might be a newer way of accomplishing what i’m after.

Thank you!

2

Indexing may be slow because the sql_query is not optimal, e.g. you may be not using a key in Mariadb or not using sql_query_range or smth else.

To write from the db to a real-time index you can write a small script which reads from Mariadb in batches and writes to Manticore.