Performance issue with Manticore Search

General Discussions - English
1

Hello, the response time of my index on Manticore Search has increased a lot recently (some requests take several seconds). There are about 40 million documents currently indexed. How can I increase the performance of Manticore Search? Is there any configuration I can do that would improve it? I have put below my current configuration.

index my_index
{
    type                = rt
    path                = /mnt/.../manticore/data/my_index

    rt_field            = field1
    rt_field            = field2
    stored_fields       =

    rt_attr_timestamp   = attr1
    rt_attr_string      = attr2
    rt_attr_string      = attr3
    rt_attr_string      = attr4

    charset_table       = non_cjk
    ngram_len           = 1
    ngram_chars         = cjk

    rt_mem_limit        = 2048M
}

searchd
{
    listen              = 127.0.0.1:9312
    listen              = 127.0.0.1:9306:mysql41
    log                 = /var/log/manticore/searchd.log
    pid_file            = /var/run/manticore/searchd.pid
    seamless_rotate     = 1
    preopen_tables      = 1
    unlink_old          = 1
    net_workers         = 2
    binlog_path         = /mnt/.../manticore/data
    rt_flush_period     = 3600
}
2

There may be many reasons why your response time has increased latency. What can help you figure out the root cause I’d recommend you first of all look into the searchd log.

Then if you don’t see anything suspicious there, enable query log, let it run for some time, then inspect the query log to find out the slowest queries, then run them manually with SHOW META and SHOW PROFILE

Another thing which often helps is looking at SHOW THREADS at the moment when you see the system is overloaded.

You didn’t say what version you are running. Perhaps upgrading to a newer version will help.

If you are running an old version without automatic rt table compaction, you may have too many disk chunks which may be another reason of the slowdown. Or, vice versa, you may need to do FLUSH RAMCHUNK my_index to flush the ram chunk as a disk chunk. It sometimes helps a lot.

3

First of all, thank you for responding so quickly. I have several questions.

Regarding the version of Manticore Search I use version 6.0.4

I am currently using Manticore Search on a Basic Droplet, Premium AMD, 8 vCPUs and 16 GiB of RAM from DigitalOcean (the vCPUs are shared). Is this hardware configuration supposed to be sufficient to run Manticore Search?

Should the value of rt_mem_limit be large 2G, 4G or on the contrary small like the default value which is 128K?

Is optimize_cutoff supposed to set the number of disk_chunks or is this value automatic? I tried to force optimize_cutoff to 16 in the configuration then I restarted the Manticore Search and executed the following commands:
ALTER TABLE my_index RECONFIGURE;
FLUSH RAMCHUNK my_index;
OPTIMIZE TABLE my_index;
but the number of disk_chunks remained at 17.

Should I specify the cutoff option myself in the OPTIMIZE request?
The documentation says that the table’s disk chunks should be # of CPU cores * 2, can setting cutoff to the value of the number of CPUs improve the performance? Should the optimize_cutoff configuration be set globally so that the automatic OPTIMIZE uses this value?

When using SHOW THREADS I observed that most of the requests use all the available workers, is this a normal behavior?

Regarding the analysis of the queries, all the queries are sorted to the results by the attr1 (ORDER BY attr1 DESC, attr1 is a field of type rt_attr_timestamp), without the sorting the server responds faster but still not as fast as it should.

Is there a configuration to make Manticore Search load more stuff in RAM?

4

It depends on various factors, but the default setting might be suitable for many cases.

  • A larger value results in higher RAM consumption and fewer disk chunk merges, leading to lower CPU and I/O consumption.
  • A higher value results in lower RAM consumption and more frequent disk chunk merges, leading to higher CPU and I/O consumption.

Is optimize_cutoff supposed to set the number of disk_chunks or is this value automatic?

optimize_cutoff can be applied at three levels:

  1. In the searchd config - Manticore Search Manual: Server settings > Searchd
  2. Per-table - Manticore Search Manual: Creating a table > Local tables > Plain and real-time table settings
  3. When calling OPTIMIZE - Manticore Search Manual: Securing and compacting a table > Compacting a table

Should I specify the cutoff option myself in the OPTIMIZE request?

No, you shouldn’t.

Should the optimize_cutoff configuration be set globally so that the automatic OPTIMIZE uses this value?

Yes, it can be set globally, as I mentioned earlier.

When using SHOW THREADS I observed that most of the requests use all the available workers, is this a normal behavior?

Yes, as long as pseudo-sharding is not disabled. The idea is that you have paid for a powerful multi-core CPU, and Manticore tries to make the most of it to reduce response time.

Is there a configuration to make Manticore Search load more stuff in RAM?

Yes, there is. Please refer to Manticore Search Manual: Creating a table > Local tables > Plain and real-time table settings