How to control CPU usage

General Discussions - English
1

Is there any configuration to control to max amount of CPU manticore can use?

We are experiencing high CPU usage (constant 90%+ by the searchd processes) on our server, since manticore 4.2.

2

there was a change of how work items got utilized by threads at 4.2

you could set at daemon config options

searchd.threads as described here Manticore Search Manual: Searching > Options to lower cpu threads that your box has

or set searchd.max_connections as described here Manticore Search Manual: Server settings > Searchd to limit amount of clients daemon will process

2 Likes
3

This seems to not work really well.

I have 11 CPU’s available for only manticore (no other services).
If I set threads=4 then still the CPU of the searchd service peaks to 95% in soms cases which should be impossible.

I also have set max_threads_per_query = 1

Is there something I am missing?

4

If I set threads=4

As a search option or in the config?

What does show threads option format=all show at the moment the CPU utilization is 95%?

5

We found out that the CPU is under control. But the server is swapping extremely because of the searchd service. The server only runs Manticore.

It uses 3G swap constantly and growing, but there is enough RAM available (swapiness is set to 10).

Have you seen this before?

6

No, I haven’t seen the situation when there’s enough RAM and swapping at the same time. Make sure there’s enough RAM by mlock’ing all the files https://manual.manticoresearch.com/Creating_an_index/Local_indexes/Plain_and_real-time_index_settings#Accessing-index-files
Then if there’s not enough RAM search would fail right on start since it wouldn’t be able to mlock all the files. It may be also that there’s not enough RAM for some complex queries, but then you should see it in free and /proc/<pid>/smaps should show high consumption of anonymous memory.

7

We tried this:

access_plain_attrs = mlock
access_blob_attrs = mlock
access_doclists = mlock
access_hitlists = mlock

Size of data-dir = 19G
RAM available: 90G

Logs show:
[5604] WARNING: index ‘index.0’: mlock() failed: Cannot allocate memory for attributes

Whatever we do, the available RAM is simply ignored. We tried it with an index of 200M, still mlocking fails.

Is there maybe a setting how much total RAM manticore is allowed to use?

8

you should check that user starts daemon has enough limit to lock the memory with mlock with ulimit -l and in case of docker able to set CAP_IPC_LOCK flag

9

Changing ulimit -l to a higher value indeed solved this issue