Could you provide ls -lh of your index files?
Of course:
-rw------- 1 root root 30M 12. Jun 13:03 binlog.001
-rw------- 1 root root 0 9. Jun 21:02 binlog.lock
-rw------- 1 root root 11 12. Jun 09:02 binlog.meta
-rw------- 1 manticore manticore 418M 5. Jun 11:01 realtime.63.spa
-rw------- 1 manticore manticore 1,7G 5. Jun 11:04 realtime.63.spd
-rw------- 1 manticore manticore 13M 5. Jun 11:04 realtime.63.spe
-rw------- 1 manticore manticore 3,8K 5. Jun 11:04 realtime.63.sph
-rw------- 1 manticore manticore 171M 5. Jun 11:04 realtime.63.spi
-rw------- 1 manticore manticore 1,6M 5. Jun 11:04 realtime.63.spk
-rw------- 1 manticore manticore 246M 5. Jun 11:04 realtime.63.spm
-rw------- 1 manticore manticore 444M 5. Jun 11:04 realtime.63.spp
-rw------- 1 manticore manticore 228M 5. Jun 11:04 realtime.63.sps
-rw------- 1 manticore manticore 302K 12. Jun 09:02 realtime.kill
-rw------- 1 manticore manticore 0 9. Jun 21:01 realtime.lock
-rw------- 1 root root 3,8K 12. Jun 09:02 realtime.meta
-rw------- 1 root root 244M 12. Jun 09:02 realtime.ram
What me irritates here are the permissions. Isn’t it necessary to have all files belonging to manticore:manticore?
What is memory at graphic? Is it VSS or RSS?
Ugh… I’m sorry, I don’t know. The output is by prometheus’ node_exporter. I guess it’s RSS.
Could you provide index status output after daemon start and when memory goes to max?
Yes, I will collect them. Now - roughly 24 hours later, it looks like this:
+-------------------+--------------------------------------------------------------------------------------------------------------+
| Variable_name | Value |
+-------------------+--------------------------------------------------------------------------------------------------------------+
| index_type | rt |
| indexed_documents | 6415918 |
| indexed_bytes | 30107023907 |
| ram_bytes | 1431029613 |
| disk_bytes | 3594085657 |
| ram_chunk | 303229736 |
| disk_chunks | 1 |
| mem_limit | 1073741824 |
| query_time_1min | {"queries":156, "avg_sec":0.389, "min_sec":0.003, "max_sec":2.051, "pct95_sec":1.740, "pct99_sec":1.939} |
| query_time_5min | {"queries":530, "avg_sec":0.193, "min_sec":0.003, "max_sec":2.051, "pct95_sec":0.626, "pct99_sec":1.740} |
| query_time_15min | {"queries":1985, "avg_sec":0.126, "min_sec":0.003, "max_sec":2.051, "pct95_sec":0.495, "pct99_sec":1.179} |
| query_time_total | {"queries":372917, "avg_sec":0.080, "min_sec":0.001, "max_sec":26.231, "pct95_sec":0.256, "pct99_sec":0.563} |
| found_rows_1min | {"queries":156, "avg":21, "min":0, "max":1408, "pct95":162, "pct99":289} |
| found_rows_5min | {"queries":530, "avg":32, "min":0, "max":2573, "pct95":220, "pct99":361} |
| found_rows_15min | {"queries":1985, "avg":115, "min":0, "max":19964, "pct95":271, "pct99":4025} |
| found_rows_total | {"queries":372917, "avg":40, "min":0, "max":71263, "pct95":74, "pct99":622} |
+-------------------+--------------------------------------------------------------------------------------------------------------+
Could you set worker along with max_children to make sure you do not saturate your box with clients?
Yes. If nobody has objections I’d update to 2.7.0 today and check if the problem still exists.