So if have around 10 GB of data indexed in a rt_index, how much data out of 10 GB will be loaded to the RAM when I start searchd . Is there any way I can find out how much data is loaded to the RAM in the beginning?
Searchd RAM consumption
tomat
#2
Sum up the size of index files of this type for RT index .ram , .kill, .spa, .spb, .spi, .spk, .spm, .sphi, .spt
These files types for Manticore version 3 - for previous version you could find files list at doc topic indexing/indexes.html#index-files.
ie daemon loads all files into memory but keep on disk document list and keywords positions
tomat
#4
not quite clear what is swap space - we already use mmap sys call to manages files that loads into memory that is why it already might cached out from real RAM by OS
In the above image, the system is using 36% for RAM and remaining is cached and we can see it use 58% of Swap. We have total 650GB data in index folder (RT index). Will it affect the performance of the search ?
What is the ideal RAM size we are supposed to use?
tomat
#6
you have to calculate it as I describe it in 1st reply.
You also might to issue optimize command for you RT index to lower count of disk chunk and remove duplicated data from you RT index.
tomat
#7
in case you have your RT index loaded as I see that you have swap used you might issue
mysql>SHOW INDEX rt_name STATUS;
and it will show you ram_bytes - that is how many RAM used this index
mysql> SHOW INDEX rt_index STATUS;
+-------------------+----------------------------------------------------------------------------------------------------------+
| Variable_name | Value |
+-------------------+----------------------------------------------------------------------------------------------------------+
| index_type | rt |
| indexed_documents | 1433943 |
| indexed_bytes | 904065528 |
| ram_bytes | 1545098816 |
| disk_bytes | 1755151504 |
| ram_chunk | 907529485 |
| disk_chunks | 1 |
| mem_limit | 1073741824 |
| ram_bytes_retired | 0 |
| query_time_1min | {"queries":74, "avg_sec":0.000, "min_sec":0.000, "max_sec":0.032, "pct95_sec":0.032, "pct99_sec":0.032} |
| query_time_5min | {"queries":208, "avg_sec":0.000, "min_sec":0.000, "max_sec":0.032, "pct95_sec":0.019, "pct99_sec":0.032} |
| query_time_15min | {"queries":246, "avg_sec":0.002, "min_sec":0.000, "max_sec":0.121, "pct95_sec":0.031, "pct99_sec":0.121} |
| query_time_total | {"queries":413, "avg_sec":0.004, "min_sec":0.000, "max_sec":0.175, "pct95_sec":0.016, "pct99_sec":0.073} |
| found_rows_1min | {"queries":74, "avg":0, "min":0, "max":1, "pct95":1, "pct99":1} |
| found_rows_5min | {"queries":208, "avg":0, "min":0, "max":13, "pct95":1, "pct99":13} |
| found_rows_15min | {"queries":246, "avg":0, "min":0, "max":13, "pct95":1, "pct99":13} |
| found_rows_total | {"queries":413, "avg":0, "min":0, "max":13, "pct95":1, "pct99":3} |
+-------------------+----------------------------------------------------------------------------------------------------------+
17 rows in set (0.03 sec)
mysql> OPTIMIZE INDEX rt_index;
Query OK, 0 rows affected (0.00 sec)
mysql> SHOW INDEX rt_index STATUS;
+-------------------+----------------------------------------------------------------------------------------------------------+
| Variable_name | Value |
+-------------------+----------------------------------------------------------------------------------------------------------+
| index_type | rt |
| indexed_documents | 1433943 |
| indexed_bytes | 904065528 |
| ram_bytes | 1545098816 |
| disk_bytes | 1755151504 |
| ram_chunk | 907529485 |
| disk_chunks | 1 |
| mem_limit | 1073741824 |
| ram_bytes_retired | 0 |
| query_time_1min | {"queries":99, "avg_sec":0.000, "min_sec":0.000, "max_sec":0.014, "pct95_sec":0.003, "pct99_sec":0.003} |
| query_time_5min | {"queries":295, "avg_sec":0.000, "min_sec":0.000, "max_sec":0.032, "pct95_sec":0.012, "pct99_sec":0.032} |
| query_time_15min | {"queries":345, "avg_sec":0.001, "min_sec":0.000, "max_sec":0.121, "pct95_sec":0.030, "pct99_sec":0.121} |
| query_time_total | {"queries":512, "avg_sec":0.003, "min_sec":0.000, "max_sec":0.175, "pct95_sec":0.013, "pct99_sec":0.051} |
| found_rows_1min | {"queries":99, "avg":0, "min":0, "max":0, "pct95":0, "pct99":0} |
| found_rows_5min | {"queries":295, "avg":0, "min":0, "max":13, "pct95":1, "pct99":13} |
| found_rows_15min | {"queries":345, "avg":0, "min":0, "max":13, "pct95":1, "pct99":13} |
| found_rows_total | {"queries":512, "avg":0, "min":0, "max":13, "pct95":1, "pct99":2} |
+-------------------+----------------------------------------------------------------------------------------------------------+
17 rows in set (0.00 sec)
If you see, After we run mysql> OPTIMIZE INDEX rt_index; none of the numbers changed . Also i found ram_chunk is a big number . Any suggestion ?
tomat
#9
disk_chunks 1 already shows that you have only one disk chunk.
Seems you have big index. You might use ondisk \ mmap option to keep more parts of that index at disk.
However it could be better to provide box RAM, list of index files with its sizes and describe more on issue you have as currently I see no issues here.