Hello!
I use Manticore Search 6.3.9 and working on my own backup solution. I face the RT index crash (indextool --check says it’s invalid) after several FREEZE/UNFREEZE commands. The way to reproduce it:
- Create index:
create table search_idx (article text, article2 text) stopwords=‘ru, en’ morphology = ‘lemmatize_ru_all, lemmatize_en_all’;
-
I run a script with text insertion, it in infinity loop loads 1000 texts from external sources and insert it in my index with replace and one batch.
-
I run another bash script:
#!/bin/bash
counter=0
while true; do
date
((counter++))
mysql -P 9306 --host 127.0.0.1 -e “freeze search_idx”
echo “freezed”
# Каждый третий раз выполняем проверку
if [ $((counter % 3)) -eq 0 ]; then
indextool --check search_idx
fi
sleep 140
mysql -P 9306 --host 127.0.0.1 -e "unfreeze search_idx"
echo "unfreezed"
sleep 180
done
~
- In second iteration of indextool --check i receive following errors, they are always the same:
checking doc-id lookup…
FAILED, invalid docid delta 0 at row 59018, checkpoint 1139, doc 50, last docid 2431709789
FAILED, invalid docid 0(2431709789) at row 59018, checkpoint 1139, doc 50, last docid 2431709789
FAILED, invalid docid 1(2431709790) at row 58980, checkpoint 1139, doc 51, last docid 0
FAILED, invalid docid delta 0 at row 59017, checkpoint 1139, doc 52, last docid 1
FAILED, invalid docid 0(2431709790) at row 59017, checkpoint 1139, doc 52, last docid 1
FAILED, invalid docid 1(2431709791) at row 58979, checkpoint 1139, doc 53, last docid 0
FAILED, invalid docid delta 0 at row 59016, checkpoint 1139, doc 54, last docid 1
FAILED, invalid docid 0(2431709791) at row 59016, checkpoint 1139, doc 54, last docid 1
FAILED, invalid docid 1(2431709792) at row 58978, checkpoint 1139, doc 55, last docid 0
FAILED, invalid docid delta 0 at row 59015, checkpoint 1139, doc 56, last docid 1
FAILED, invalid docid 0(2431709792) at row 59015, checkpoint 1139, doc 56, last docid 1
FAILED, invalid docid 1(2431709793) at row 58977, checkpoint 1139, doc 57, last docid 0
FAILED, invalid docid delta 0 at row 59014, checkpoint 1139, doc 58, last docid 1
- After a while manticore crashes on some search request, and after restart can’t load index and says that it’s totally invalid
searchd log fragment:
[Sun Jun 8 21:32:34.043 2025] [2176840] rt: table search_idx: merged chunks /var/opt/manticore/search_idx/search_idx.39 and /var/opt/manticore/search_idx/search_idx.48 to /var/opt/manticore/search_idx/search_idx.49 in 17s (progressive mode). Remaining chunk count: 4
[Sun Jun 8 21:32:34.220 2025] [2176840] rt: table search_idx: optimized progressive chunk(s) 5 ( left 4 ) in 16s 887ms
[Sun Jun 8 21:33:03.198 2025] [2176841] WARNING: secondary library not loaded; secondary index(es) disabled
[Sun Jun 8 21:33:03.250 2025] [2176841] rt: table search_idx: diskchunk 50(5), segments 23 saved in 1.012643 (1.065342) sec, RAM saved/new 132285211/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Sun Jun 8 21:33:10.571 2025] [2176855] WARNING: secondary library not loaded; secondary index(es) disabled
[Sun Jun 8 21:33:10.575 2025] [2176855] rt: table search_idx: merged chunks /var/opt/manticore/search_idx/search_idx.45 and /var/opt/manticore/search_idx/search_idx.50 to /var/opt/manticore/search_idx/search_idx.51 in 7s (progressive mode). Remaining chunk count: 4
[Sun Jun 8 21:33:11.249 2025] [2176855] rt: table search_idx: optimized progressive chunk(s) 5 ( left 4 ) in 7s 999ms
[Sun Jun 8 21:33:34.622 2025] [2176860] WARNING: secondary library not loaded; secondary index(es) disabled
[Sun Jun 8 21:33:34.677 2025] [2176860] rt: table search_idx: diskchunk 52(5), segments 22 saved in 1.129789 (1.184931) sec, RAM saved/new 131974553/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Sun Jun 8 21:33:45.739 2025] [2176860] WARNING: secondary library not loaded; secondary index(es) disabled
[Sun Jun 8 21:33:45.752 2025] [2176860] rt: table search_idx: merged chunks /var/opt/manticore/search_idx/search_idx.47 and /var/opt/manticore/search_idx/search_idx.52 to /var/opt/manticore/search_idx/search_idx.53 in 11s (progressive mode). Remaining chunk count: 4
[Sun Jun 8 21:33:46.025 2025] [2176860] rt: table search_idx: optimized progressive chunk(s) 5 ( left 4 ) in 11s 348ms
[Sun Jun 8 21:33:53.439 2025] [2176865] rt: table search_idx: ramchunk saved ok (mode=forced, last TID=532, current TID=542, ram=65.081 Mb, time delta=18 sec, took=0.302 sec)
[Sun Jun 8 21:37:13.505 2025] [2176850] WARNING: secondary library not loaded; secondary index(es) disabled
[Sun Jun 8 21:37:15.152 2025] [2176847] WARNING: secondary library not loaded; secondary index(es) disabled
[Sun Jun 8 21:37:15.181 2025] [2176847] rt: table search_idx: diskchunk 54(5), segments 32 saved in 1.748132 (1.776778) sec, RAM saved/new 197290164/6519690 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
[Sun Jun 8 21:37:15.244 2025] [2176847] rt: table search_idx: diskchunk 55(6), segments 1 saved in 0.057743 (1.796093) sec, RAM saved/new 6519690/0 ratio 0.900000 (soft limit 120795955, conf limit 134217728)
[Sun Jun 8 21:37:24.579 2025] [2176843] WARNING: secondary library not loaded; secondary index(es) disabled
[Sun Jun 8 21:37:24.582 2025] [2176843] rt: table search_idx: merged chunks /var/opt/manticore/search_idx/search_idx.42 and /var/opt/manticore/search_idx/search_idx.54 to /var/opt/manticore/search_idx/search_idx.56 in 9s (progressive mode). Remaining chunk count: 5
[Sun Jun 8 21:37:31.179 2025] [2176843] WARNING: secondary library not loaded; secondary index(es) disabled
[Sun Jun 8 21:37:31.187 2025] [2176860] rt: table search_idx: merged chunks /var/opt/manticore/search_idx/search_idx.49 and /var/opt/manticore/search_idx/search_idx.55 to /var/opt/manticore/search_idx/search_idx.57 in 6s (progressive mode). Remaining chunk count: 4
[Sun Jun 8 21:37:31.328 2025] [2176860] rt: table search_idx: optimized progressive chunk(s) 6 ( left 4 ) in 16s 146ms
[Sun Jun 8 21:38:05.416 2025] [2176844] rt: table search_idx: ramchunk saved ok (mode=periodic, last TID=565, current TID=573, ram=36.599 Mb, time delta=50 sec, took=0.127 sec)
It’s a bug or i can’t run freeze/unfreeze so frequently on such high insert load?