is there a way to recover corrupted index?

General Discussions - English
1

environment:

  • Manticore 6.3.6 installed from deb package
  • Debian GNU/Linux 12 (bookworm)
  • amd64 server

after few days of testing i’ve cleanly rebooted server with manticore. after the restart one of indexes was no longer available in manticore.

searchd.log:

[Tue Sep  3 18:45:07.826 2024] [11595] caught SIGTERM, shutting down
[Tue Sep  3 18:45:07.833 2024] [11646] rt: table index0: ramchunk saved in 0.003 sec
[Tue Sep  3 18:45:10.411 2024] [11646] rt: table index1: ramchunk saved in 2.576 sec
[Tue Sep  3 18:45:19.157 2024] [11646] rt: table index2: ramchunk saved in 1.252 sec
[Tue Sep  3 18:45:19.499 2024] [11595] shutdown daemon version '6.3.6 593045790@24080214 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
[Tue Sep  3 18:45:19.499 2024] [11595] shutdown complete
[Tue Sep  3 18:45:19.512 2024] [11594] watchdog: main process 11595 exited cleanly (exit code 0), shutting down
[Tue Sep  3 18:47:31.724 2024] [1628] watchdog: main process 1630 forked ok
[Tue Sep  3 18:47:31.727 2024] [1630] Using local time zone '/etc/localtime'
[Tue Sep  3 18:47:31.733 2024] [1630] starting daemon version '6.3.6 593045790@24080214 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
[Tue Sep  3 18:47:31.733 2024] [1630] listening on all interfaces for mysql, port=9306
[Tue Sep  3 18:47:31.763 2024] [1642] WARNING: table 'index1': disabled at the JSON config
[Tue Sep  3 18:47:31.763 2024] [1642] WARNING: table 'index1': prealloc: disk chunk /var/lib/manticore/data/index1/index1.91469: prealloc failed: error opening '/var/lib/manticore/data/index1/index1.91469.spknn': No such file or directory - NOT SERVING
[Tue Sep  3 18:47:31.764 2024] [1642] Index header format is not json, will try it as binary...
[Tue Sep  3 18:47:32.421 2024] [1652] last message repeated 2 times
[Tue Sep  3 18:47:32.421 2024] [1652] prereading 2 tables
[Tue Sep  3 18:47:32.421 2024] [1630] WARNING: [BUDDY] no SPHINX or HTTP listeners found, disabled
[Tue Sep  3 18:47:32.421 2024] [1630] accepting connections
[Tue Sep  3 18:47:33.505 2024] [1652] preread 2 tables in 1.084 sec

before the restart manticore was idle for long time [ hours ], i did not reboot it in the middle of any heavy write operations.

is there any way to recover from such state without losing the whole index? sadly i no longer have files left by it.

thank you!

2

no you have to reindex the data. It could be better to catch the flow what operations cause knn file missed?

3

thx for info. i’ll get back with github issue if i manage to find a reliable workflow to reproduce it. few days earlier i’ve done ALTER on that index and added new column for embeddings.

4

maybe that is the optimize issue when optimize merges disk chunks with the different schema but I not sure about that. It could be better to start daemon with --logdebug to log all files operations then track back in time when the bad disk chunk created and by what operation

5

thx for the hint!

6

i can reproduce this issue; i’ve submitted a bug report at adding column with float_vector leads to index corruption · Issue #2544 · manticoresoftware/manticoresearch · GitHub