How to resolve index file locking error?

General Discussions - English
1

Hi,

We’ve implemented a windows service that incrementally updates an index every 4 minutes The service performs a seeding process where the initial population of the index is done in groups of records using incremental updates to the main index. The steps taken are:

  1. Populate an incremental index with the updated group of records
  2. Merge the incremental index with the permanent index

All the indexes are plain table indexes.

During the process, we are getting a seemingly random occurring errors:

[Thu May 30 21:53:25.059 2024] [9588] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_indexA_incremental.old.spi’, error ‘Permission denied’

[Thu May 30 21:53:25.062 2024] [9588] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_ indexA _incremental.old.spe’, error ‘Permission denied’

[Thu May 30 21:53:25.063 2024] [9588] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_ indexA _incremental.old.spm’, error ‘Permission denied’

It happens even after granting full-access rights to the user ‘Everyone’. this is a windows box.

Any help to resolve this error would be greatly appreciated.

2

could you enable verbose log via SphinxQL SET GLOBAL log_level = debug then provide more lines (+1000) from the log around that error.

3

is there a way to set verbose debugging using powershell? i don’t have mysql client installed and i don’t have curl installed either (not allowed to). thank you for your patience.

4

you could restart daemon with cli --logdebug it will log verbose events from the start

5

i was able to set verbose errors, and re-ran the services. it didn’t spit out that much. here is the dump, hope it gives some clues:

Fri May 31 16:32:11.713 2024] [4036] DEBUG: StartGlobalWorkpool

[Fri May 31 16:32:11.714 2024] [4036] starting daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …

[Fri May 31 16:32:11.714 2024] [4036] listening on xx.xx.xx.xx:9312 for sphinx and http(s)

[Fri May 31 16:32:11.714 2024] [4036] listening on xx.xx.xx.xx:9306 for mysql

[Fri May 31 16:32:11.715 2024] [4036] DEBUG: Compiled-in value KNOWN_CREATE_SIZE (16) is less than measured (48).

[Fri May 31 16:32:11.715 2024] [4036] DEBUG: Frame expression stack for creation is 48 (mocked, as no env MANTICORE_KNOWN_CREATE_SIZE=48 found)

[Fri May 31 16:32:11.715 2024] [4036] DEBUG: Starting expression stack for creation is 4120 (mocked, as no env MANTICORE_START_KNOWN_CREATE_SIZE=4120 found)

[Fri May 31 16:32:11.716 2024] [4036] DEBUG: Compiled-in value KNOWN_EXPR_SIZE (32) is less than measured (96).

[Fri May 31 16:32:11.716 2024] [4036] DEBUG: Frame expression stack for eval/deletion is 96 (mocked, as no env MANTICORE_KNOWN_EXPR_SIZE=96 found)

[Fri May 31 16:32:11.716 2024] [4036] DEBUG: Starting expression stack for eval/deletion is 272 (mocked, as no env MANTICORE_START_KNOWN_EXPR_SIZE=272 found)

[Fri May 31 16:32:11.722 2024] [4036] DEBUG: Compiled-in value KNOWN_FILTER_SIZE (224) is less than measured (288).

[Fri May 31 16:32:11.722 2024] [4036] DEBUG: Frame filter stack delta is 288 (mocked, as no env MANTICORE_KNOWN_FILTER_SIZE=288 found)

[Fri May 31 16:32:11.723 2024] [4036] DEBUG: Starting filter stack delta is 12728 (mocked, as no env MANTICORE_START_KNOWN_FILTER_SIZE=12728 found)

[Fri May 31 16:32:11.730 2024] [4036] DEBUG: Frame fulltext match stack delta is 240 (mocked, as no env MANTICORE_KNOWN_MATCH_SIZE=240 found)

[Fri May 31 16:32:11.731 2024] [4036] DEBUG: Starting fulltext match stack delta is 15880 (mocked, as no env MANTICORE_START_KNOWN_MATCH_SIZE=15880 found)

[Fri May 31 16:32:11.731 2024] [4036] DEBUG: ‘rt_flush_period’ - nothing specified, using default value 36000000000

[Fri May 31 16:32:11.736 2024] [652] DEBUG: Locking the table via file E:\Manticore\indexes\manticore_document_index.spl

[Fri May 31 16:32:11.737 2024] [652] DEBUG: lock E:\Manticore\indexes\manticore_document_index.spl success

[Fri May 31 16:32:11.742 2024] [652] DEBUG: Locking the table via file E:\Manticore\indexes\manticore_document_incremental.spl

[Fri May 31 16:32:11.743 2024] [652] DEBUG: lock E:\Manticore\indexes\manticore_document_incremental.spl success

[Fri May 31 16:32:11.750 2024] [652] DEBUG: Locking the table via file E:\Manticore\indexes\manticore_case_index.spl

[Fri May 31 16:32:11.751 2024] [652] DEBUG: lock E:\Manticore\indexes\manticore_case_index.spl success

[Fri May 31 16:32:11.757 2024] [652] DEBUG: Locking the table via file E:\Manticore\indexes\manticore_case_incremental.spl

[Fri May 31 16:32:11.758 2024] [652] DEBUG: lock E:\Manticore\indexes\manticore_case_incremental.spl success

[Fri May 31 16:32:11.764 2024] [652] DEBUG: Locking the table via file E:\Manticore\indexes\manticore_case_seeder.spl

[Fri May 31 16:32:11.765 2024] [652] DEBUG: lock E:\Manticore\indexes\manticore_case_seeder.spl success

[Fri May 31 16:32:11.775 2024] [652] DEBUG: Locking the table via file E:\Manticore\indexes\manticore_people_index.spl

[Fri May 31 16:32:11.776 2024] [652] DEBUG: lock E:\Manticore\indexes\manticore_people_index.spl success

[Fri May 31 16:32:11.782 2024] [652] DEBUG: Locking the table via file E:\Manticore\indexes\manticore_people_incremental.spl

[Fri May 31 16:32:11.782 2024] [652] DEBUG: lock E:\Manticore\indexes\manticore_people_incremental.spl success

[Fri May 31 16:32:11.789 2024] [652] DEBUG: Locking the table via file E:\Manticore\indexes\manticore_people_seeder.spl

[Fri May 31 16:32:11.789 2024] [652] DEBUG: lock E:\Manticore\indexes\manticore_people_seeder.spl success

[Fri May 31 16:32:11.791 2024] [4036] DEBUG: ‘binlog_max_log_size’ - nothing specified, using default value 268435456

[Fri May 31 16:32:11.801 2024] [4036] DEBUG: MAC address xx for uuid-short server_id

[Fri May 31 16:32:11.801 2024] [4036] DEBUG: uid-short server_id 71, started 160504331, seed 5118781992523005952

[Fri May 31 16:32:11.805 2024] [416] DEBUG: SaveMeta: Done (E:\Manticore/var/log/manticore/bin.log/binlog.meta.new)

[Fri May 31 16:32:11.806 2024] [4908] prereading 8 tables

[Fri May 31 16:32:11.806 2024] [4908] DEBUG: prereading table ‘manticore_document_index’

[Fri May 31 16:32:11.806 2024] [4908] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_document_index’(E:\Manticore\indexes\manticore_document_index)

[Fri May 31 16:32:11.806 2024] [4908] DEBUG: Preread successfully finished

[Fri May 31 16:32:11.807 2024] [4908] DEBUG: preread table ‘manticore_document_index’ in 0.001 sec

[Fri May 31 16:32:11.807 2024] [4908] DEBUG: prereading table ‘manticore_document_incremental’

[Fri May 31 16:32:11.807 2024] [4908] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_document_incremental’(E:\Manticore\indexes\manticore_document_incremental)

[Fri May 31 16:32:11.807 2024] [4908] DEBUG: Preread successfully finished

[Fri May 31 16:32:11.808 2024] [4908] DEBUG: preread table ‘manticore_document_incremental’ in 0.001 sec

[Fri May 31 16:32:11.808 2024] [4908] DEBUG: prereading table ‘manticore_case_index’

[Fri May 31 16:32:11.808 2024] [4908] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_case_index’(E:\Manticore\indexes\manticore_case_index)

[Fri May 31 16:32:11.812 2024] [4036] DEBUG: dlopen(libcurl-x64.dll)=00007FFCD7B60000

[Fri May 31 16:32:11.863 2024] [4908] DEBUG: Preread successfully finished

[Fri May 31 16:32:11.864 2024] [4908] DEBUG: preread table ‘manticore_case_index’ in 0.055 sec

[Fri May 31 16:32:11.864 2024] [4908] DEBUG: prereading table ‘manticore_case_incremental’

[Fri May 31 16:32:11.864 2024] [4908] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_case_incremental’(E:\Manticore\indexes\manticore_case_incremental)

[Fri May 31 16:32:11.864 2024] [4908] DEBUG: Preread successfully finished

[Fri May 31 16:32:11.865 2024] [4908] DEBUG: preread table ‘manticore_case_incremental’ in 0.001 sec

[Fri May 31 16:32:11.865 2024] [4908] DEBUG: prereading table ‘manticore_case_seeder’

[Fri May 31 16:32:11.865 2024] [4908] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_case_seeder’(E:\Manticore\indexes\manticore_case_seeder)

[Fri May 31 16:32:11.865 2024] [4908] DEBUG: Preread successfully finished

[Fri May 31 16:32:11.866 2024] [4908] DEBUG: preread table ‘manticore_case_seeder’ in 0.001 sec

[Fri May 31 16:32:11.866 2024] [4908] DEBUG: prereading table ‘manticore_people_index’

[Fri May 31 16:32:11.866 2024] [4908] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_people_index’(E:\Manticore\indexes\manticore_people_index)

[Fri May 31 16:32:11.866 2024] [4036] accepting connections

[Fri May 31 16:32:11.942 2024] [4908] DEBUG: Preread successfully finished

[Fri May 31 16:32:11.943 2024] [4908] DEBUG: preread table ‘manticore_people_index’ in 0.077 sec

[Fri May 31 16:32:11.943 2024] [4908] DEBUG: prereading table ‘manticore_people_incremental’

[Fri May 31 16:32:11.943 2024] [4908] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_people_incremental’(E:\Manticore\indexes\manticore_people_incremental)

[Fri May 31 16:32:11.943 2024] [4908] DEBUG: Preread successfully finished

[Fri May 31 16:32:11.944 2024] [4908] DEBUG: preread table ‘manticore_people_incremental’ in 0.001 sec

[Fri May 31 16:32:11.944 2024] [4908] DEBUG: prereading table ‘manticore_people_seeder’

[Fri May 31 16:32:11.944 2024] [4908] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_people_seeder’(E:\Manticore\indexes\manticore_people_seeder)

[Fri May 31 16:32:11.944 2024] [4908] DEBUG: Preread successfully finished

[Fri May 31 16:32:11.945 2024] [4908] DEBUG: preread table ‘manticore_people_seeder’ in 0.001 sec

[Fri May 31 16:32:11.945 2024] [4908] preread 8 tables in 0.138 sec

[Fri May 31 16:32:12.329 2024] [8016] [BUDDY] started v1.0.18 ‘“E:\Manticore\bin\manticore-executor\manticore-executor.exe” “E:\Manticore/share/modules/manticore-buddy\src\main.php” --listen=http://10.62.224.55:9312 --threads=4’ at http://127.0.0.1:51751

[Fri May 31 16:32:12.439 2024] [652] [BUDDY] Loaded plugins:

[Fri May 31 16:32:12.439 2024] [652] [BUDDY] core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva

[Fri May 31 16:32:12.439 2024] [652] [BUDDY] local:

[Fri May 31 16:32:12.439 2024] [652] [BUDDY] extra:

[Fri May 31 16:33:14.423 2024] [4036] caught SIGHUP (seamless=0, in_rotate=0, need_rotate=0)

[Fri May 31 16:33:14.424 2024] [652] DEBUG: CheckRotate invoked

[Fri May 31 16:33:14.426 2024] [652] DEBUG: add deferred table manticore_document_index

[Fri May 31 16:33:14.426 2024] [652] DEBUG: keep existing table manticore_document_index

[Fri May 31 16:33:14.426 2024] [652] DEBUG: Keep existing before local table manticore_document_index

[Fri May 31 16:33:14.426 2024] [652] DEBUG: keep existing table manticore_document_incremental

[Fri May 31 16:33:14.426 2024] [652] DEBUG: Keep existing before local table manticore_document_incremental

[Fri May 31 16:33:14.427 2024] [652] DEBUG: keep existing table manticore_case_index

[Fri May 31 16:33:14.427 2024] [652] DEBUG: Keep existing before local table manticore_case_index

[Fri May 31 16:33:14.427 2024] [652] DEBUG: keep existing table manticore_case_incremental

[Fri May 31 16:33:14.427 2024] [652] DEBUG: Keep existing before local table manticore_case_incremental

[Fri May 31 16:33:14.427 2024] [652] DEBUG: keep existing table manticore_case_seeder

[Fri May 31 16:33:14.427 2024] [652] DEBUG: Keep existing before local table manticore_case_seeder

[Fri May 31 16:33:14.428 2024] [652] DEBUG: keep existing table manticore_people_index

[Fri May 31 16:33:14.428 2024] [652] DEBUG: Keep existing before local table manticore_people_index

[Fri May 31 16:33:14.428 2024] [652] DEBUG: keep existing table manticore_people_incremental

[Fri May 31 16:33:14.428 2024] [652] DEBUG: Keep existing before local table manticore_people_incremental

[Fri May 31 16:33:14.428 2024] [652] DEBUG: keep existing table manticore_people_seeder

[Fri May 31 16:33:14.428 2024] [652] DEBUG: Keep existing before local table manticore_people_seeder

[Fri May 31 16:33:14.429 2024] [652] DEBUG: will rotate manticore_document_index

[Fri May 31 16:33:14.429 2024] [652] DEBUG: greedy rotate local manticore_document_index

[Fri May 31 16:33:14.429 2024] [652] DEBUG: RotateIndexGreedy for ‘manticore_document_index’ invoked

[Fri May 31 16:33:14.431 2024] [652] DEBUG: E:\Manticore\indexes\manticore_document_index.old unlinked

[Fri May 31 16:33:14.442 2024] [652] DEBUG: last message repeated 13 times

[Fri May 31 16:33:14.442 2024] [652] DEBUG: E:\Manticore\indexes\manticore_document_index unlinked

[Fri May 31 16:33:14.466 2024] [652] DEBUG: last message repeated 13 times

[Fri May 31 16:33:14.466 2024] [652] DEBUG: rotating table ‘manticore_document_index’: applying other tables killlists

[Fri May 31 16:33:14.466 2024] [652] DEBUG: rotating table ‘manticore_document_index’: applying other tables killlists… DONE

[Fri May 31 16:33:14.466 2024] [652] DEBUG: rotating table ‘manticore_document_index’: apply killlist from this table to other tables (killlist_target)

[Fri May 31 16:33:14.466 2024] [652] DEBUG: rotating table ‘manticore_document_index’: apply killlist from this table to other tables (killlist_target)… DONE

[Fri May 31 16:33:14.466 2024] [652] rotating table ‘manticore_document_index’: success

[Fri May 31 16:33:14.466 2024] [652] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_document_index’(E:\Manticore\indexes\manticore_document_index)

[Fri May 31 16:33:14.467 2024] [652] DEBUG: Preread successfully finished

[Fri May 31 16:33:14.467 2024] [652] rotating finished

[Fri May 31 16:33:31.882 2024] [4036] caught SIGHUP (seamless=0, in_rotate=0, need_rotate=0)

[Fri May 31 16:33:31.883 2024] [652] DEBUG: CheckRotate invoked

[Fri May 31 16:33:31.884 2024] [652] DEBUG: keep existing table manticore_document_index

[Fri May 31 16:33:31.884 2024] [652] DEBUG: Keep existing before local table manticore_document_index

[Fri May 31 16:33:31.884 2024] [652] DEBUG: add deferred table manticore_document_incremental

[Fri May 31 16:33:31.884 2024] [652] DEBUG: keep existing table manticore_document_incremental

[Fri May 31 16:33:31.884 2024] [652] DEBUG: Keep existing before local table manticore_document_incremental

[Fri May 31 16:33:31.885 2024] [652] DEBUG: keep existing table manticore_case_index

[Fri May 31 16:33:31.885 2024] [652] DEBUG: Keep existing before local table manticore_case_index

[Fri May 31 16:33:31.885 2024] [652] DEBUG: keep existing table manticore_case_incremental

[Fri May 31 16:33:31.885 2024] [652] DEBUG: Keep existing before local table manticore_case_incremental

[Fri May 31 16:33:31.886 2024] [652] DEBUG: keep existing table manticore_case_seeder

[Fri May 31 16:33:31.886 2024] [652] DEBUG: Keep existing before local table manticore_case_seeder

[Fri May 31 16:33:31.886 2024] [652] DEBUG: keep existing table manticore_people_index

[Fri May 31 16:33:31.886 2024] [652] DEBUG: Keep existing before local table manticore_people_index

[Fri May 31 16:33:31.886 2024] [652] DEBUG: keep existing table manticore_people_incremental

[Fri May 31 16:33:31.886 2024] [652] DEBUG: Keep existing before local table manticore_people_incremental

[Fri May 31 16:33:31.887 2024] [652] DEBUG: keep existing table manticore_people_seeder

[Fri May 31 16:33:31.887 2024] [652] DEBUG: Keep existing before local table manticore_people_seeder

[Fri May 31 16:33:31.887 2024] [652] DEBUG: will rotate manticore_document_incremental

[Fri May 31 16:33:31.887 2024] [652] DEBUG: greedy rotate local manticore_document_incremental

[Fri May 31 16:33:31.887 2024] [652] DEBUG: RotateIndexGreedy for ‘manticore_document_incremental’ invoked

[Fri May 31 16:33:31.890 2024] [652] DEBUG: E:\Manticore\indexes\manticore_document_incremental.old unlinked

[Fri May 31 16:33:31.903 2024] [652] DEBUG: last message repeated 13 times

[Fri May 31 16:33:31.903 2024] [652] DEBUG: E:\Manticore\indexes\manticore_document_incremental unlinked

[Fri May 31 16:33:31.929 2024] [652] WARNING: last message repeated 13 times

[Fri May 31 16:33:31.929 2024] [652] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_document_incremental.old.spi’, error ‘Permission denied’

[Fri May 31 16:33:31.931 2024] [652] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_document_incremental.old.spe’, error ‘Permission denied’

[Fri May 31 16:33:31.934 2024] [652] DEBUG: rotating table ‘manticore_document_incremental’: applying other tables killlists

[Fri May 31 16:33:31.934 2024] [652] DEBUG: rotating table ‘manticore_document_incremental’: applying other tables killlists… DONE

[Fri May 31 16:33:31.934 2024] [652] DEBUG: rotating table ‘manticore_document_incremental’: apply killlist from this table to other tables (killlist_target)

[Fri May 31 16:33:31.934 2024] [652] DEBUG: rotating table ‘manticore_document_incremental’: apply killlist from this table to other tables (killlist_target)… DONE

[Fri May 31 16:33:31.934 2024] [652] rotating table ‘manticore_document_incremental’: success

[Fri May 31 16:33:31.934 2024] [652] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_document_incremental’(E:\Manticore\indexes\manticore_document_incremental)

[Fri May 31 16:33:31.936 2024] [652] DEBUG: Preread successfully finished

[Fri May 31 16:33:31.937 2024] [652] rotating finished

[Fri May 31 16:33:32.148 2024] [4036] caught SIGHUP (seamless=0, in_rotate=0, need_rotate=0)

[Fri May 31 16:33:32.148 2024] [8016] DEBUG: CheckRotate invoked

[Fri May 31 16:33:32.150 2024] [8016] DEBUG: add deferred table manticore_document_index

[Fri May 31 16:33:32.150 2024] [8016] DEBUG: keep existing table manticore_document_index

[Fri May 31 16:33:32.150 2024] [8016] DEBUG: Keep existing before local table manticore_document_index

[Fri May 31 16:33:32.150 2024] [8016] DEBUG: keep existing table manticore_document_incremental

[Fri May 31 16:33:32.150 2024] [8016] DEBUG: Keep existing before local table manticore_document_incremental

[Fri May 31 16:33:32.151 2024] [8016] DEBUG: keep existing table manticore_case_index

[Fri May 31 16:33:32.151 2024] [8016] DEBUG: Keep existing before local table manticore_case_index

[Fri May 31 16:33:32.151 2024] [8016] DEBUG: keep existing table manticore_case_incremental

[Fri May 31 16:33:32.151 2024] [8016] DEBUG: Keep existing before local table manticore_case_incremental

[Fri May 31 16:33:32.152 2024] [8016] DEBUG: keep existing table manticore_case_seeder

[Fri May 31 16:33:32.152 2024] [8016] DEBUG: Keep existing before local table manticore_case_seeder

[Fri May 31 16:33:32.152 2024] [8016] DEBUG: keep existing table manticore_people_index

[Fri May 31 16:33:32.152 2024] [8016] DEBUG: Keep existing before local table manticore_people_index

[Fri May 31 16:33:32.152 2024] [8016] DEBUG: keep existing table manticore_people_incremental

[Fri May 31 16:33:32.152 2024] [8016] DEBUG: Keep existing before local table manticore_people_incremental

[Fri May 31 16:33:32.153 2024] [8016] DEBUG: keep existing table manticore_people_seeder

[Fri May 31 16:33:32.153 2024] [8016] DEBUG: Keep existing before local table manticore_people_seeder

[Fri May 31 16:33:32.153 2024] [8016] DEBUG: will rotate manticore_document_index

[Fri May 31 16:33:32.153 2024] [8016] DEBUG: greedy rotate local manticore_document_index

[Fri May 31 16:33:32.153 2024] [8016] DEBUG: RotateIndexGreedy for ‘manticore_document_index’ invoked

[Fri May 31 16:33:32.156 2024] [8016] DEBUG: E:\Manticore\indexes\manticore_document_index.old unlinked

[Fri May 31 16:33:32.167 2024] [8016] DEBUG: last message repeated 13 times

[Fri May 31 16:33:32.167 2024] [8016] DEBUG: E:\Manticore\indexes\manticore_document_index unlinked

[Fri May 31 16:33:32.190 2024] [8016] DEBUG: last message repeated 13 times

[Fri May 31 16:33:32.190 2024] [8016] DEBUG: rotating table ‘manticore_document_index’: applying other tables killlists

[Fri May 31 16:33:32.191 2024] [8016] DEBUG: rotating table ‘manticore_document_index’: applying other tables killlists… DONE

[Fri May 31 16:33:32.191 2024] [8016] DEBUG: rotating table ‘manticore_document_index’: apply killlist from this table to other tables (killlist_target)

[Fri May 31 16:33:32.191 2024] [8016] DEBUG: rotating table ‘manticore_document_index’: apply killlist from this table to other tables (killlist_target)… DONE

[Fri May 31 16:33:32.191 2024] [8016] rotating table ‘manticore_document_index’: success

[Fri May 31 16:33:32.191 2024] [8016] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_document_index’(E:\Manticore\indexes\manticore_document_index)

[Fri May 31 16:33:32.191 2024] [8016] DEBUG: Preread successfully finished

[Fri May 31 16:33:32.192 2024] [8016] rotating finished

[Fri May 31 16:37:22.774 2024] [4036] caught SIGHUP (seamless=0, in_rotate=0, need_rotate=0)

[Fri May 31 16:37:22.774 2024] [8016] DEBUG: CheckRotate invoked

[Fri May 31 16:37:22.775 2024] [8016] DEBUG: keep existing table manticore_document_index

[Fri May 31 16:37:22.775 2024] [8016] DEBUG: Keep existing before local table manticore_document_index

[Fri May 31 16:37:22.776 2024] [8016] DEBUG: add deferred table manticore_document_incremental

[Fri May 31 16:37:22.776 2024] [8016] DEBUG: keep existing table manticore_document_incremental

[Fri May 31 16:37:22.776 2024] [8016] DEBUG: Keep existing before local table manticore_document_incremental

[Fri May 31 16:37:22.777 2024] [8016] DEBUG: keep existing table manticore_case_index

[Fri May 31 16:37:22.777 2024] [8016] DEBUG: Keep existing before local table manticore_case_index

[Fri May 31 16:37:22.777 2024] [8016] DEBUG: keep existing table manticore_case_incremental

[Fri May 31 16:37:22.777 2024] [8016] DEBUG: Keep existing before local table manticore_case_incremental

[Fri May 31 16:37:22.777 2024] [8016] DEBUG: keep existing table manticore_case_seeder

[Fri May 31 16:37:22.777 2024] [8016] DEBUG: Keep existing before local table manticore_case_seeder

[Fri May 31 16:37:22.778 2024] [8016] DEBUG: keep existing table manticore_people_index

[Fri May 31 16:37:22.778 2024] [8016] DEBUG: Keep existing before local table manticore_people_index

[Fri May 31 16:37:22.778 2024] [8016] DEBUG: keep existing table manticore_people_incremental

[Fri May 31 16:37:22.778 2024] [8016] DEBUG: Keep existing before local table manticore_people_incremental

[Fri May 31 16:37:22.779 2024] [8016] DEBUG: keep existing table manticore_people_seeder

[Fri May 31 16:37:22.779 2024] [8016] DEBUG: Keep existing before local table manticore_people_seeder

[Fri May 31 16:37:22.779 2024] [8016] DEBUG: will rotate manticore_document_incremental

[Fri May 31 16:37:22.779 2024] [8016] DEBUG: greedy rotate local manticore_document_incremental

[Fri May 31 16:37:22.779 2024] [8016] DEBUG: RotateIndexGreedy for ‘manticore_document_incremental’ invoked

[Fri May 31 16:37:22.783 2024] [8016] DEBUG: E:\Manticore\indexes\manticore_document_incremental.old unlinked

[Fri May 31 16:37:22.796 2024] [8016] DEBUG: last message repeated 13 times

[Fri May 31 16:37:22.796 2024] [8016] DEBUG: E:\Manticore\indexes\manticore_document_incremental unlinked

[Fri May 31 16:37:22.814 2024] [8016] WARNING: last message repeated 13 times

[Fri May 31 16:37:22.814 2024] [8016] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_document_incremental.old.spi’, error ‘Permission denied’

[Fri May 31 16:37:22.817 2024] [8016] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_document_incremental.old.spe’, error ‘Permission denied’

[Fri May 31 16:37:22.817 2024] [8016] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_document_incremental.old.spm’, error ‘Permission denied’

[Fri May 31 16:37:22.819 2024] [8016] DEBUG: rotating table ‘manticore_document_incremental’: applying other tables killlists

[Fri May 31 16:37:22.820 2024] [8016] DEBUG: rotating table ‘manticore_document_incremental’: applying other tables killlists… DONE

[Fri May 31 16:37:22.820 2024] [8016] DEBUG: rotating table ‘manticore_document_incremental’: apply killlist from this table to other tables (killlist_target)

[Fri May 31 16:37:22.820 2024] [8016] DEBUG: rotating table ‘manticore_document_incremental’: apply killlist from this table to other tables (killlist_target)… DONE

[Fri May 31 16:37:22.820 2024] [8016] rotating table ‘manticore_document_incremental’: success

[Fri May 31 16:37:22.820 2024] [8016] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_document_incremental’(E:\Manticore\indexes\manticore_document_incremental)

[Fri May 31 16:37:22.820 2024] [8016] DEBUG: Preread successfully finished

[Fri May 31 16:37:22.821 2024] [8016] rotating finished

[Fri May 31 16:41:27.205 2024] [4036] caught SIGHUP (seamless=0, in_rotate=0, need_rotate=0)

[Fri May 31 16:41:27.205 2024] [8016] DEBUG: CheckRotate invoked

[Fri May 31 16:41:27.207 2024] [8016] DEBUG: keep existing table manticore_document_index

[Fri May 31 16:41:27.207 2024] [8016] DEBUG: Keep existing before local table manticore_document_index

[Fri May 31 16:41:27.208 2024] [8016] DEBUG: add deferred table manticore_document_incremental

[Fri May 31 16:41:27.208 2024] [8016] DEBUG: keep existing table manticore_document_incremental

[Fri May 31 16:41:27.208 2024] [8016] DEBUG: Keep existing before local table manticore_document_incremental

[Fri May 31 16:41:27.209 2024] [8016] DEBUG: keep existing table manticore_case_index

[Fri May 31 16:41:27.209 2024] [8016] DEBUG: Keep existing before local table manticore_case_index

[Fri May 31 16:41:27.209 2024] [8016] DEBUG: keep existing table manticore_case_incremental

[Fri May 31 16:41:27.209 2024] [8016] DEBUG: Keep existing before local table manticore_case_incremental

[Fri May 31 16:41:27.210 2024] [8016] DEBUG: keep existing table manticore_case_seeder

[Fri May 31 16:41:27.210 2024] [8016] DEBUG: Keep existing before local table manticore_case_seeder

[Fri May 31 16:41:27.210 2024] [8016] DEBUG: keep existing table manticore_people_index

[Fri May 31 16:41:27.210 2024] [8016] DEBUG: Keep existing before local table manticore_people_index

[Fri May 31 16:41:27.211 2024] [8016] DEBUG: keep existing table manticore_people_incremental

[Fri May 31 16:41:27.211 2024] [8016] DEBUG: Keep existing before local table manticore_people_incremental

[Fri May 31 16:41:27.212 2024] [8016] DEBUG: keep existing table manticore_people_seeder

[Fri May 31 16:41:27.212 2024] [8016] DEBUG: Keep existing before local table manticore_people_seeder

[Fri May 31 16:41:27.212 2024] [8016] DEBUG: will rotate manticore_document_incremental

[Fri May 31 16:41:27.212 2024] [8016] DEBUG: greedy rotate local manticore_document_incremental

[Fri May 31 16:41:27.212 2024] [8016] DEBUG: RotateIndexGreedy for ‘manticore_document_incremental’ invoked

[Fri May 31 16:41:27.216 2024] [8016] DEBUG: E:\Manticore\indexes\manticore_document_incremental.old unlinked

[Fri May 31 16:41:27.234 2024] [8016] DEBUG: last message repeated 13 times

[Fri May 31 16:41:27.234 2024] [8016] DEBUG: E:\Manticore\indexes\manticore_document_incremental unlinked

6

the rest of the dump:

[Fri May 31 16:41:27.253 2024] [8016] WARNING: last message repeated 13 times

[Fri May 31 16:41:27.253 2024] [8016] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_document_incremental.old.spi’, error ‘Permission denied’

[Fri May 31 16:41:27.255 2024] [8016] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_document_incremental.old.spe’, error ‘Permission denied’

[Fri May 31 16:41:27.255 2024] [8016] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_document_incremental.old.spm’, error ‘Permission denied’

[Fri May 31 16:41:27.257 2024] [8016] DEBUG: rotating table ‘manticore_document_incremental’: applying other tables killlists

[Fri May 31 16:41:27.257 2024] [8016] DEBUG: rotating table ‘manticore_document_incremental’: applying other tables killlists… DONE

[Fri May 31 16:41:27.257 2024] [8016] DEBUG: rotating table ‘manticore_document_incremental’: apply killlist from this table to other tables (killlist_target)

[Fri May 31 16:41:27.257 2024] [8016] DEBUG: rotating table ‘manticore_document_incremental’: apply killlist from this table to other tables (killlist_target)… DONE

[Fri May 31 16:41:27.257 2024] [8016] rotating table ‘manticore_document_incremental’: success

[Fri May 31 16:41:27.257 2024] [8016] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_document_incremental’(E:\Manticore\indexes\manticore_document_incremental)

[Fri May 31 16:41:27.258 2024] [8016] DEBUG: Preread successfully finished

[Fri May 31 16:41:27.259 2024] [8016] rotating finished

[Fri May 31 16:45:36.315 2024] [4036] caught SIGHUP (seamless=0, in_rotate=0, need_rotate=0)

[Fri May 31 16:45:36.315 2024] [416] DEBUG: CheckRotate invoked

[Fri May 31 16:45:36.317 2024] [416] DEBUG: keep existing table manticore_document_index

[Fri May 31 16:45:36.317 2024] [416] DEBUG: Keep existing before local table manticore_document_index

[Fri May 31 16:45:36.317 2024] [416] DEBUG: add deferred table manticore_document_incremental

[Fri May 31 16:45:36.317 2024] [416] DEBUG: keep existing table manticore_document_incremental

[Fri May 31 16:45:36.317 2024] [416] DEBUG: Keep existing before local table manticore_document_incremental

[Fri May 31 16:45:36.318 2024] [416] DEBUG: keep existing table manticore_case_index

[Fri May 31 16:45:36.318 2024] [416] DEBUG: Keep existing before local table manticore_case_index

[Fri May 31 16:45:36.318 2024] [416] DEBUG: keep existing table manticore_case_incremental

[Fri May 31 16:45:36.318 2024] [416] DEBUG: Keep existing before local table manticore_case_incremental

[Fri May 31 16:45:36.318 2024] [416] DEBUG: keep existing table manticore_case_seeder

[Fri May 31 16:45:36.318 2024] [416] DEBUG: Keep existing before local table manticore_case_seeder

[Fri May 31 16:45:36.319 2024] [416] DEBUG: keep existing table manticore_people_index

[Fri May 31 16:45:36.319 2024] [416] DEBUG: Keep existing before local table manticore_people_index

[Fri May 31 16:45:36.319 2024] [416] DEBUG: keep existing table manticore_people_incremental

[Fri May 31 16:45:36.319 2024] [416] DEBUG: Keep existing before local table manticore_people_incremental

[Fri May 31 16:45:36.320 2024] [416] DEBUG: keep existing table manticore_people_seeder

[Fri May 31 16:45:36.320 2024] [416] DEBUG: Keep existing before local table manticore_people_seeder

[Fri May 31 16:45:36.320 2024] [416] DEBUG: will rotate manticore_document_incremental

[Fri May 31 16:45:36.320 2024] [416] DEBUG: greedy rotate local manticore_document_incremental

[Fri May 31 16:45:36.320 2024] [416] DEBUG: RotateIndexGreedy for ‘manticore_document_incremental’ invoked

[Fri May 31 16:45:36.323 2024] [416] DEBUG: E:\Manticore\indexes\manticore_document_incremental.old unlinked

[Fri May 31 16:45:36.336 2024] [416] DEBUG: last message repeated 13 times

[Fri May 31 16:45:36.336 2024] [416] DEBUG: E:\Manticore\indexes\manticore_document_incremental unlinked

[Fri May 31 16:45:36.354 2024] [416] WARNING: last message repeated 13 times

[Fri May 31 16:45:36.354 2024] [416] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_document_incremental.old.spi’, error ‘Permission denied’

[Fri May 31 16:45:36.357 2024] [416] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_document_incremental.old.spe’, error ‘Permission denied’

[Fri May 31 16:45:36.357 2024] [416] WARNING: unlink failed (file ‘E:\Manticore\indexes\manticore_document_incremental.old.spm’, error ‘Permission denied’

[Fri May 31 16:45:36.359 2024] [416] DEBUG: rotating table ‘manticore_document_incremental’: applying other tables killlists

[Fri May 31 16:45:36.359 2024] [416] DEBUG: rotating table ‘manticore_document_incremental’: applying other tables killlists… DONE

[Fri May 31 16:45:36.359 2024] [416] DEBUG: rotating table ‘manticore_document_incremental’: apply killlist from this table to other tables (killlist_target)

[Fri May 31 16:45:36.359 2024] [416] DEBUG: rotating table ‘manticore_document_incremental’: apply killlist from this table to other tables (killlist_target)… DONE

[Fri May 31 16:45:36.359 2024] [416] rotating table ‘manticore_document_incremental’: success

[Fri May 31 16:45:36.359 2024] [416] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_document_incremental’(E:\Manticore\indexes\manticore_document_incremental)

[Fri May 31 16:45:36.361 2024] [416] DEBUG: Preread successfully finished

[Fri May 31 16:45:36.361 2024] [416] rotating finished

[Fri May 31 16:45:37.237 2024] [4036] caught SIGHUP (seamless=0, in_rotate=0, need_rotate=0)

[Fri May 31 16:45:37.237 2024] [416] DEBUG: CheckRotate invoked

[Fri May 31 16:45:37.239 2024] [416] DEBUG: add deferred table manticore_document_index

[Fri May 31 16:45:37.239 2024] [416] DEBUG: keep existing table manticore_document_index

[Fri May 31 16:45:37.239 2024] [416] DEBUG: Keep existing before local table manticore_document_index

[Fri May 31 16:45:37.239 2024] [416] DEBUG: keep existing table manticore_document_incremental

[Fri May 31 16:45:37.239 2024] [416] DEBUG: Keep existing before local table manticore_document_incremental

[Fri May 31 16:45:37.240 2024] [416] DEBUG: keep existing table manticore_case_index

[Fri May 31 16:45:37.240 2024] [416] DEBUG: Keep existing before local table manticore_case_index

[Fri May 31 16:45:37.240 2024] [416] DEBUG: keep existing table manticore_case_incremental

[Fri May 31 16:45:37.240 2024] [416] DEBUG: Keep existing before local table manticore_case_incremental

[Fri May 31 16:45:37.241 2024] [416] DEBUG: keep existing table manticore_case_seeder

[Fri May 31 16:45:37.241 2024] [416] DEBUG: Keep existing before local table manticore_case_seeder

[Fri May 31 16:45:37.241 2024] [416] DEBUG: keep existing table manticore_people_index

[Fri May 31 16:45:37.241 2024] [416] DEBUG: Keep existing before local table manticore_people_index

[Fri May 31 16:45:37.241 2024] [416] DEBUG: keep existing table manticore_people_incremental

[Fri May 31 16:45:37.241 2024] [416] DEBUG: Keep existing before local table manticore_people_incremental

[Fri May 31 16:45:37.242 2024] [416] DEBUG: keep existing table manticore_people_seeder

[Fri May 31 16:45:37.242 2024] [416] DEBUG: Keep existing before local table manticore_people_seeder

[Fri May 31 16:45:37.242 2024] [416] DEBUG: will rotate manticore_document_index

[Fri May 31 16:45:37.242 2024] [416] DEBUG: greedy rotate local manticore_document_index

[Fri May 31 16:45:37.242 2024] [416] DEBUG: RotateIndexGreedy for ‘manticore_document_index’ invoked

[Fri May 31 16:45:37.245 2024] [416] DEBUG: E:\Manticore\indexes\manticore_document_index.old unlinked

[Fri May 31 16:45:37.256 2024] [416] DEBUG: last message repeated 13 times

[Fri May 31 16:45:37.256 2024] [416] DEBUG: E:\Manticore\indexes\manticore_document_index unlinked

[Fri May 31 16:45:37.279 2024] [416] DEBUG: last message repeated 13 times

[Fri May 31 16:45:37.279 2024] [416] DEBUG: rotating table ‘manticore_document_index’: applying other tables killlists

[Fri May 31 16:45:37.279 2024] [416] DEBUG: rotating table ‘manticore_document_index’: applying other tables killlists… DONE

[Fri May 31 16:45:37.279 2024] [416] DEBUG: rotating table ‘manticore_document_index’: apply killlist from this table to other tables (killlist_target)

[Fri May 31 16:45:37.279 2024] [416] DEBUG: rotating table ‘manticore_document_index’: apply killlist from this table to other tables (killlist_target)… DONE

[Fri May 31 16:45:37.279 2024] [416] rotating table ‘manticore_document_index’: success

[Fri May 31 16:45:37.279 2024] [416] DEBUG: CSphIndex_VLN::Preread invoked ‘manticore_document_index’(E:\Manticore\indexes\manticore_document_index)

[Fri May 31 16:45:37.280 2024] [416] DEBUG: Preread successfully finished

[Fri May 31 16:45:37.281 2024] [416] rotating finished

[Fri May 31 16:47:22.315 2024] [4036] DEBUG: Unlocking the table (lock E:\Manticore\indexes\manticore_document_index.spl)

[Fri May 31 16:47:22.315 2024] [4036] DEBUG: File ID ok, closing lock FD 10, unlinking E:\Manticore\indexes\manticore_document_index.spl

[Fri May 31 16:47:22.316 2024] [4036] DEBUG: Unlocking the table (lock E:\Manticore\indexes\manticore_document_incremental.spl)

[Fri May 31 16:47:22.316 2024] [4036] DEBUG: File ID ok, closing lock FD 14, unlinking E:\Manticore\indexes\manticore_document_incremental.spl

[Fri May 31 16:47:22.317 2024] [4036] DEBUG: Unlocking the table (lock E:\Manticore\indexes\manticore_case_index.spl)

[Fri May 31 16:47:22.317 2024] [4036] DEBUG: File ID ok, closing lock FD 19, unlinking E:\Manticore\indexes\manticore_case_index.spl

[Fri May 31 16:47:22.317 2024] [4036] DEBUG: Unlocking the table (lock E:\Manticore\indexes\manticore_case_incremental.spl)

[Fri May 31 16:47:22.317 2024] [4036] DEBUG: File ID ok, closing lock FD 24, unlinking E:\Manticore\indexes\manticore_case_incremental.spl

[Fri May 31 16:47:22.318 2024] [4036] DEBUG: Unlocking the table (lock E:\Manticore\indexes\manticore_case_seeder.spl)

[Fri May 31 16:47:22.318 2024] [4036] DEBUG: File ID ok, closing lock FD 29, unlinking E:\Manticore\indexes\manticore_case_seeder.spl

[Fri May 31 16:47:22.318 2024] [4036] DEBUG: Unlocking the table (lock E:\Manticore\indexes\manticore_people_index.spl)

[Fri May 31 16:47:22.318 2024] [4036] DEBUG: File ID ok, closing lock FD 34, unlinking E:\Manticore\indexes\manticore_people_index.spl

[Fri May 31 16:47:22.319 2024] [4036] DEBUG: Unlocking the table (lock E:\Manticore\indexes\manticore_people_incremental.spl)

[Fri May 31 16:47:22.319 2024] [4036] DEBUG: File ID ok, closing lock FD 39, unlinking E:\Manticore\indexes\manticore_people_incremental.spl

[Fri May 31 16:47:22.319 2024] [4036] DEBUG: Unlocking the table (lock E:\Manticore\indexes\manticore_people_seeder.spl)

[Fri May 31 16:47:22.319 2024] [4036] DEBUG: File ID ok, closing lock FD 44, unlinking E:\Manticore\indexes\manticore_people_seeder.spl

[Fri May 31 16:47:22.390 2024] [4036] DEBUG: StopGlobalWorkPool

7

Even though you have set permissions for ‘Everyone’, it’s worthwhile to double-check the effective permissions on the files at the time of the error. You can use the ‘icacls’ command to view or modify file permissions:

icacls "E:\Manticore\indexes\manticore_indexA_incremental.old.spi"

This command will show you the permissions active on the file, ensuring they are correctly set.

8

I went ahead and used the command to double-check effective file permissions, and it did include “Everyone: (I) (F)” in the list. Any other ideas/suggestions would be greatly appreciated.

9

these are the indexing commands i use:

   indexer --config E:\Manticore\etc\manticoresearch\manticore.conf manticore_incremental_document_index --rotate";



  indexer --config E:\Manticore\etc\manticoresearch\manticore.conf --merge manticore_create_document_index manticore_incremental_document_index --rotate;
10

maybe you issue indexes merge at the same time as daemon try to pick up and rotate indexes. You need to wait between these operations.

11

apologies for posting incomplete code – I do already wait between operations. in the application, i pass the first one to a Processor method that uses cmd.exe, and wait for an exit code, then if successful exit code, I pass the second indexer command to the Processor method.

Processor method code:

public static bool RunProcessor(string indexerCommand)

                            {

                                            try

                                            {

                                                           

                                                            ProcessStartInfo startInfo = new ProcessStartInfo();

                                                            startInfo.FileName = "C:\\Windows\\System32\\cmd.exe";

                                                            startInfo.Arguments = $"/C \"{indexerCommand}\"";



                                                            startInfo.UseShellExecute = false;

                                                            startInfo.RedirectStandardOutput = true;

                                                            startInfo.RedirectStandardError = true;

                                                            startInfo.CreateNoWindow = true;



                                                            int exitCode = 0;

                                                            string result = string.Empty;

                                                            string error = string.Empty;



                                                            using (Process process = Process.Start(startInfo))

                                                            {

                                                                            using (var reader = process.StandardOutput)

                                                                            {

                                                                                            result = reader.ReadToEnd();

                                                                                           

                                                                            }

                                                                           



                                                                            exitCode = process.ExitCode;

                                                                            process.WaitForExit();

                                                                           



                                                            }



                                                            if (exitCode != 0)

                                                            {

                                                                            if (result.Length > 0)

                                                                            {

                                                                                            NotifyTeam("Error Running Processor: " + result);

                                                                            }

                                                                            if (error.Length > 0)

                                                                            {

                                                                                            NotifyTeam("Error Running Processor: " + error);

                                                                            }

                                                            }



                                                            return exitCode == 0;



                                            }

                                            catch (Exception e)

                                            {



                                                            ManticoreManagerHelpers.NotifyTeamOfError(e, "RunProcessor");

                                                            return false;

                                            }



                            }
12

after indexer finished its work it sends signal to damon then daemon going to pickup and rotate the index. You can not sure that daemon finished rotation just by looking at the indexer.

13

i added a timer delay between the two indexing commands and the seeding process completed without errors! thank you @tomat and @Sergey for the invaluable help in resolving this issue.

1 Like