While we start the searchd service it’s started but immediately service is going to stopped and searchd service is not running. We check in the /var/log/manticore/manticore.log we got the below error.
ERROR:
FATAL: binlog: commit: unexpected table id (id=84, max=1, pos=[121194730]
We are using manticore search for the mail archive application. We have a manticore 6.2.12 version.
Looks like a binlog corruption. Please read about --replay-flags here Manticore Search Manual: Starting the server > Manually
E.g. you can try:
sudo systemctl stop manticore
sudo -u manticore searchd --console --replay-flags=ignore-all-errors
<CTRL-C> after it's fully started
sudo systemctl start manticore
searchd --config /usr/local/etc/piler/manticore.conf --console --replay-flags=ignore-all-errors
We are not run the manticore service through systemctl instead we are running like above command. Please check the syntex is correct with your given solution command.
Yah, Now manticore is started successfully in port 9306 without any issue.
Thanks for your fastest response it’s very helpful to fixed our production issue.
Please let me know why binlog corruption? And how to avoid this will not happened in future.
searchd service it’s started but immediately service is going to stoped and searchd service is not running.
we got the below error. how can we remove the (blop=84, pos=14944188)
binlog: replaying log /var/piler/manticore/binlog.001
WARNING: binlog:unexpected entry (blop=84, pos=14944188)
WARNING: binlog:replay error at pos=14944188
binlog:table pilerl: skipped at tid 120079590 and max binlog tid 120051863
binlog: replay stats: 7558 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 1tables
binlog: finished replaying /var/piler/manticore/binlog.001; 140.9 MB in 0.058 sec
binlog: finished replaying total 1 in 0.058 sec
prereading 3 tables
accepting connections
preread 3 tables in 190.942 sec
could you provide whole search.log?
could you start daemon with --logdebug and provide searchd.log ? or check the dmesg for any traces of the crash?
I have paste searchd.log, and also I was execute the command with debug option
$ searchd --config /usr/local/etc/piler/manticore.conf --console --replay-flags=ignore-all-errors --logdebug
Now error message was didn’t come. now searchd was started & running.
will run the same when will error comes again the same will post.
$ searchd --config /usr/local/etc/piler/manticore.conf --console --replay-flags=ignore-all-errors --logdebug
Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)
[42:54.027] [3369992] using config file ‘/usr/local/etc/piler/manticore.conf’ (3536 chars)…
[42:54.030] [3369992] DEBUG: ‘read_timeout’ - nothing specified, using default value 5000000
[42:54.030] [3369992] DEBUG: ‘sphinxql_timeout’ - nothing specified, using default value 900000000
[42:54.030] [3369992] DEBUG: ‘client_timeout’ - nothing specified, using default value 300000000
[42:54.030] [3369992] DEBUG: SetMaxChildrenThreads to 16
[42:54.030] [3369992] DEBUG: ‘read_unhinted’ - nothing specified, using default value 32768
[42:54.030] [3369992] DEBUG: ‘read_buffer’ - nothing specified, using default value 262144
[42:54.030] [3369992] DEBUG: ‘read_buffer_docs’ - nothing specified, using default value 262144
[42:54.030] [3369992] DEBUG: ‘read_buffer_hits’ - nothing specified, using default value 262144
[42:54.030] [3369992] DEBUG: ‘attr_flush_period’ - nothing specified, using default value 0
[42:54.030] [3369992] DEBUG: ‘max_packet_size’ - nothing specified, using default value 8388608
[42:54.030] [3369992] DEBUG: ‘rt_merge_maxiosize’ - nothing specified, using default value 0
[42:54.030] [3369992] DEBUG: ‘ha_ping_interval’ - nothing specified, using default value 1000000
[42:54.030] [3369992] DEBUG: ‘ha_period_karma’ - nothing specified, using default value 60000000
[42:54.030] [3369992] DEBUG: ‘query_log_min_msec’ - nothing specified, using default value 0
[42:54.030] [3369992] DEBUG: ‘agent_connect_timeout’ - nothing specified, using default value 1000000
[42:54.030] [3369992] DEBUG: ‘agent_query_timeout’ - nothing specified, using default value 3000000
[42:54.030] [3369992] DEBUG: ‘agent_retry_delay’ - nothing specified, using default value 500000
[42:54.030] [3369992] DEBUG: ‘net_wait_tm’ - nothing specified, using default value -1
[42:54.030] [3369992] DEBUG: ‘docstore_cache_size’ - nothing specified, using default value 16777216
[42:54.030] [3369992] DEBUG: ‘skiplist_cache_size’ - nothing specified, using default value 67108864
[42:54.030] [3369992] DEBUG: ‘qcache_max_bytes’ - nothing specified, using default value 16777216
[42:54.030] [3369992] DEBUG: ‘qcache_thresh_msec’ - nothing specified, using default value 3000000
[42:54.030] [3369992] DEBUG: ‘qcache_ttl_sec’ - nothing specified, using default value 60000000
DEBUG: StartGlobalWorkpool
starting daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
listening on 127.0.0.1:9312 for sphinx and http(s)
listening on 127.0.0.1:9306 for mysql
DEBUG: Compiled-in value KNOWN_CREATE_SIZE (16) is less than measured (208).
DEBUG: Frame expression stack for creation is 208 (mocked, as no env MANTICORE_KNOWN_CREATE_SIZE=208 found)
DEBUG: Starting expression stack for creation is 3832 (mocked, as no env MANTICORE_START_KNOWN_CREATE_SIZE=3832 found)
DEBUG: Frame expression stack for eval/deletion is 32 (mocked, as no env MANTICORE_KNOWN_EXPR_SIZE=32 found)
DEBUG: Starting expression stack for eval/deletion is 104 (mocked, as no env MANTICORE_START_KNOWN_EXPR_SIZE=104 found)
DEBUG: Frame filter stack delta is 144 (mocked, as no env MANTICORE_KNOWN_FILTER_SIZE=144 found)
DEBUG: Starting filter stack delta is 11296 (mocked, as no env MANTICORE_START_KNOWN_FILTER_SIZE=11296 found)
DEBUG: Frame fulltext match stack delta is 336 (mocked, as no env MANTICORE_KNOWN_MATCH_SIZE=336 found)
DEBUG: Starting fulltext match stack delta is 14392 (mocked, as no env MANTICORE_START_KNOWN_MATCH_SIZE=14392 found)
DEBUG: ‘rt_flush_period’ - nothing specified, using default value 36000000000
DEBUG: ‘embedded_limit’ - nothing specified, using default value 16384
DEBUG: ‘attr_update_reserve’ - nothing specified, using default value 131072
DEBUG: ‘docstore_block_size’ - nothing specified, using default value 16384
precaching table ‘piler1’
DEBUG: ‘embedded_limit’ - nothing specified, using default value 16384
DEBUG: ‘attr_update_reserve’ - nothing specified, using default value 131072
DEBUG: ‘docstore_block_size’ - nothing specified, using default value 16384
precaching table ‘tag1’
DEBUG: ‘embedded_limit’ - nothing specified, using default value 16384
DEBUG: ‘attr_update_reserve’ - nothing specified, using default value 131072
DEBUG: ‘docstore_block_size’ - nothing specified, using default value 16384
precaching table ‘note1’
precached 3 tables in 5.415 sec
DEBUG: uid-short server_id 0, started 100000, seed 1677721600000
DEBUG: SaveMeta: Done (/var/piler/manticore/binlog.meta.new)
prereading 3 tables
DEBUG: prereading table ‘piler1’
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.812’(/var/piler/manticore/piler1.812)
accepting connections
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.814’(/var/piler/manticore/piler1.814)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.816’(/var/piler/manticore/piler1.816)
DEBUG: dlopen(libzstd.so.1)=0x7ffbe8000f40
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.818’(/var/piler/manticore/piler1.818)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.820’(/var/piler/manticore/piler1.820)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.822’(/var/piler/manticore/piler1.822)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.824’(/var/piler/manticore/piler1.824)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.826’(/var/piler/manticore/piler1.826)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.828’(/var/piler/manticore/piler1.828)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.830’(/var/piler/manticore/piler1.830)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.832’(/var/piler/manticore/piler1.832)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.842’(/var/piler/manticore/piler1.842)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.844’(/var/piler/manticore/piler1.844)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.846’(/var/piler/manticore/piler1.846)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.848’(/var/piler/manticore/piler1.848)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.850’(/var/piler/manticore/piler1.850)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.852’(/var/piler/manticore/piler1.852)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.854’(/var/piler/manticore/piler1.854)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.856’(/var/piler/manticore/piler1.856)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.858’(/var/piler/manticore/piler1.858)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.860’(/var/piler/manticore/piler1.860)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.862’(/var/piler/manticore/piler1.862)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.864’(/var/piler/manticore/piler1.864)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.866’(/var/piler/manticore/piler1.866)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.868’(/var/piler/manticore/piler1.868)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.870’(/var/piler/manticore/piler1.870)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.874’(/var/piler/manticore/piler1.874)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.876’(/var/piler/manticore/piler1.876)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.880’(/var/piler/manticore/piler1.880)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.882’(/var/piler/manticore/piler1.882)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.884’(/var/piler/manticore/piler1.884)
DEBUG: Preread successfully finished
DEBUG: CSphIndex_VLN::Preread invoked ‘/var/piler/manticore/piler1.886’(/var/piler/manticore/piler1.886)
DEBUG: Preread successfully finished
DEBUG: preread table ‘piler1’ in 113.977 sec
DEBUG: prereading table ‘tag1’
DEBUG: preread table ‘tag1’ in 0.000 sec
DEBUG: prereading table ‘note1’
DEBUG: preread table ‘note1’ in 0.000 sec
preread 3 tables in 113.977 sec
Thanks for your timely support.
