Manticore search crashed suddenly and won't restart with " binlog: log open error: failed to open /var/lib/manticore/data/binlog.001: No such file or directory" error

General Discussions - English
1

Hello,
We recently switched from sphinx to manticoresearch.
Of two servers with same settings, one manticorserach crashed after about 2 days, won’t restart with " binlog: log open error: failed to open /var/lib/manticore/data/binlog.001: No such file or directory" error.
I have checked the server, and the server has no errors in syslog, lots of free memory(is actually a server with 512 GB of RAM) and /var/lib/manticore/data exists and is writable.

What could cause this and what should I do?

Here is content of /var/lib/manticore/data
ls /var/lib/manticore/data -al
total 4
drwxr-xr-x 2 manticore manticore 56 Feb 12 00:46 .
drwxr-xr-x 4 manticore manticore 91 Feb 11 00:09 …
-rw------- 1 manticore manticore 0 Feb 12 00:46 binlog.lock
-rw------- 1 manticore manticore 11 Feb 11 04:00 binlog.meta

Here is log file:
thd 94 (work_188), proto sphinx, state query, command search
thd 95 (work_192), proto sphinx, state query, command search
thd 96 (work_194), proto sphinx, state query, command search
thd 97 (work_195), proto sphinx, state query, command search
thd 98 (work_198), proto sphinx, state query, command search
thd 99 (work_199), proto sphinx, state query, command search
— Totally 101 threads, and 100 client-working threads —
------- CRASH DUMP END -------
[Sun Feb 12 00:15:47.817 2023] [2942007] watchdog: main process 2942008 crashed via CRASH_EXIT (exit code 2), will be restarted
[Sun Feb 12 00:15:47.818 2023] [2942007] watchdog: main process 3137437 forked ok
[Sun Feb 12 00:15:47.843 2023] [3137437] starting daemon version ‘6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)’ …
[Sun Feb 12 00:15:47.843 2023] [3137437] listening on [REDACTED]:9312 for sphinx and http(s)
[Sun Feb 12 00:15:47.843 2023] [3137437] listening on [REDACTED]:3307 for mysql
[Sun Feb 12 00:15:47.844 2023] [3137438] WARNING: table ‘template’: key ‘path’ not found - NOT SERVING
[Sun Feb 12 00:15:50.390 2023] [3137440] binlog: replaying log /var/lib/manticore/data/binlog.001
[Sun Feb 12 00:15:50.390 2023] [3137440] FATAL: binlog: log open error: failed to open /var/lib/manticore/data/binlog.001: No such file or directory
[Sun Feb 12 00:15:50.433 2023] [2942007] watchdog: main process 3137437 exited cleanly (exit code 1), shutting down
[Sun Feb 12 00:15:50.820 2023] [3137648] watchdog: main process 3137649 forked ok
[Sun Feb 12 00:15:50.838 2023] [3137649] starting daemon version ‘6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)’ …
[Sun Feb 12 00:15:50.838 2023] [3137649] listening on [REDACTED]:9312 for sphinx and http(s)
[Sun Feb 12 00:15:50.838 2023] [3137649] listening on [REDACTED]:3307 for mysql
[Sun Feb 12 00:15:50.838 2023] [3137650] WARNING: table ‘template’: key ‘path’ not found - NOT SERVING
[Sun Feb 12 00:15:53.383 2023] [3137652] binlog: replaying log /var/lib/manticore/data/binlog.001
[Sun Feb 12 00:15:53.383 2023] [3137652] FATAL: binlog: log open error: failed to open /var/lib/manticore/data/binlog.001: No such file or directory
[Sun Feb 12 00:15:53.422 2023] [3137648] watchdog: main process 3137649 exited cleanly (exit code 1), shutting down
[Sun Feb 12 00:46:02.797 2023] [3142609] watchdog: main process 3142610 forked ok
[Sun Feb 12 00:46:02.816 2023] [3142610] starting daemon version ‘6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)’ …
[Sun Feb 12 00:46:02.816 2023] [3142610] listening on [REDACTED]:9312 for sphinx and http(s)
[Sun Feb 12 00:46:02.816 2023] [3142610] listening on [REDACTED]:3307 for mysql
[Sun Feb 12 00:46:02.816 2023] [3142611] WARNING: table ‘template’: key ‘path’ not found - NOT SERVING
[Sun Feb 12 00:46:03.716 2023] [3142611] rotating table ‘[REDACTED]’: success
[Sun Feb 12 00:46:05.188 2023] [3142611] rotating table ‘[REDACTED]’: success
[Sun Feb 12 00:46:05.220 2023] [3142611] rotating table ‘[REDACTED]’: success
[Sun Feb 12 00:46:05.384 2023] [3142611] rotating table ‘[REDACTED]’: success
[Sun Feb 12 00:46:05.387 2023] [3142613] binlog: replaying log /var/lib/manticore/data/binlog.001
[Sun Feb 12 00:46:05.387 2023] [3142613] FATAL: binlog: log open error: failed to open /var/lib/manticore/data/binlog.001: No such file or directory
[Sun Feb 12 00:46:05.426 2023] [3142609] watchdog: main process 3142610 exited cleanly (exit code 1), shutting down

Additional information:
#rpm -qa|grep manti
manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64
manticore-devel-6.0.2_230210.89c7a5139-1.el8.noarch
manticore-executor-0.6.2_23012605.d95e43e-1.x86_64
manticore-tools-6.0.2_230210.89c7a5139-1.el8.x86_64
manticore-repo-0.1.0-1.noarch
manticore-columnar-lib-2.0.0_230130.a7c703d-1.el8.x86_64
manticore-common-6.0.2_230210.89c7a5139-1.el8.noarch
manticore-extra-0.6.2_23012604.d95e43e-1.noarch
manticore-icudata-5.0.3_221123.d2d9e5e56-1.el7.centos.noarch
manticore-backup-0.5.2_23020607.4a37932-1.noarch
manticore-6.0.2_230210.89c7a5139-1.el8.x86_64
manticore-buddy-0.3.4_23020607.4798582-1.noarch
manticore-server-core-6.0.2_230210.89c7a5139-1.el8.x86_64

# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.7 (Ootpa)

2

I tried deleting the /var/lib/manticore/data files, and it still won’t start.
So I tried uninstalling manticoresearch, and it seems something is broken and won’t even uninstall.
Of note, this server had no problems running sphinx for a long long time and is currently very stable for other usage.

# yum remove manticore*
Updating Subscription Management repositories.

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

Dependencies resolved.
=======================================================================================================================================================================================================================================
 Package                                                    Architecture                               Version                                                                 Repository                                         Size
=======================================================================================================================================================================================================================================
Removing:
 manticore                                                  x86_64                                     6.0.2_230210.89c7a5139-1.el8                                            \@manticore                                          0  
 manticore-backup                                           noarch                                     0.5.2_23020607.4a37932-1                                                \@manticore                                        515 k
 manticore-buddy                                            noarch                                     0.3.4_23020607.4798582-1                                                \@manticore                                        1.7 M
 manticore-columnar-lib                                     x86_64                                     2.0.0_230130.a7c703d-1.el8                                              \@manticore                                        6.8 M
 manticore-common                                           noarch                                     6.0.2_230210.89c7a5139-1.el8                                            \@manticore                                         77 k
 manticore-devel                                            noarch                                     6.0.2_230210.89c7a5139-1.el8                                            \@manticore                                        9.5 k
 manticore-executor                                         x86_64                                     0.6.2_23012605.d95e43e-1                                                \@manticore                                         18 M
 manticore-extra                                            noarch                                     0.6.2_23012604.d95e43e-1                                                \@manticore                                        3.1 k
 manticore-icudata                                          noarch                                     5.0.3_221123.d2d9e5e56-1.el7.centos                                     \@manticore                                         27 M
 manticore-repo                                             noarch                                     0.1.0-1                                                                 \@\@commandline                                     768  
 manticore-server                                           x86_64                                     6.0.2_230210.89c7a5139-1.el8                                            \@manticore                                        3.3 k
 manticore-server-core                                      x86_64                                     6.0.2_230210.89c7a5139-1.el8                                            \@manticore                                         20 M
 manticore-tools                                            x86_64                                     6.0.2_230210.89c7a5139-1.el8                                            \@manticore                                         61 M

Transaction Summary
=======================================================================================================================================================================================================================================
Remove  13 Packages

Freed space: 134 M
Is this ok [y/N]: y
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                               1/1 
  Running scriptlet: manticore-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                                 1/1 
  Erasing          : manticore-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                                1/13 
  Erasing          : manticore-extra-0.6.2_23012604.d95e43e-1.noarch                                                                                                                                                              2/13 
  Running scriptlet: manticore-extra-0.6.2_23012604.d95e43e-1.noarch                                                                                                                                                              2/13 
  Running scriptlet: manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                         3/13 
/var/tmp/rpm-tmp.8LQ4Jw: line 1: fg: no job control
error: %preun(manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64) scriptlet failed, exit status 1

Error in PREUN scriptlet in rpm package manticore-server
  Erasing          : manticore-executor-0.6.2_23012605.d95e43e-1.x86_64                                                                                                                                                           4/13 
error: manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64: erase failed

  Erasing          : manticore-buddy-0.3.4_23020607.4798582-1.noarch                                                                                                                                                              5/13 
  Running scriptlet: manticore-buddy-0.3.4_23020607.4798582-1.noarch                                                                                                                                                              5/13 
  Erasing          : manticore-devel-6.0.2_230210.89c7a5139-1.el8.noarch                                                                                                                                                          6/13 
  Erasing          : manticore-icudata-5.0.3_221123.d2d9e5e56-1.el7.centos.noarch                                                                                                                                                 7/13 
  Erasing          : manticore-repo-0.1.0-1.noarch                                                                                                                                                                                8/13 
  Erasing          : manticore-tools-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                          9/13 
  Erasing          : manticore-server-core-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                   10/13 
  Erasing          : manticore-common-6.0.2_230210.89c7a5139-1.el8.noarch                                                                                                                                                        11/13 
warning: /etc/manticoresearch/manticore.conf saved as /etc/manticoresearch/manticore.conf.rpmsave

  Erasing          : manticore-backup-0.5.2_23020607.4a37932-1.noarch                                                                                                                                                            12/13 
  Running scriptlet: manticore-backup-0.5.2_23020607.4a37932-1.noarch                                                                                                                                                            12/13 
  Erasing          : manticore-columnar-lib-2.0.0_230130.a7c703d-1.el8.x86_64                                                                                                                                                    13/13 
  Running scriptlet: manticore-columnar-lib-2.0.0_230130.a7c703d-1.el8.x86_64                                                                                                                                                    13/13 
  Verifying        : manticore-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                                1/13 
  Verifying        : manticore-backup-0.5.2_23020607.4a37932-1.noarch                                                                                                                                                             2/13 
  Verifying        : manticore-buddy-0.3.4_23020607.4798582-1.noarch                                                                                                                                                              3/13 
  Verifying        : manticore-columnar-lib-2.0.0_230130.a7c703d-1.el8.x86_64                                                                                                                                                     4/13 
  Verifying        : manticore-common-6.0.2_230210.89c7a5139-1.el8.noarch                                                                                                                                                         5/13 
  Verifying        : manticore-devel-6.0.2_230210.89c7a5139-1.el8.noarch                                                                                                                                                          6/13 
  Verifying        : manticore-executor-0.6.2_23012605.d95e43e-1.x86_64                                                                                                                                                           7/13 
  Verifying        : manticore-extra-0.6.2_23012604.d95e43e-1.noarch                                                                                                                                                              8/13 
  Verifying        : manticore-icudata-5.0.3_221123.d2d9e5e56-1.el7.centos.noarch                                                                                                                                                 9/13 
  Verifying        : manticore-repo-0.1.0-1.noarch                                                                                                                                                                               10/13 
  Verifying        : manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                        11/13 
  Verifying        : manticore-server-core-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                   12/13 
  Verifying        : manticore-tools-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                         13/13 
Installed products updated.

Removed:
  manticore-6.0.2_230210.89c7a5139-1.el8.x86_64                manticore-backup-0.5.2_23020607.4a37932-1.noarch    manticore-buddy-0.3.4_23020607.4798582-1.noarch           manticore-columnar-lib-2.0.0_230130.a7c703d-1.el8.x86_64
  manticore-common-6.0.2_230210.89c7a5139-1.el8.noarch         manticore-devel-6.0.2_230210.89c7a5139-1.el8.noarch manticore-executor-0.6.2_23012605.d95e43e-1.x86_64        manticore-extra-0.6.2_23012604.d95e43e-1.noarch         
  manticore-icudata-5.0.3_221123.d2d9e5e56-1.el7.centos.noarch manticore-repo-0.1.0-1.noarch                       manticore-server-core-6.0.2_230210.89c7a5139-1.el8.x86_64 manticore-tools-6.0.2_230210.89c7a5139-1.el8.x86_64     
Failed:
  manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                                                 

Error: Transaction failed


# yum remove manticore*
Updating Subscription Management repositories.

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

Dependencies resolved.
=======================================================================================================================================================================================================================================
 Package                                                  Architecture                                   Version                                                                Repository                                        Size
=======================================================================================================================================================================================================================================
Removing:
 manticore-server                                         x86_64                                         6.0.2_230210.89c7a5139-1.el8                                           \@\@System                                         3.3 k

Transaction Summary
=======================================================================================================================================================================================================================================
Remove  1 Package

Freed space: 3.3 k
Is this ok [y/N]: y
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                               1/1 
  Running scriptlet: manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                          1/1 
/var/tmp/rpm-tmp.rpzuDs: line 1: fg: no job control
error: %preun(manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64) scriptlet failed, exit status 1

Error in PREUN scriptlet in rpm package manticore-server
  Verifying        : manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                          1/1 
Installed products updated.

Failed:
  manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                                                 

Error: Transaction failed
3

After doing yum reinstall manticore-server, the server does start, but it seems it breaks yum database, anyone know to fix this mess :frowning: ?

[root\@f89 data]# rpm --rebuilddb
[root\@f89 data]# yum remove manticore-server
Updating Subscription Management repositories.

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

Dependencies resolved.
=======================================================================================================================================================================================================================================
 Package                                                    Architecture                                Version                                                                  Repository                                       Size
=======================================================================================================================================================================================================================================
Removing:
 manticore-server                                           x86_64                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                      3.3 k
Removing dependent packages:
 manticore                                                  x86_64                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                        0  
Removing unused dependencies:
 manticore-backup                                           noarch                                      0.5.2_23020607.4a37932-1                                                 \@manticore                                      515 k
 manticore-buddy                                            noarch                                      0.3.4_23020607.4798582-1                                                 \@manticore                                      1.7 M
 manticore-common                                           noarch                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                       77 k
 manticore-devel                                            noarch                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                      9.5 k
 manticore-icudata                                          noarch                                      5.0.3_221123.d2d9e5e56-1.el7.centos                                      \@manticore                                       27 M
 manticore-server-core                                      x86_64                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                       20 M
 manticore-tools                                            x86_64                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                       61 M

Transaction Summary
=======================================================================================================================================================================================================================================
Remove  9 Packages

Freed space: 110 M
Is this ok [y/N]: y
Running transaction check
Error: transaction check vs depsolve:
manticore-server-core = 6.0.2_230210.89c7a5139-1.el8 is needed by (installed) manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64
To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'.
You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue.
[root\@f89 data]# yum reinstall manticore-server
Updating Subscription Management repositories.

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

Last metadata expiration check: 0:04:58 ago on Sun 12 Feb 2023 01:13:43 AM KST.
Dependencies resolved.
=======================================================================================================================================================================================================================================
 Package                                                  Architecture                                   Version                                                               Repository                                         Size
=======================================================================================================================================================================================================================================
Reinstalling:
 manticore-server                                         x86_64                                         6.0.2_230210.89c7a5139-1.el8                                          manticore                                         9.3 k
     replacing  manticore-server.x86_64 6.0.2_230210.89c7a5139-1.el8

Transaction Summary
=======================================================================================================================================================================================================================================

Total download size: 9.3 k
Is this ok [y/N]: y
Downloading Packages:
manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64.rpm                                                                                                                                               6.9 MB/s | 9.3 kB     00:00    
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                  931 kB/s | 9.3 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                               1/1 
  Running scriptlet: manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                          1/1 
  Reinstalling     : manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                          1/3 
  Running scriptlet: manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                          1/3 
/var/tmp/rpm-tmp.0WCdOS: line 5: fg: no job control
To start Manticore Search service:
  > systemctl start manticore


  Running scriptlet: manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                          2/3 
/var/tmp/rpm-tmp.0wlKlO: line 1: fg: no job control
error: %preun(manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64) scriptlet failed, exit status 1

Error in PREUN scriptlet in rpm package manticore-server
  Running scriptlet: manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                          3/3 
error: manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64: erase failed
/var/tmp/rpm-tmp.NHlSJj: line 1: fg: no job control
error: %preun(manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64) scriptlet failed, exit status 1

Error in PREUN scriptlet in rpm package manticore-server
error: manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64: erase failed

[/usr/lib/tmpfiles.d/searchd.conf:1] Line references path below legacy directory /var/run/, updating /var/run/manticore → /run/manticore; please update the tmpfiles.d/ drop-in file accordingly.

  Verifying        : manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                          1/3 
  Verifying        : manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                          2/3 
  Verifying        : manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                          3/3 
Installed products updated.

Reinstalled:
  manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                                                                                                                                                 
Failed:
  manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                               manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64                                                              

Error: Transaction failed
[root\@f89 data]# yum remove manticore-server
Updating Subscription Management repositories.

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

Dependencies resolved.
=======================================================================================================================================================================================================================================
 Package                                                    Architecture                                Version                                                                  Repository                                       Size
=======================================================================================================================================================================================================================================
Removing:
 manticore-server                                           x86_64                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                      3.3 k
Removing dependent packages:
 manticore                                                  x86_64                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                        0  
Removing unused dependencies:
 manticore-backup                                           noarch                                      0.5.2_23020607.4a37932-1                                                 \@manticore                                      515 k
 manticore-buddy                                            noarch                                      0.3.4_23020607.4798582-1                                                 \@manticore                                      1.7 M
 manticore-common                                           noarch                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                       77 k
 manticore-devel                                            noarch                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                      9.5 k
 manticore-icudata                                          noarch                                      5.0.3_221123.d2d9e5e56-1.el7.centos                                      \@manticore                                       27 M
 manticore-server-core                                      x86_64                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                       20 M
 manticore-tools                                            x86_64                                      6.0.2_230210.89c7a5139-1.el8                                             \@manticore                                       61 M

Transaction Summary
=======================================================================================================================================================================================================================================
Remove  9 Packages

Freed space: 110 M
Is this ok [y/N]: y
Running transaction check
Error: transaction check vs depsolve:
manticore-server-core = 6.0.2_230210.89c7a5139-1.el8 is needed by (installed) manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64
manticore-server-core = 6.0.2_230210.89c7a5139-1.el8 is needed by (installed) manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64
To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'.
You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue.
4

Hopefully the bugs will be fixed in manticore itself, but just in case leaving a comment on a workaround for the above bugs to help others.

Bugs we found:
Server won’t start after sudden crashing(unknown reason) with the following error in logs(even after deleting binlog* files):
binlog: log open error: failed to open /var/lib/manticore/data/binlog.001: No such file or directory" error.

also manticore-server doesn’t get deleted using dnf remove due to the following error:
Error in PREUN scriptlet in rpm package manticore-server

Workaround for the above bugs:
delete /var/lib/manticore/data/binlog*
remove and reinstall manicore, but to remove manticore-server use if dnf fails
rpm --erase --nodeps --allmatches manticore-server --noscripts
then manticore server will restart

5 
[Sun Feb 12 00:46:05.387 2023] [3142613] FATAL: binlog: log open error: failed to open /var/lib/manticore/data/binlog.001: No such file or directory

means there’s a reference to this file in binlog.meta, but the file itself doesn’t exist.

After doing yum reinstall manticore-server, the server does start, but it seems it breaks yum database, anyone know to fix this mess :frowning: ?

yum remove --setopt=tsflags=noscripts manticore* should help.

Server won’t start after sudden crashing(unknown reason) with the following error in logs(even after deleting binlog* files):

Can you share more from the searchd log? A few records before the crash, the whole crash log and a few records after it.

also manticore-server doesn’t get deleted using dnf remove due to the following error:
Error in PREUN scriptlet in rpm package manticore-server

Good catch. We’ll look into it.

6

Regarding:
means there’s a reference to this file in binlog.meta , but the file itself doesn’t exist.

I actually did that, and it still didn’t work.
Of note, although I might have made some mistake, I doubt I installed manticore incorrectly as I installed on multiple servers with same commands and with a very simple step by step follow of install instructions.

Regarding:
yum remove --setopt=tsflags=noscripts manticore

That doesn’t work in above case because of reinstall attempt after breakage.
The following error

Error: transaction check vs depsolve:
manticore-server-core = 6.0.2_230210.89c7a5139-1.el8 is needed by (installed) manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64
manticore-server-core = 6.0.2_230210.89c7a5139-1.el8 is needed by (installed) manticore-server-6.0.2_230210.89c7a5139-1.el8.x86_64
To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'.

requires “–allmatches” option.
also to allow other manticore files to uninstall with proper scripts I added --nodeps to target the problem rpm only. as it would be a bad idea to uninstall all manticore rpm with noscripts option.

Regarding
Can you share more from the searchd log? A few records before the crash, the whole crash log and a few records after it.

If you are part of the manticore team, please tell me your email.
As we wish to keep our log data private.

Thank you.
Have a nice day!

7

sergey@manticoresearch.com

Also feel free to file up an issue on GitHub - GitHub - manticoresoftware/manticoresearch: Easy to use open source fast database for search | Good alternative to Elasticsearch now | Drop-in replacement for E in the ELK soon and use our write-only S3 to share your log https://manual.manticoresearch.com/Reporting_bugs#Uploading-your-data

8

Again thank you for your responses.

sergey@manticoresearch.com

Thank you.
I have sent our related searchd and query logs to your email.

Of note, we have now turned off manticore binlog and setup monit (open source tool) to monitor manticore and restart it automatically if it is not running.