I have a dedicated Manticore Server and a webserver. On the Manticore Server, the index are showing fine but on the other servers, no index is found even though the connection has been established. I even added the local IP (10.142.0.4) for searchd to listen.
Update : Remote servers can now see the index.
Steps made :
Stop manticore using any of these commands:
service manticore stop
systemctl stop manticore
searchd --stop or --stopwait
Wait for a few seconds or a minute before starting manticore using any of these commands:
service manticore start
systemctl start manticore
searchd --console
Usually during start process I encounter the error “service start request repeated too quickly, refusing to start”. If I encounter this and usually it fails to start I do any of the following:
delete binlog files and search.pid.
force skill searchd (sudo pkill -f httpd)
Then try to restart manticore/searchd again.
Unknowingly after a couple of start/stop the indexes showed on the remote servers. I did not even bother to reindex or rotate them again.
Yes, I have noticed same behavior many times as well. Searchd STOP should not exit until it has actually stopped all background work and is ready to quit. My understanding is that watchdog process should issue a stop signal to main searchd daemon and only when main searchd daemon says it has finished and ready to exit - then watchdog exits as well.
Same on startup - you can start it , it may take a minute to replay binlogs but you do not know it as command line has been executed and if that that time you issue command to STOP - it will not stop normally, it will actually create a mess as you are trying to stop a process in the middle of binlog replay.
And systemctl does not know if process has actually started in full or stopped in full for those reasons allowing you to try restarting or stopping when you should not.
Searchd daemon startup/binlog replay/binlog saving/exiting process should be reviewed to be more informative to the command line entries and to the systemctl.
Make sure you don’t have two instances running on the same port. Manticore doesn’t prohibit doing it as soon as you have different pid files. In this case your mysql client may be connecting not to the instance you are expecting.
searchd --console
Make sure you run under user manticore, otherwise there may be a permission issue since systemctl runs under manticore, meaning all the indexes, binlog files etc are also created under user manticore and can’t be read when created under another user (e.g. root).
Usually during start process I encounter the error “service start request repeated too quickly, refusing to start”.
To debug it further go through your steps checking what’s in the searchd log after each. Then you should understand what’s going wrong.
Searchd STOP should not exit until it has actually stopped all background work and is ready to quit.
What do you mean by “Searchd STOP”?
Same on startup - you can start it , it may take a minute to replay binlogs but you do not know it as command line has been executed and if that that time you issue command to STOP - it will not stop normally, it will actually create a mess as you are trying to stop a process in the middle of binlog replay.
If you can make a clear reproducible test case please do it and file a bug report on github. We don’t want any mess even when you try to stop an instance which is being started yet, just need to understand how exactly this can be reproduced, so we can think how to fix it.
