I’m planning to migrate from Sphinx 2.11 to Manticore 3. Due to the change in kill-list concept, I’m trying to develop new scheme for live index update. RT indices have not been considered yet and I want to avoid raising the issue in this discussion in order to keep focus. Currently I stick to the
delta scheme and index merging. I want to tell about my vision of this scheme in the hope of getting opinions and, possibly, adjusting it.
First, I will mention facts that was verified experimentally and are relevant to this case. However, maybe I’m missing something:
- The documentation about index merging says:
“Note, however, that the “old” keywords will not be automatically removed in such cases. For example, if there’s a keyword “old” associated with document 123 in DSTINDEX, and a keyword “new” associated with it in SRCINDEX, document 123 will be found by both keywords after the merge”.
However I’m not able to reproduce this behavior. After the merge I can only find a document by new words if the keywords associated with the document have changed. Am I doing something wrong?
- SRCINDEX kill-list is applied to DSTINDEX during index merge. This can be used to exclude deleted documents instead of using the
--merge-dst-rangeoption. But I’m not sure if there is a difference between these approaches in terms of final result. Do we get identical indices in both cases?
As for the index update scheme, we want to achieve the following:
deltaindex to update data and merge it to
- Exclude deleted documents using kill-list
- Be able to reindex
deltawithout applying kill-list. We want to apply it only at the merge stage
To achieve the described goals I came up with the option of using two configuration files that differ only in the composition of the indices:
- A configuration file that includes only the
mainindex. This file is used to run searchd and allows to avoid
- A configuration file that includes
deltaindices. This file is used to run indexer. Since
deltaindex is not locked by searchd, we can update it without rotation (
--rotateoption) and to apply kill-list only at the merge stage. We are doing merge with rotation, so searchd could pick-up the changed
I suppose this scheme is a bit tricky, but if we do not consider RT indexes, at the moment I do not see any other way to implement live index update. What do you think about this approach?
Thanks in advance.