Kill List suppress or Delete target IDs

General Discussions - English
1

Manticore Search Manual: Data creation and modification > Adding data from external storages > Adding data to tables > Killlist in plain tables and every other page referring to Killlist is always saying suppress IDs in the main index. Delta carries kill list IDs, passes onto main index and main index suppresses them.
The bigger the kill list - the more “Hidden” or “Suppressed” IDs we have in the main index which actually makes index greater (and slower in turn).

Wouldn’t it be logical to actually DELETE the IDs from main index if it is set this way:

killlist_target = main:kl

Or is it actually deleting IDs but documentation is a little outdated?

If it is not deleting IDs then I am forced to either keep my Kill list in the database or do full reindex daily upon kill list deletion from database to avoid IDs that were suppressed in main index to pop up in searches.

2

Kill-list has fixed size, that is raw bitmap covering all the documents. So, despite number of bits you set there it will not be slower or faster because of it.
Apart bitmap we also store num of killed as a number - just to have faster access to them. That is only thing we can change in-place, as other structures are much more complex.
Actual deletion (wiping killed) performed now on merging; simple ‘delete’ instead of suppress is too resource-consuming (actually that is rebuilding whole index ‘from scratch’, no inplace changes possible).

3

It was not mentioned in the docs that Merging Main+Delta will wipe out the kill list IDs from main index. Must be a new useful undocumented feature. I was running:

/usr/local/manticore/usr/bin/indexer --config /usr/local/manticore/etc/manticoresearch/manticore.conf’ --merge jobs_flexible_newest jobs_flexible_delta --merge-dst-range deleted 0 0 --rotate

and this brings the next question:

On Manticore Search Manual: Data creation and modification > Adding data from external storages > Adding data to tables > Merging tables we have filters. They are useful only in corner cases only where someone wants to remove certain type of documents from main index. And that feature can stay but in a case of a simple kill list use - a switch:

merge-dst-range

is not needed anymore in my case if --merge does all the kill list IDs wiping from main index. Am I correct with this assumption?

Kill list might be compact in fixed size on Manticore but I have to maintain in on my MySQL database until Manticore wipes out the kill list items from main index. And that Kill List on my database can grow very big.

To keep it small I had to do a FULL REINDEX every night, and empty that MySQL table then. As in the past MERGE was only suppressing the Kill List items. And what was happening was this:

Kill list items in MySQL= 1, 000, 0000
Documents in Main Index = 3, 000, 000
Displayed Items in search = 2,000,000

If I were to do this in the past:

Empty MySQL table
Run Delta index (say Delta finds 1 new document)
MERGE Main+Delta indexes
I used to get this:

Documents in Main Index = 3, 000, 001
Displayed Items in search = 3,000,001

But 1,000,000 items that are actually not available on the website are still being referenced and searched because they were not being suppressed by Kill List anymore.

The only fix was to RUN FULL REINDEX.

But if you say that MERGE does all the wiping from main index now - It saves me from doing FULL REINDEX every night before I empty MySQL Kill List table. Which is a great thing if that is the case. And Documentation should say that as that is a huge advantage if comparing to Sphinx.

4

After some testing I see Manticore does actually Wipe out Kill list IDs from main index after Main+Delta merge. Which is Great undocumented feature. And that makes Manticore more resource friendly than Sphinx. And this feature allows me to clear Kill List from MySQL after each merge not once a day and it also frees me up from making FULL REINDEX after emptying Kill List in MySQL.

Is there an option to use in config to stop receiving this warning after each delta reindex?

WARNING: killlist_target is specified but kill-list is empty

Since delta reindex runs often with a cron job - unnecessary warnings in email become nuisance.

5

Can’t say anything about current sphinx (3.x), but in sphinx 2.x killlists similarly applied on merge. Idea is having big main + relatively small delta operating as single solid index. So, killed/replaced document just suppressed in output using kill-list, and it finally wiped out on merge.
Difference is in moment of applying: on sphinx killlist is part of delta index and applied on each search, making delta index ‘binded’ to main. On manticore it is persistently applied to main index on startup and became it’s part, so that it is in effect despite of further presence of delta. On rotation of delta new kill-list will be applied in addition, and so on. On merge killed documents wiped out and killlist is zeroed (but still exists). There is no way to ‘unapply’ kill-list, however technically it is possible (if you zero kill-list bitmap from outside - no documents will be then suppressed)

6

Thank you. Would be good to have the information from this topic added to the documentation as it needs those clarifications.

Sphinx 3 Is never wiping Kill List items on Merge from main index which is creating inconveniences I mentioned earlier.

Is there an option to use in Manticore config to stop receiving this warning after each delta reindex (as I empty Kill list in MySQL after each MERGE and there are no new deleted times for some time)?

WARNING: killlist_target is specified but kill-list is empty

Since delta reindex runs often with a cron job - unnecessary warnings in email become nuisance.

7

No, there’s no setting for it now.

8

I’m disabling emails/output from Delta indexer cron job for now as I will miss any error/warning email anyway among killlist_target warnings which comes every 4 minutes (as cron runs). Will reenable it if such setting becomes available in future.

9

Instead you can some indexer|grep -v "kill-list is empty"|egrep "WARNING|ERROR" && mail ... instead. Just not to miss a real warning/error.

10

In my case the best solution is giving a “fake” non empty kill list when it becomes empty which works well:

#Add a fake id on the killlist to have “something” on the killlist and not get the warning about empty kill list when nothing has been deleted recently and kill list is empty:
sql_query_killlist = SELECT my_id FROM my_kill_list UNION SELECT 1