New killlist strategy in Manticore 3


#1

Hi guys!

I am updating my Manticore from 2.4.1 to 3.1.2. and i have a problems with the using of new features of killlists.

I have three indexes: delta, main and archive. I am using tsv to update them and killists to keep data updated. For example, killist of delta contained all data in delta and removed data and it affected main and archive in the distributed index search. I have a frequent update of a data and it’s very important for me.

Now if i rotate delta and then main, all data from delta will be permanetly deleted from main, until i rotate delta. And main contains no data and if i rotate delta with max id from main i will lose my data with previous strategy.

Whats the new way to keep such mechanism working?

Thanks in advance!


#2

not quite clear do you merge you delta into main or just reindex main from scratch and reset delta


#3

Yes, i am rebuilding main from scratch and reseting delta, but my old delta (not empty) can suppress new main


#4

could you rotate your reset delta then main index?

This way you set empty killist to apply by delta then loads main without suppress by delta index


#5

No, we cannot, because rotating main or archive takes a lot of time (up to 3 days).


#6

you could also issue indexing as --rotate main delta this way daemon checks dependencies and rotate indexes properly on its own


#7

could you also try to use RT index instead of delta - main schema?

You could attach main plain index to RT index and reset all collected data then push new \ delta data into that RT index on top of existed data.

Then in case many disk chunks collected issue OPTIMIZE command for that RT index or reindex main from scratch with indexer and attach it again into RT index


#9

not quite sure why you can not rotate empty delta along with you large main index?
Why --rotate main delta does not work for you?


#10

Other way as I suggested is to use RT index there is no delta - main issues.


#11

Thanks, now its more clear for me!


#12

And one more question. Does killlist_target work in the rt index to suppress removed data in the big archive index? And what do you mean on attach main to RT?


#13

Does killlist_target work in the rt index to suppress removed data

killlist_target got applied to any kind of indexes either to plain or RT

However for RT index you could remove document from RT index itself via DELETE statement without managing indirect killists.

what do you mean on attach main to RT?

You could attach plain index to RT index with ATTACH statement that allows you to faster indexing your data into plain index with indexer (in case you has hug amount of data and want to index it fastest possible way) then rotate that plain index into daemon then issue statement

ATTACH INDEX plain1 TO RTINDEX rt1

as described here https://manticoresearch.gitlab.io/dev/sphinxql_reference/attach_index_syntax.html

This way you might reindex your data from scratch in case it convenient for you then ATTACH that reindex data to existed RT index. And after that apply all new data not to the delta index (as you do now) but directly to that RT index.


#14

My bad. If i set

killlist_target = archive_idx

in the RT index configuration, where archive_idx is usual plain index, how will it work? Will killlist data be updated if i insert or delete value in rt?


#15

kill list at index is valid only to plain index as kill list got obtained via sql_query_killlist

so killlist_target even valid for RT index is actually empty and will not suppress any data in target indexes.

But either RT and plain indexes might be target for killlist_target on any plain index.