invalid field id: not in doclist mask from indextool after upgrade

General Discussions - English
1

I’m testing an update from the rather old 3.4.2 searchd to current 6.2.12. On the old version, indextool --check passes cleanly. After a clean stop + copy + start to get a version on 6.2.12, search starts up and handles most queries successfully but occasionally crashes and core dumps.

Running indextool --check from the new version produces many failures like:
FAILED, invalid field id: not in doclist mask (segment=0, word=0, wordid=0(0), rowid=0, field_id=39, field_mask=0)
(The final line of output reports check FAILED, 99 of 17786436 failures reported, 495.2 sec elapsed)

There is a single rt index and no clustering configured.

From the breaking changes referenced in the changelog, columnar storage and secondary indices are not in use. Config additions of stored_fields = and pseudo_sharding = 0 were made.

What does this indextool error indicate? Is it likely related to the crashes?

2

What does this indextool error indicate? Is it likely related to the crashes?

It indicates the table is corrupted and it’s very likely related to the crashes.

3

Do you recognise it as relating to a particular missed changelog item / migration step / config change? Following the same process on a bunch of instances yields the same result, so it doesn’t seem like random or disk level corruption.

4

Not sure, but just an idea: try FLUSH RAMCHUNK in 3.4.2 before running indextool 6.2.12 against it.

5

A very quick test shows encouraging results from FLUSH RAMCHUNK, thanks. A small index that was showing the error did now migrate and indexcheck cleanly. Will need to test some more to confirm whether this was the whole issue.