I dumped data from one node.
… deleted the whole deployment (deleted the helm release, as well as the pv claims)
Recreated the cluster with helm
… then imported the table back into one node.
Data is now identical on all nodes (ie test KNN queries return same results). So dont know how to reproduce it at the moment!
Only difference I can see is this was all imported in one go (piping the dump via mysql client)
whereas the first time, my own PHP injected the rows ‘piecemeal’. (inserted 1000 rows, runs some tests, then inserted some more etc. Each set might of gone to different worker nodes, which seems to have replicated the data fine (all rows present), while the KNN index still ended up different)
Suppose will have to try doing that again. But will be harder to create as a reproducible example!
Will have to log the queries as go.
Edit: Just to confirm, I’ve now recreated the whole index, with my original PHP code. And the resultant index seems fine. KNN queries return same (good) results on all nodes.
… what can’t guarantee that inserted into exactly the same nodes in the same sequence as original. possible avoided some sort race condition, orgiinally writtne to nodes, while still replicating.
So for now will write it off a as one-off corruption.