Auto UUID creation seems improper? Existing docs getting overwritten even with insert operation

General Discussions - English
1

When adding new documents into a table and using the auto UUID feature, manticore seems to be generating UUIDs that already exist in the table and is therefore rewriting documents.
I am unable to give an exact situation and MRE but I am noticing this in some of my tables.

For example, I wrote a list of all ids in a table named gnanetra into a txt file first. There are around 47050 ids. Here is a sample of those:

1677721603112, 1677721600184, 1677721603113, 1677721600454

There are many more, but I have included 1677721600184 from the list because when I insert a new doc as below, the new doc gets 1677721600184 assigned as its id, and the original doc is replaced, though I have done an insert.
Also the id seems to start at 1677721600000 for any new index (irrespective of when the server started) and on different machines as well, which is a bit odd as I thought UUID uses the timestamp of server start as part of its string.

1677721600000 is the timestamp of 2 March 2023, which is pretty close to the release of version 6.0.2(??)

resp = indexApi.insert({"index" : "gnanetra", "id":0, "doc" : doc})
print(resp)

Prints out:

{'created': True,
 'found': None,
 'id': 1677721600184,
 'index': 'gnanetra',
 'result': 'created'}

This happens with or without forcing id to 0. Python client version is 4.0

Manticore Search Version:

6.3.0

Operating System Version:

Windows 11 & Windows Server 2013

1 Like
2

Could it be that for those documents you didn’t use implicitly specified the ids?

Also the id seems to start at 1677721600000 for any new index

Pls confirm that if you do the below, you get the same “id” each time:

mysql> drop table if exists t; create table t; insert into t values(0); select * from t;
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.01 sec)

Query OK, 1 row affected (0.00 sec)

+---------------------+
| id                  |
+---------------------+
| 1515939651171385658 |
+---------------------+
1 row in set (0.00 sec)
--- 1 out of 1 results in 0ms ---

mysql> drop table if exists t; create table t; insert into t values(0); select * from t;
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.01 sec)

Query OK, 1 row affected (0.00 sec)

+---------------------+
| id                  |
+---------------------+
| 1515939651171385659 |
+---------------------+
1 row in set (0.00 sec)
--- 1 out of 1 results in 0ms ---

mysql> drop table if exists t; create table t; insert into t values(0); select * from t;
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.04 sec)

Query OK, 1 row affected (0.00 sec)

+---------------------+
| id                  |
+---------------------+
| 1515939651171385660 |
+---------------------+
1 row in set (0.00 sec)
--- 1 out of 1 results in 1ms ---
3

I’m thinking if it’s some Windows’ issue. @tomat pls try reproducing it in Windows.

4

I used the auto-id feature for all documents. I did not specify my own id at any time.
FYI, I’ve been using Manticore 5.0.2 till recently and never seemed to face this issue. This seemed to popup after shifting to 6.3. While shifting from 5.0.2 to 6.3 I did a full re-index of my documents. I am not using the old index. While re-indexing, I made sure not to copy over the “id” field of each doc.

This is the result I get when I do Create and Drop table thrice

[{'id': 1677721600001}]
[{'id': 1677721600002'}]
[{'id': 1677721600003}]

Interestingly, when I restart the server and do the same process, I get the same ids. In my understanding, the ids should be different as the server restart timestamp is involved?

Also I am using tables specified via a config file, if that matters.

Another question, as a temp work around, if I wish to generate my own ids for now, are negative integers ok as ids? Sometimes the uuid 64 bit to integer conversion generates negative integers.

5

I can not reproduce issue at Windows 10 or Windows 11 with Manticore search 6.3.0 release.
After I started daemon I issue

mysql> drop table if exists t; create table t; insert into t values(0); select * from t;
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.01 sec)

Query OK, 1 row affected (0.00 sec)

+---------------------+
| id                  |
+---------------------+
| 1011553662943625217 |
+---------------------+
1 row in set (0.00 sec)
--- 1 out of 1 results in 0ms ---

then I restarted daemon and could insert new documents without replacing the old document

mysql> insert into t values(0); select * from t;
Query OK, 1 row affected (0.00 sec)

+---------------------+
| id                  |
+---------------------+
| 1011553663245615105 |
| 1011553662943625217 |
+---------------------+
2 rows in set (0.00 sec)

even if I recreate table I still see the new doc id

mysql> drop table if exists t; create table t; insert into t values(0); select * from t;
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.01 sec)

Query OK, 1 row affected (0.00 sec)

+---------------------+
| id                  |
+---------------------+
| 1011553663245615106 |
+---------------------+
1 row in set (0.00 sec)
6

could you check that you do not start daemon with --test cli?
create you create MRE that reproduces issue locally and provide all files?

7

@regstuff ^