Packets out of order error

I am starting to get error like the following:

PHP Warning: Packets out of order. Expected 68 received 69. Packet size=3338050
PHP Warning: mysqli::query(): (00000/0):
PHP Warning: Manticore call failed: SELECT * FROM myhb_ru WHERE MATCH (’ (inclisiran|incluiran|incision|indiciran|incluiram) ') AND (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5)

I am investigating this so far it seems to affect just one index myhb_ru (Russian language documents) and if I try “max_matches=10” the query runs fine.

This problem just started appearing with version 5.*

Any suggestions on connection/query/query results/buffering parameters I should look into ?

The mysql database runs just fine, only the Manticore queries via the mysql connector seem to cause the problem.

any ideas appreciated
Thanks
Roberto

What do you get if you run the same query via a Linux mysql client?
What’s in the query log?
Anything interesting in the searchd log?

Some more information:

Running the query from tty (mysql -P 9306 -h 0):

Try 1
SELECT * FROM myhb_ru WHERE MATCH (’ (inclisiran|incluiran|incision|indiciran|incluiram) ') AND (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);

I get : “ERROR 2000 (HY000): Unknown MySQL error”

Try 2 (max_matches=10):
SELECT * FROM myhb_ru WHERE MATCH (’ (inclisiran|incluiran|incision|indiciran|incluiram) ') AND (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=10, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);

works fine and displays the data.

I noted something that to me does not look right, but I have a few questions first (I am using the latest Manticore version with columnar but I am not currently using it).

The SQL table used to build the indices (including myhb_ru) has 11 fields which contain a path to a file whose content needs to be indexed (they are defines as sql_file_field in the source definition) , now when I check the results of the sql query I see that for these fields the query returns not the path to the file (which is stored in the table) but its content (btw I want the content to be indexed but not stored as an attribute …).

I wonder if this behaviour (which is new and causing all memory buffers to fill up as the files are big) is introduced by columnar or the new Manticore version, if I remember correctly the query used to show the path and not the content of the file.

I hope this makes sense, let me know if you have questions.

Thanks

It always worked so.

I want the content to be indexed but not stored as an attribute

In 5.0.2 it’s indexed and stored in docstore by default (not as an attribute). Use Manticore Search Manual to control the behaviour.

SELECT * FROM myhb_ru WHERE MATCH (’ (inclisiran|incluiran|incision|indiciran|incluiram) ') AND (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
I get : “ERROR 2000 (HY000): Unknown MySQL error”

Can you try to minimize the query until it stops failing, e.g. does SELECT * FROM myhb_ru OPTION max_matches=100 fail?

The basic query you suggest works, also the original query works if instead of “select * …” I only do “select id, commercial_name …”.

To me the problem seems to be the size of the fields defined as: sql_file_field.

Below are all the query tests I ran.

-------------------- Query tests -----------------------
Original query:
SELECT * FROM myhb_ru WHERE MATCH (’ (inclisiran|incluiran|incision|indiciran|incluiram) ') AND (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);

Query:
SELECT * FROM myhb_ru OPTION max_matches=100
Result: Works, 20 rows in set

Query:
SELECT * FROM myhb_ru;
Result: Works, 20 rows in set

Query:
SELECT * FROM myhb_ru where (language_code = 23) OPTION max_matches=100;
Result: Works, 20 rows in set

Query:
SELECT * FROM myhb_ru where (language_code = 23) OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
Result: Works, 20 rows in set

Query;
SELECT * FROM myhb_ru where (language_code = 23) LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
Result: works, 100 rows in set

Query:
SELECT * FROM myhb_ru where (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
Result: works, 100 rows in set

Query:
SELECT * FROM myhb_ru where MATCH (‘inclisiran’) and (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
Result: Works, 2 rows in set

Query:
SELECT * FROM myhb_ru where MATCH (‘inclisiran|incluiran’) and (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
Result: Works, 29 rows in set

Query:
SELECT * FROM myhb_ru where MATCH (‘inclisiran|incluiran|incision’) and (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
Result: Works, 100 rows in set

Query:
SELECT * FROM myhb_ru where MATCH (‘inclisiran|incluiran|incision|indiciran’) and (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
Result: ERROR 2000 (HY000): Unknown MySQL error

Query:
SELECT * FROM myhb_ru where MATCH (‘inclisiran|incluiran|incision|incluiram’) and (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
Result: Works, 100 rows in set

Query:
SELECT * FROM myhb_ru where MATCH (‘inclisiran|incluiran|indiciran’) and (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
ERROR 2000 (HY000): Unknown MySQL error

Query:
SELECT id, commercial_name FROM myhb_ru where MATCH (‘inclisiran|incluiran|indiciran’) and (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
Result: Works, 100 rows in set

Query:
SELECT * FROM myhb_ru where MATCH (‘inclisiran|indiciran’) and (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
ERROR 2000 (HY000): Unknown MySQL error

Query:
SELECT * FROM myhb_ru where MATCH (‘indiciran’) and (language_code = 23) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
ERROR 2000 (HY000): Unknown MySQL error

Query:
SELECT * FROM myhb_en where MATCH (‘indiciran’) and (language_code = 8) GROUP BY med_id LIMIT 0,100 OPTION max_matches=100, ranker=sph04, field_weights=(commercial_name=2, pharmaceutical_form=2, active_ingredient=2, mah=2, dosage=2, barcode=5);
Result: Works, 12 rows in set


I checked the manual again but I am still confused:

"By default when an index is defined in a configuration file, full-text fields’ original content is both indexed and stored. "

It seems like defining a field as “sql_file_field” makes it a full text field which is by default indexed and stored (can you please confirm ?). The manual entry for sql_file_field: “This directive makes indexer interpret field contents as a file name, and load and index the referred file.” It is not clear from the manual whether this field is stored or not and also if it is stored as a file name or as its content.

From what I see I have been storing Gbytes of files data that I wanted to be indexed but not stored.

Can you please clarify the above ?

What shall I do to specify that I want those files indexed but not stored ?

Thanks
Roberto

as described at the stored_fields section you could add stored_fields = option to index section to prevent of storing any field but only indexing them

So basically unless I do "stored_fields= "
the default behaviour for a field would be the same as defining it as a “sql_field_string” ?

So basically unless I do "stored_fields= "
the default behaviour for a field would be the same as defining it as a “sql_field_string” ?

Yes, except that sql_field_string will make ft field + stored in docstore + string attr, while sql_field_field will not make it a string attr.

@rlattuad this looks like a bug. Can you create an issue on GitHub and provide your index, so we can reproduce this on our side?

Yes, I can send all the .spa, .spb files etc… for the Russian language index but it is over 4Gb of data, is there a way I can send big files to you ? do you have a ftp area I can upload the files into ?
Thanks

Yes - Manticore Search Manual

Will plain

ftp dev.manticoresearch.com
work ?
I can see the port for ftp (21) being open but I cannot connect tried using IP also

Sorry, my bad, the ftp wasn’t accessible. It’s fixed now.

I filed the bug and I am transferring the index files, I also set "stored_fields= " so that files content is not stored and all the queries run fine.

The related issue is Select query produces ERROR 2000 (HY000): Unknown MySQL error · Issue #835 · manticoresoftware/manticoresearch · GitHub