Data source error: bad searchd response length

General Discussions - English
1

Hello,
I am indexing a table with size of more than 20GB. But while connecting via SphinxSE, some of the queries worked and some shows “bad searchd response length” error,
Below are the table creation and the queries which are executed in SphinxSE. The “Paragraph” in the table need to be indexed, where the datatype of the column is text and length is more.
Also I tried adding “sql_field_string = paragraph” in config file, but same error repeats. Does I missing anything in config file ?
I tried adding attributes in searchd also " access_doclists=file / access_hitlists=file / read_buffer_docs = 1M / read_buffer_hits = 1M" but same error or sometimes shows empty result if I changed mode as “extended2” to “any”.

Table Creation:
CREATE TABLE bbbb.bbbb (
id BIGINT(10) UNSIGNED NOT NULL,
weight INT(11) NOT NULL,
QUERY LONGTEXT NOT NULL,
filing_date BIGINT(20) DEFAULT NULL,
_sph_count INT(11) DEFAULT NULL,
_sph_groupby INT(11) DEFAULT NULL,
KEY QUERY (QUERY(768))
) ENGINE=SPHINX DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci CONNECTION=‘sphinx://localhost:9312/test’;

Query result coming:
SELECT DISTINCT S.id , S.Filing_date
FROM xxx.xxx S WHERE QUERY=‘select=*;
range=id,42811;
index=test;
limit=1000;offset=0;maxmatches=1000;mode=extended2;
groupby=attr:id;’
ORDER BY S.filing_date DESC;

Error query:
SELECT DISTINCT S.id , S.Filing_date
FROM xxx.xxx S WHERE QUERY=‘select=*;
range=filing_date,1641189600,1641189600;
index=test;
limit=1000;offset=0;maxmatches=1000;mode=extended2;
groupby=attr:id;’
ORDER BY S.filing_date DESC;

Error Code: 1430
There was a problem processing the query on the foreign data source. Data source error: bad searchd response length (length=48211600)

Configuration file:

source GmSource
{
type = mysql
sql_host = 127.0.0.1
sql_user = xxx
sql_pass = xxx
sql_db = xxx
sql_port = 3306 # optional, default is 3306
sql_query = SELECT doc.id, UNIX_TIMESTAMP(doc.filing_date) AS filing_date, doc.paragraph
FROM xxx.xxx doc ;

sql_attr_timestamp = filing_date
sql_field_string = paragraph

sql_ranged_throttle = 300
unpack_mysqlcompress_maxsize = 50M

}
table test
{
source = GmSource
path = /disk2/TEXT_SEARCH/MANTICORE/index_year
morphology = stem_en
wordforms = /usr/local/sphinx/etc/wordforms.04june2011.txt
}
indexer
{
mem_limit = 1024M
}
searchd
{
listen = 127.0.0.1:9312:sphinx
listen = localhost:9306:mysql
listen = localhost:9308:http
mysql_version_string=5.0.37
max_packet_size=64M
log = /disk2/TEXT_SEARCH/MANTICORE/log/searchd.log
query_log = /disk2/TEXT_SEARCH/MANTICORE/log/query.log
pid_file = /disk2/TEXT_SEARCH/MANTICORE/log/searchd.pid
query_log_format = sphinxql
seamless_rotate = 1
unlink_old = 1
max_filters = 256
max_filter_values = 1000000
watchdog = 1
threads=10
}

Searchd log:
[Thu Aug 17 04:54:39.079 2023] [20829] WARNING: timed out while performing SyncSend to flush network buffers, sock=29

Query log:
/* Thu Aug 17 04:54:34.058 2023 conn 36 real 0.016 wall 0.017 found 2159 */ SELECT * FROM test WHERE filing_date BETWEEN 1641189600 AND 1641189600 GROUP BY id WITHIN GROUP ORDER BY weight() desc LIMIT 1000 OPTION cutoff=0, retry_count=0, retry_delay=0;

2

as replied here Error in Connecting via SphinxSE - #4 by Adhisashan change of listen = localhost:9312 to listen = localhost:9312:sphinx at the daemon config helped

3

Hello Tomat,
Thanks for the quick response. As I mentioned in question, I already used “listen = localhost:9312:sphinx” in my configuration file.

4

Hi,
I found the issue, if the string size greater than 16MB, then it shows “Data source error: bad searchd response length”. Is there any properties missing in my configuration file.

Thanks in advance

5

Hi ,
After many attempts & searches , I found the solution for “Data source error: bad searchd response length”. Instead of select=* , changed to select=required_fields worked in below SphinxSE query. Hope this will help others.

SELECT DISTINCT S.id , S.Filing_date
FROM xxx.xxx S WHERE QUERY=‘select=*;
range=filing_date,1641189600,1641189600;
index=test;
limit=1000;offset=0;maxmatches=1000;mode=extended2;
groupby=attr:id;’
ORDER BY S.filing_date DESC;

1 Like