Hello Everyone,
So we are trying to migrate to Manticore from Sphinx 2.2.1 and our custom UDF is experiencing some issues. I originally wrote this many years ago in C and works fairly well with Sphinx 2.2.1. Keep in mind my C programming skills are not what they used to be as I don’t really use it.
So the initial problem I had in the migration was that I had to pack my 64 bit numbers in a different sequence (Which I dont understand why yet. Maybe architecture change on my laptop?). The original sequence was:
mva_val = (sphinx_int64_t)mva[i+1] << 32;
mva_val += (sphinx_int64_t)mva[i];
Now with Manticore:
mva_val = (sphinx_int64_t)mva[i] << 32;
mva_val += (sphinx_int64_t)mva[i+1];
So the issue is that our search will randomly fail during our regression tests when we use our custom function BETWEEN_MVA(). I am posting that code here in the hopes that someone can help me figure out why.
I am also posting the crash dump that was recorded
----- CODE
// between_mva.c
//
// Sphinx - User Defined Function - Plugin
//
// This function will accept a 64 or 32 bit Multi Value Attribute (MVA) and a min and max of same type.
// It will then compare each value of the MVA by checking if that value exists within a the min and max range and return it.
// Note: This method returns 0 for value not found.
//
//
// Function Interface:
// -------------------
// @param INT64/32 array $mva The MVA to iterate over and scan
// @param INT64/32 $min The minimum of the range to search from
// @param INT64/32 $max The maximum of the range to search to
//
// @return INT64 The value that exists within the min and max range or 0 if none
//
// @throws Exception On invalid inputs or types
//
#include “sphinxudf.h”
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#ifdef _MSC_VER
#define snprintf _snprintf
#define DLLEXPORT __declspec(dllexport)
#else
#define DLLEXPORT
#endif
/// UDF version control
/// gets called once when the library is loaded
DLLEXPORT int between_mva_ver ()
{
return SPH_UDF_VERSION;
}
/// UDF initialization
/// Verifies inputs
DLLEXPORT int between_mva_init ( SPH_UDF_INIT * init, SPH_UDF_ARGS * args, char * error_message )
{
// Validate Inputs
if ( args->arg_count < 3 )
{
snprintf ( error_message, SPH_UDF_ERROR_LEN, “BETWEEN_MVA() requires (1 MVA argument, (INT64)min, (INT64)max).” );
return 1;
}
if ( args->arg_types[0] != SPH_UDF_TYPE_INT64SET && args->arg_types[0] != SPH_UDF_TYPE_UINT32SET )
{
snprintf ( error_message, SPH_UDF_ERROR_LEN, “BETWEEN_MVA() requires that argument 1 be an MVA list of type INT64 or UINT32.” );
return 1;
}
if ( args->arg_types[1] != SPH_UDF_TYPE_INT64 && args->arg_types[1] != SPH_UDF_TYPE_UINT32 )
{
snprintf ( error_message, SPH_UDF_ERROR_LEN, “BETWEEN_MVA() requires that argument 2 be of type INT64 or UINT32.” );
return 1;
}
if ( args->arg_types[2] != SPH_UDF_TYPE_INT64 && args->arg_types[2] != SPH_UDF_TYPE_UINT32 )
{
snprintf ( error_message, SPH_UDF_ERROR_LEN, “BETWEEN_MVA() requires that argument 3 be of type INT64 or UINT32.” );
return 1;
}
// Store the mva32 vs mva64 flag to func_data
init->func_data = (void *)(long)( args->arg_types[0]==SPH_UDF_TYPE_INT64SET ? 1 : 0 );
return 0;
}
/// UDF deinitialization
/// gets called on every query, when query ends
DLLEXPORT void between_mva_deinit ( SPH_UDF_INIT * init )
{
// deallocate storage
if ( init->func_data )
{
// free ( init->func_data );
// init->func_data = NULL;
}
}
/// UDF Main Function
DLLEXPORT sphinx_int64_t between_mva ( SPH_UDF_INIT * init, SPH_UDF_ARGS * args, char * error_flag )
{
unsigned int * mva = (unsigned int *)args->arg_values[0];
sphinx_int64_t min = *(sphinx_int64_t *)args->arg_values[1];
sphinx_int64_t max = *(sphinx_int64_t *)args->arg_values[2];
sphinx_int64_t mva_val = 0;
int i, n, is64;
if ( !mva )
{
return 0;
}
// pull “mva32 or mva64” flag (that we stored in _init) from func_data
is64 = (long)(init->func_data) != 0;
if ( is64 )
{
// handle mva64
n = mva[0];
for ( i=1; i<=n; i+=2 )
{
mva_val = (sphinx_int64_t)mva[i] << 32;
mva_val += (sphinx_int64_t)mva[i+1];
if ( mva_val >= min && mva_val <= max)
{
return mva_val;
}
}
}
else
{
// handle mva32
n = mva[0];
for ( i=1; i<=n; i++ )
{
mva_val = mva[i];
if ( mva_val >= min && mva_val <= max )
{
return mva_val;
}
}
}
return 0;
}
----- DUMP
— crashed SphinxQL request dump —
SELECT id, pc_prod_override, BETWEEN_MVA(pc_prod_override, 19000000000, 99999999999)
AS mva FROM RewardTranslationIndexEn WHERE id = 72
— request dump end —
— local index:RewardTranslationIndexEn
Manticore 4.0.2 af497f245@210921 release
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 12.0.1
Configured with flags: Configured with these definitions: -DDISTR_BUILD=focal -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_JEMALLOC=1 -DWITH_ZLIB=1 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data -DFULL_SHARE_DIR=/usr/share/manticore
Host OS is Linux x86_64
Stack bottom = 0x7f220c023e70, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x20000)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x20000, stack=0x7f220c020000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
searchd[0x5c5da0]
searchd[0x47780f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f224f27e3c0]
/var/lib/manticore/data/plugins/between_mva.so(between_mva+0xab)[0x7f224ed6032e]
searchd[0x7e5656]
searchd[0x57ba0e]
searchd[0x58bc1b]
searchd[0x6046c9]
searchd[0x53cd82]
searchd[0x547b79]
searchd[0x5485cf]
searchd[0x4d48e2]
searchd[0x955ac7]
searchd[0xd2a10c]
searchd[0x48de6c]
searchd[0x48f123]
searchd[0x48b983]
searchd[0x4b0985]
searchd[0x4dd3e2]
searchd[0x46c0dd]
searchd[0x469a8b]
searchd[0xd2c27c]
searchd[0xd2ce8f]
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (Issues · manticoresoftware/manticoresearch · GitHub)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter ‘Reporting bugs’ in the manual
(Manticore Search Manual: Reporting bugs)
Dump with GDB via watchdog
[Thu Sep 30 17:17:10.448 2021] [431] watchdog: got error 4, Interrupted system call
— active threads —
thd 0 (work_3), proto mysql, state query, command select
— Totally 2 threads, and 1 client-working threads —
------- CRASH DUMP END -------
Thanks for any help.