Server Crash with Custom UDF from Sphinx

Hello Everyone,

So we are trying to migrate to Manticore from Sphinx 2.2.1 and our custom UDF is experiencing some issues. I originally wrote this many years ago in C and works fairly well with Sphinx 2.2.1. Keep in mind my C programming skills are not what they used to be as I don’t really use it.

So the initial problem I had in the migration was that I had to pack my 64 bit numbers in a different sequence (Which I dont understand why yet. Maybe architecture change on my laptop?). The original sequence was:
mva_val = (sphinx_int64_t)mva[i+1] << 32;
mva_val += (sphinx_int64_t)mva[i];

Now with Manticore:
mva_val = (sphinx_int64_t)mva[i] << 32;
mva_val += (sphinx_int64_t)mva[i+1];

So the issue is that our search will randomly fail during our regression tests when we use our custom function BETWEEN_MVA(). I am posting that code here in the hopes that someone can help me figure out why.

I am also posting the crash dump that was recorded


----- CODE

// between_mva.c
//
// Sphinx - User Defined Function - Plugin
//
// This function will accept a 64 or 32 bit Multi Value Attribute (MVA) and a min and max of same type.
// It will then compare each value of the MVA by checking if that value exists within a the min and max range and return it.
// Note: This method returns 0 for value not found.
//
//
// Function Interface:
// -------------------
// @param INT64/32 array $mva The MVA to iterate over and scan
// @param INT64/32 $min The minimum of the range to search from
// @param INT64/32 $max The maximum of the range to search to
//
// @return INT64 The value that exists within the min and max range or 0 if none
//
// @throws Exception On invalid inputs or types
//

#include “sphinxudf.h”
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#ifdef _MSC_VER
#define snprintf _snprintf
#define DLLEXPORT __declspec(dllexport)
#else
#define DLLEXPORT
#endif

/// UDF version control
/// gets called once when the library is loaded
DLLEXPORT int between_mva_ver ()
{
return SPH_UDF_VERSION;
}

/// UDF initialization
/// Verifies inputs
DLLEXPORT int between_mva_init ( SPH_UDF_INIT * init, SPH_UDF_ARGS * args, char * error_message )
{
// Validate Inputs
if ( args->arg_count < 3 )
{
snprintf ( error_message, SPH_UDF_ERROR_LEN, “BETWEEN_MVA() requires (1 MVA argument, (INT64)min, (INT64)max).” );
return 1;
}
if ( args->arg_types[0] != SPH_UDF_TYPE_INT64SET && args->arg_types[0] != SPH_UDF_TYPE_UINT32SET )
{
snprintf ( error_message, SPH_UDF_ERROR_LEN, “BETWEEN_MVA() requires that argument 1 be an MVA list of type INT64 or UINT32.” );
return 1;
}
if ( args->arg_types[1] != SPH_UDF_TYPE_INT64 && args->arg_types[1] != SPH_UDF_TYPE_UINT32 )
{
snprintf ( error_message, SPH_UDF_ERROR_LEN, “BETWEEN_MVA() requires that argument 2 be of type INT64 or UINT32.” );
return 1;
}
if ( args->arg_types[2] != SPH_UDF_TYPE_INT64 && args->arg_types[2] != SPH_UDF_TYPE_UINT32 )
{
snprintf ( error_message, SPH_UDF_ERROR_LEN, “BETWEEN_MVA() requires that argument 3 be of type INT64 or UINT32.” );
return 1;
}

// Store the mva32 vs mva64 flag to func_data
init->func_data = (void *)(long)( args->arg_types[0]==SPH_UDF_TYPE_INT64SET ? 1 : 0 );
return 0;
}

/// UDF deinitialization
/// gets called on every query, when query ends
DLLEXPORT void between_mva_deinit ( SPH_UDF_INIT * init )
{
// deallocate storage
if ( init->func_data )
{
// free ( init->func_data );
// init->func_data = NULL;
}
}

/// UDF Main Function
DLLEXPORT sphinx_int64_t between_mva ( SPH_UDF_INIT * init, SPH_UDF_ARGS * args, char * error_flag )
{
unsigned int * mva = (unsigned int *)args->arg_values[0];
sphinx_int64_t min = *(sphinx_int64_t *)args->arg_values[1];
sphinx_int64_t max = *(sphinx_int64_t *)args->arg_values[2];
sphinx_int64_t mva_val = 0;
int i, n, is64;

if ( !mva )
{
return 0;
}

// pull “mva32 or mva64” flag (that we stored in _init) from func_data
is64 = (long)(init->func_data) != 0;
if ( is64 )
{
// handle mva64
n = mva[0];
for ( i=1; i<=n; i+=2 )
{
mva_val = (sphinx_int64_t)mva[i] << 32;
mva_val += (sphinx_int64_t)mva[i+1];
if ( mva_val >= min && mva_val <= max)
{
return mva_val;
}
}
}
else
{
// handle mva32
n = mva[0];
for ( i=1; i<=n; i++ )
{
mva_val = mva[i];
if ( mva_val >= min && mva_val <= max )
{
return mva_val;
}
}
}

return 0;
}


----- DUMP

— crashed SphinxQL request dump —
SELECT id, pc_prod_override, BETWEEN_MVA(pc_prod_override, 19000000000, 99999999999)
AS mva FROM RewardTranslationIndexEn WHERE id = 72
— request dump end —
— local index:RewardTranslationIndexEn
Manticore 4.0.2 af497f245@210921 release
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 12.0.1
Configured with flags: Configured with these definitions: -DDISTR_BUILD=focal -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_JEMALLOC=1 -DWITH_ZLIB=1 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data -DFULL_SHARE_DIR=/usr/share/manticore
Host OS is Linux x86_64
Stack bottom = 0x7f220c023e70, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x20000)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x20000, stack=0x7f220c020000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
searchd[0x5c5da0]
searchd[0x47780f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f224f27e3c0]
/var/lib/manticore/data/plugins/between_mva.so(between_mva+0xab)[0x7f224ed6032e]
searchd[0x7e5656]
searchd[0x57ba0e]
searchd[0x58bc1b]
searchd[0x6046c9]
searchd[0x53cd82]
searchd[0x547b79]
searchd[0x5485cf]
searchd[0x4d48e2]
searchd[0x955ac7]
searchd[0xd2a10c]
searchd[0x48de6c]
searchd[0x48f123]
searchd[0x48b983]
searchd[0x4b0985]
searchd[0x4dd3e2]
searchd[0x46c0dd]
searchd[0x469a8b]
searchd[0xd2c27c]
searchd[0xd2ce8f]
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter ‘Reporting bugs’ in the manual
(Manticore Search Manual: Reporting bugs)
Dump with GDB via watchdog
[Thu Sep 30 17:17:10.448 2021] [431] watchdog: got error 4, Interrupted system call
— active threads —
thd 0 (work_3), proto mysql, state query, command select
— Totally 2 threads, and 1 client-working threads —
------- CRASH DUMP END -------

Thanks for any help.

could you install debug symbol from manticore then reproduce the crash again to dump stack of the crash?

Sorry for the very late reply. Got sidetracked and put this aside for a bit. So I am testing a possible solution by getting rid of the manual packing of integer values and simply treating as 64bit always. I will post the fix once I am certain it will work for us.