Доброго времени суток!
На кластере:
Server version: 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822) git branch manticore-6.2.12…origin/manticore-6.2.12
Во время ALTER кластерного индекса products получили падение сервиса manticore.
Команда:
ALTER TABLE products ADD COLUMN school_purpose_id_list multi;
Схема:
+---------------------------------------+-----------+----------------+
| Field | Type | Properties |
+---------------------------------------+-----------+----------------+
| id | bigint | |
| name | text | stored |
| slug | text | stored |
| description | text | stored |
| reasons_to_buy | text | stored |
| additional_images | text | stored |
| school_umk_title | text | indexed stored |
| seo_description | text | stored |
| relations_set | text | indexed stored |
| author_full_name_list | text | indexed stored |
| publisher_series_name | text | indexed stored |
| publisher_name | text | indexed stored |
| publisher_brand_name | text | indexed stored |
| manufacturer_name | text | indexed stored |
| manufacturer_brand_name | text | indexed stored |
| availability_status | uint | |
| availability_quantity | uint | |
| vendor_id | uint | |
| nds | uint | |
| literature_work_publishing_year | uint | |
| printing_page_count | uint | |
| printing_copy_count | uint | |
| weight | uint | |
| height | uint | |
| width | uint | |
| length | uint | |
| rating_weight | uint | |
| rating_star | uint | |
| rating_count | uint | |
| review_count | uint | |
| purchase_stats_week_count | uint | |
| purchase_stats_month_count | uint | |
| purchase_stats_year_count | uint | |
| purchase_stats_day_avg_count | uint | |
| school_purpose_id | uint | |
| school_education_system_id | uint | |
| school_umk_id | uint | |
| school_exam_year_id | uint | |
| publisher_series_id | uint | |
| publisher_id | uint | |
| publisher_brand_id | uint | |
| manufacturer_id | uint | |
| manufacturer_brand_id | uint | |
| literature_work_cycle_id | uint | |
| literature_work_cycle_volume_id | uint | |
| binding_id | uint | |
| product_type_id | uint | |
| article_number_id | uint | |
| stationery_format_id | uint | |
| stationery_line_type_id | uint | |
| stationery_ink_color_id | uint | |
| stationery_lead_diameter_id | uint | |
| stationery_case_shape_id | uint | |
| stationery_mechanism_type_id | uint | |
| stationery_diameter_id | uint | |
| stationery_feature_id | uint | |
| stationery_colors_quantity_id | uint | |
| stationery_gender_id | uint | |
| stationery_length_id | uint | |
| stationery_staple_number_id | uint | |
| stationery_stapler_number_id | uint | |
| stationery_material_id | uint | |
| stationery_punched_sheets_quantity_id | uint | |
| stationery_pen_thickness_id | uint | |
| stationery_mounting_type_id | uint | |
| stationery_pen_tip_shape_id | uint | |
| stationery_ink_base_id | uint | |
| stationery_calculator_capacity_id | uint | |
| stationery_calendar_year_id | uint | |
| stationery_calendar_type_id | uint | |
| stationery_calendar_subject_id | uint | |
| stationery_clasp_type_id | uint | |
| stationery_compartments_quantity_id | uint | |
| stationery_cover_binding_id | uint | |
| stationery_package_type_id | uint | |
| stationery_cover_surface_id | uint | |
| stationery_universal_id | uint | |
| stationery_sheets_quantity_id | uint | |
| stationery_color_id | uint | |
| stationery_volume_id | uint | |
| game_audience_id | uint | |
| game_child_age_id | uint | |
| game_series_id | uint | |
| game_duration_id | uint | |
| constructor_nation_id | uint | |
| constructor_equipment_type_id | uint | |
| souvenir_format_id | uint | |
| souvenir_set_quantity_id | uint | |
| toy_type_id | uint | |
| toy_height_id | uint | |
| age_restriction | bigint | |
| preorder_available_at | timestamp | |
| released_at | timestamp | |
| is_school_prepare | bool | |
| is_out_of_class_reading | bool | |
| price_retail | float | |
| price_discounted_for_guest | float | |
| price_discount_percent_for_guest | float | |
| price_discounted_for_authorized | float | |
| price_discount_percent_for_authorized | float | |
| price_discounted_for_subscriber | float | |
| price_discount_percent_for_subscriber | float | |
| rating_average | float | |
| image | string | |
| isbns | string | |
| printing_page_format | string | |
| seo_title | string | |
| excerpts | json | |
| marketing_status_list | mva | |
| school_grade_id_list | mva | |
| school_subject_id_list | mva | |
| school_material_type_id_list | mva | |
| school_exam_id_list | mva | |
| main_category_id_list | mva | |
| category_id_list | mva | |
| tbk_id_list | mva | |
| ekn_id_list | mva | |
| author_id_list | mva | |
| translator_id_list | mva | |
| illustrator_id_list | mva | |
| tag_id_list | mva | |
| product_set_id_list | mva | |
| stationery_brush_shape_id_list | mva | |
| stationery_brush_material_id_list | mva | |
| stationery_brush_number_id_list | mva | |
| stationery_painting_technique_id_list | mva | |
| stationery_lead_hardness_id_list | mva | |
| comic_universal_id_list | mva | |
| comic_character_id_list | mva | |
| comic_genre_id_list | mva | |
| comic_series_id_list | mva | |
| comic_type_id_list | mva | |
| comic_line_id_list | mva | |
| comic_section_id_list | mva | |
| comic_subject_id_list | mva | |
| game_player_quantity_id_list | mva | |
| game_use_case_id_list | mva | |
| game_skill_id_list | mva | |
| constructor_detail_quantity_id_list | mva | |
| souvenir_reason_id_list | mva | |
| gift_hobby_id_list | mva | |
| gift_for_children_id_list | mva | |
| gift_new_year_id_list | mva | |
| gift_section_id_list | mva | |
| gift_books_on_interest_id_list | mva | |
| shop_id_list | mva | |
| city_id_list | mva | |
| shop_brand_id_list | mva | |
+---------------------------------------+-----------+----------------+
Объем диска для индекса ~ 3.4G
Количество строк в индексе ~1310000
crash dump:
------- FATAL: CRASH DUMP -------
[Mon May 20 21:46:02.671 2024] [2608850]
--- crashed SphinxQL request dump ---
SELECT
id,
product_id,
shop_id,
city_id,
shop_brand_id,
price_retail,
price_discounted_for_guest,
price_discount_percent_for_guest,
price_discounted_for_authorized,
price_discount_percent_for_authorized,
price_discounted_for_subscriber,
price_discount_percent_for_subscriber,
price_discounted_with_loyalty_card,
price_discount_percent_with_loyalty_card,
availability_quantity,
in_shop_locations,
1000000000000 * literature_work_publishing_year
+ released_at as newness
FROM retail_products
WHERE ALL(tbk_id_list) NOT IN
(32, 30) AND product_id IN (2598032) AND city_id IN (213)
LIMIT 100
OFFSET 0
OPTION
max_matches = 100
--- request dump end ---
--- local index:@groupby desc
Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 15.0.7
Configured with flags: Configured with these definitions: -DDISTR_BUILD=jammy -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (jammy) (cross-compiled)
Stack bottom = 0x7feffad7bc80, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7feffad80000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x22a)[0x561cb28ee9fa]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x355)[0x561cb276d2b5]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7ff6d331b520]
/lib/x86_64-linux-gnu/libc.so.6(+0x1aedcd)[0x7ff6d3487dcd]
/usr/bin/searchd(_ZN18IndexAlterHelper_c26Alter_AddRemoveRowwiseAttrERK10CSphSchemaS2_PKjjPKhR14WriteWrapper_cS8_bRK10CSphString+0x293)[0x561cb36659d3]
/usr/bin/searchd(_ZN13CSphIndex_VLN18AddRemoveAttributeEbRK18AttrAddRemoveCtx_tR10CSphString+0x66a)[0x561cb27fd22a]
/usr/bin/searchd(_ZN9RtIndex_c18AddRemoveAttributeEbRK18AttrAddRemoveCtx_tR10CSphString+0x3d5)[0x561cb3523b35]
/usr/bin/searchd(+0xea8142)[0x561cb27bb142]
/usr/bin/searchd(_ZN15ClientSession_c7ExecuteESt4pairIPKciER11RowBuffer_i+0x193b)[0x561cb27b7c3b]
/usr/bin/searchd(_Z20ProcessSqlQueryBuddySt4pairIPKciERhR21GenericOutputBuffer_c+0x52)[0x561cb2717d52]
/usr/bin/searchd(_Z8SqlServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EE+0x105d)[0x561cb26fd19d]
/usr/bin/searchd(_Z10MultiServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EESt4pairIitE7Proto_e+0x43)[0x561cb26f8f13]
/usr/bin/searchd(+0xde6af2)[0x561cb26f9af2]
/usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB_+0x1c)[0x561cb3a3ddac]
/usr/bin/searchd(make_fcontext+0x37)[0x561cb3a5e167]
Trying boost backtrace:
0# sphBacktrace(int, bool) in /usr/bin/searchd
1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
2# 0x00007FF6D331B520 in /lib/x86_64-linux-gnu/libc.so.6
3# 0x00007FF6D3487DCD in /lib/x86_64-linux-gnu/libc.so.6
4# IndexAlterHelper_c::Alter_AddRemoveRowwiseAttr(CSphSchema const&, CSphSchema const&, unsigned int const*, unsigned int, unsigned char const*, WriteWrapper_c&, WriteWrapper_c&, bool, CSphString const&) in /usr/bin/searchd
5# CSphIndex_VLN::AddRemoveAttribute(bool, AttrAddRemoveCtx_t const&, CSphString&) in /usr/bin/searchd
6# RtIndex_c::AddRemoveAttribute(bool, AttrAddRemoveCtx_t const&, CSphString&) in /usr/bin/searchd
7# 0x0000561CB27BB142 in /usr/bin/searchd
8# ClientSession_c::Execute(std::pair<char const*, int>, RowBuffer_i&) in /usr/bin/searchd
9# ProcessSqlQueryBuddy(std::pair<char const*, int>, unsigned char&, GenericOutputBuffer_c&) in /usr/bin/searchd
10# SqlServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in /usr/bin/searchd
11# MultiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >, std::pair<int, unsigned short>, Proto_e) in /usr/bin/searchd
12# 0x0000561CB26F9AF2 in /usr/bin/searchd
13# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
14# make_fcontext in /usr/bin/searchd
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Dump with GDB via watchdog
ALTERT выполнялся из mysql сессии, соответственно в момент падения, консоль mysql вывела сообщение о том, что соединение с сервером потеряно.
После переподключения по mysql протоколу, попытки взаимодействия с индексом были не успешны.
Получали сообщения о том что индекс products не находится в кластере (что противоречило информации из SHOW STATUS;), обращения напрямую (минуя кластер) так же приводили к ответу, что такого индекса нет. При том его можно было увидеть через SHOW TABLES.
Ручная остановка сервиса manticore с последующим удалением каталога индекса и его пересоздание и ALTER без наполнения данными прошли успешно.
Что привело к падению и как бы такого избежать?