Hello, i begin starting project which use Manticore on production environment, and found this situation.
First of all i declare settings and next ask question.
In kubernetes I have a manticore running in cluster mode with 1 manticore-balancer and 3 manticore-worker, default settings, meaning only replication without sharding.
Balancer:
mysql> describe orders;
+---------------------------+-------------------+
| Agent | Type |
+---------------------------+-------------------+
| 172.22.109.31:9312:orders | remote_1_mirror_1 |
| 172.22.111.37:9312:orders | remote_1_mirror_2 |
| 172.22.112.33:9312:orders | remote_1_mirror_3 |
+---------------------------+-------------------+
worker
mysql> describe orders;
+--------------+-----------+----------------+
| Field | Type | Properties |
+--------------+-----------+----------------+
| id | bigint | |
| _id | text | indexed stored |
| search | text | indexed stored |
| created_at | timestamp | |
| done_at | timestamp | |
| org | string | |
| vendor | string | |
| event | string | |
| status | string | |
| tickets_type | string | |
+--------------+-----------+----------------+
Index have 41M docs. Application make first request to get docs just ordered by created_at. I found that is failed. I try to inspect failing situation. I found different behavior between requests into balancer and worker directly. Examples below I make to different kubernetes services.
Request ASC ordering to manticore-worker service of kubernetes
time echo -n '{"_source": ["_id"], "index": "orders", "limit": 20, "offset": 0, "query": {"bool": {"must": []}}, "sort": [{"created_at": "asc"}]}' | http POST http://localhost:19308/search
HTTP/1.1 200 OK
Content-Length: 1807
Content-Type: application/json; charset=UTF-8
Server: 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
{
"hits": {
"hits": [
...
],
"total": 41857336,
"total_relation": "eq"
},
"timed_out": false,
"took": 876
}
real 0m1,137s
user 0m0,215s
sys 0m0,035s
Request DESC ordering to manticore-worker service of kubernetes
time echo -n '{"_source": ["_id"], "index": "orders", "limit": 20, "offset": 0, "query": {"bool": {"must": []}}, "sort": [{"created_at": "desc"}]}' | http POST http://localhost:19308/search
HTTP/1.1 200 OK
Content-Length: 1808
Content-Type: application/json; charset=UTF-8
Server: 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
{
"hits": {
"hits": [
...
],
"total": 41857388,
"total_relation": "eq"
},
"timed_out": false,
"took": 2879
}
real 0m3,147s
user 0m0,240s
sys 0m0,016s
Request ASC ordering to manticore-balancer service of kubernetes
time echo -n '{"_source": ["_id"], "index": "orders", "limit": 20, "offset": 0, "query": {"bool": {"must": []}}, "sort": [{"created_at": "asc"}]}' | http POST http://localhost:19308/search
HTTP/1.1 200 OK
Content-Length: 1808
Content-Type: application/json; charset=UTF-8
Server: 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
{
"hits": {
"hits": [
...
],
"total": 41857494,
"total_relation": "eq"
},
"timed_out": false,
"took": 1545
}
real 0m1,857s
user 0m0,229s
sys 0m0,024s
Request DESC ordering to manticore-balancer service of kubernetes
time echo -n '{"_source": ["_id"], "index": "orders", "limit": 20, "offset": 0, "query": {"bool": {"must": []}}, "sort": [{"created_at": "desc"}]}' | http POST http://localhost:19308/search
HTTP/1.1 200 OK
Content-Length: 92
Content-Type: application/json; charset=UTF-8
Server: 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
[
{
"error": "index orders: agent 172.22.109.31:9312: query timed out",
"total": 0,
"warning": ""
}
]
real 0m3,291s
user 0m0,226s
sys 0m0,019s
Logs on balancer pod:
/* Thu Sep 07 12:25:09.416 2023 conn 522156 real 3.003 wall 3.003 found 0 */ /*{"_source": ["_id"], "index": "orders", "limit": 20, "offset": 0, "query": {"bool": {"must": []}}, "sort": [{"created_at": "desc"}]} */ /*error=index orders: agent 172.22.111.37:9312: query timed out */
/* Thu Sep 07 12:25:14.115 2023 conn 522161 real 3.007 wall 3.007 found 0 */ /*{"_source": ["_id"], "index": "orders", "limit": 20, "offset": 0, "query": {"bool": {"must": []}}, "sort": [{"created_at": "desc"}]} */ /*error=index orders: agent 172.22.109.31:9312: query timed out */
/* Thu Sep 07 12:25:19.711 2023 conn 522166 real 3.000 wall 3.000 found 0 */ /*{"_source": ["_id"], "index": "orders", "limit": 20, "offset": 0, "query": {"bool": {"must": []}}, "sort": [{"created_at": "desc"}]} */ /*error=index orders: agent 172.22.112.33:9312: query timed out */
log on worker pod:
/* Thu Sep 07 12:26:53.786 2023 conn 1734186 real 5.308 wall 5.309 found 41858275 */ /*{"bool":{"must":[]}} */
[Thu Sep 7 12:26:53.846 2023] [77] WARNING: send() failed: 32: Broken pipe, sock=101
WARNING: send() failed: 32: Broken pipe, sock=101
Success version of request (sorting is ASC) looks like this:
/* Thu Sep 07 12:30:14.702 2023 conn 1588403 real 0.923 wall 0.924 found 41858467 */ /*{"bool":{"must":[]}} */
I am a new user with Manticore and have no any experience in SphinxSearch, my question are:
- Have any idea why request have success when using worker but failed with balancer?
- For example i want to speedup my target request, in RDB with SQL it will be looks like
SELECT _id FROM orders ORDER BY created_at DESCand creating index in table for created_at field. Does Manticore provide any instruments to do similar?