Manticore as a load balancer.

hey !
I am currently handling a large volume of log data generated every second, which needs to be inserted into a Manticore table/index called logs. Inserting all these records into a single Manticore server could potentially cause a crash due to the high load. To mitigate this, I have employed Nginx as a load balancer to distribute the insertion workload across multiple Manticore servers, such as serverA_manticore, serverB_manticore, serverC_manticore, and so on.

I am seeking an alternative approach to load balancing or a native solution within Manticore that supports this kind of distributed insertion behavior. Thank you.

why not use Logstash with distributed pipeline?

I have configured Fluent Bit to monitor the log file and utilize its output plugin to seamlessly insert the log data into Manticore. The configuration file fluent-bit.conf specifies the HTTP output plugin, directing the data flow to the designated load balancer, identified by ${LOADBALANCER_HOST} and ${LOADBALANCER_PORT}.

Name http
Match *
etc …

This is planned and there’s some progress, but it’s far from completeness yet - Manticore public roadmap (large tasks only planned for the nearest future, not bugfixes / smaller issues) · GitHub

1 Like