Neural search plugins / Bert

Feature Requests
1

Hi guys,

Hope you are all doing well !

I found an interesting project called NBoost. It is a scalable, search-api-boosting platform for deploying transformer models to improve the relevance of search results on different platforms (i.e. Elasticsearch).

And another, HackerBert, a showcase of combining Elasticsearch with BERT on the HackerNews public data.

So my question/request is if manticoresearch has planned to integrate some neural search plugins in the roadmap as it is for sure the future of data retrieval. They all use BERT models.

If yes, it can be huge and awesome :slight_smile:

Thanks in advance for your inputs and insights on these questions that can be turned as a request feature also.

Ps. Interesting article about neural search motivations, I Left Tencent AI and Raised $2M to Build the Next Neural Search Company in Open Source · Han Xiao Tech Blog - Neural Search & AI Engineering

Cheers,
Luc Michalski

2

Hi Luc

Yes, we have it in plans, but not in the nearest. We experimented with BERT, DESM in combination with BM25 and without and got some results. We just need time and resources to integrate it into Manticore. Perhaps we’ll have time for it in Q3-Q4 this year.

We also experimented with BERT and Ernie and other models for another ML task - unsupervised clustering and topic extraction, also got some good results and it looks even more promising than the relevance improvements direction, but we also need to find place for that in our schedule.