We currently use a standard keyword-based search with multipage PDFs. Each page is loaded/indexed as a document and Manticore returns the most relevant pages. We are thinking of changing this to a hybrid search. If we use internal chunking, then this would help with pages that have a lot of text, for example, a newspaper page or a dense journal page. But I wonder if it would be better to load the entire PDF, so that the internal chunking is not artificially stopped by page boundaries, which can split sentences and paragraphs. Also, some multipage PDFs have little text per page and for those a single chunk should probably include several pages.
But if the entire PDF is loaded as a whole, how would we know from the search results which page the relevant page is? In other words, how does one go from a chunk back to a page in the PDF?
Any suggestions on how this could be handled best?
Thank you. Yes, some facility like this would be very useful.
I would expand it to include keyword search as well. Also, for keyword searching in a longer document (not a page-split one), getting a character or word index that identifies the location of the match would make sense.
Then the need to split PDFs, as we do now, would be eliminated. One could pass entire documents but still identify the page/location of the match.
For keyword search there’s already the highlighting functionality which can do that - Searching > Highlighting | Manticore Search Manual
Or am I missing something and you need a specific offset in the keyword mode too?
Actually, the same highlighting functions can already highlight text in vector mode with auto-chunking. There’s a Docker image in the pull request. Would you like to check it out to see if it solves your issue?
Sergey, the highlighting functionality you have is not what we need because we provide from the search results a link back to the PDF page. Thus, we not only need a text snippet with highlighting, but we also need the page number in the PDF where the match happens. Right now, we get this through the document ID by splitting PDFs into pages and loading each page as a document.
Knowing the page where a match happened in a PDF is the same type of information as knowing where a matching chunk is located in a document. Breaking a PDF into pages is its own version of chunking. Thus, I think they could ideally be handled the same.
Just to provide more context. We have a full-text search engine that searches in about 5000 PDFs. The search identifies the pages in the PDFs where a match exists. We do not show text snippets but graphical snippets from the PDF where matches happened. These graphical snippets are then linked to a PDF viewer that displays the full page of the PDF. You can see how the search results are displayed here Lybrary.com Articles
Since we are on the topic of chunking, I am missing from your internal chunkers one that supports structure-aware semantic chunking. Meaning one that preferentially cuts at headings.
For example, it could comprehend basic Markdown formatting, or accept structured formats like the JSON Docling outputs that can include all the necessary information to link back to the PDF page, and even the bounding boxes where on the page.