I just read with interest that Manticore can now chunk documents internally. (See Better Vector Search for Long Documents: Chunking Inside Manticore Search ) What is the best strategy to use this new feature in combination with multipage PDFs?
We currently use a standard keyword-based search with multipage PDFs. Each page is loaded/indexed as a document and Manticore returns the most relevant pages. We are thinking of changing this to a hybrid search. If we use internal chunking, then this would help with pages that have a lot of text, for example, a newspaper page or a dense journal page. But I wonder if it would be better to load the entire PDF, so that the internal chunking is not artificially stopped by page boundaries, which can split sentences and paragraphs. Also, some multipage PDFs have little text per page and for those a single chunk should probably include several pages.
But if the entire PDF is loaded as a whole, how would we know from the search results which page the relevant page is? In other words, how does one go from a chunk back to a page in the PDF?
Any suggestions on how this could be handled best?