Building a Fully Self-Hosted RAG System in inspirehep.net

01 Sep, 2025

What is INSPIRE?

INSPIRE (inspirehep.net) is the leading High Energy Physics open access digital library, used by tens of thousands of scientists around the world. It is a key tool in physics researchers' workflows, used to find academic papers, bibliography and metadata, track citations, or browse a collection of seminars, jobs, datasets and other resources.

A new INSPIRE in the making

At CERN we are always exploring ways of improving the user experience and supporting physicists when doing their research. That is why we have been recently reimagining how users find information in INSPIRE and how they integrate it into their research workflow.

Retrieval-Augmented Generation (RAG) combines large language models with external knowledge sources to provide more accurate, concise and grounded responses allowing users to look for concepts, as opposed to searching for keywords, titles, categories or filters. This can save them precious time navigating through papers trying to find the information they need.

However, implementing such a system is not straightforward and involves many components, from the infrastructure and backend, to the development of an alternative UI to adapt to this new way of research. This blog post will present an overview of a first version of a fully self-hosted system and discuss some of the challenges found and technical decisions we have taken to overcome them.

Overview

1. Paper ingestion
2. Embedding generation
3. Vector storage
4. Model
5. Orchestration
6. Observability
7. UI
8. Future work
9. Conclusion

Paper ingestion

Parsing and processing PDF content is not an easy process. PDF files are not homogeneous and have a variety of layouts, fonts, figures, tables, citation formats, etc. Multiple tools exist to handle this task, like Docling or Nougat, to name a few. This process needs to be scalable to the more than 1.7 million literature records that INSPIRE contains.

What information do we want to extract?

For our use case, we want to focus on extracting good hierarchical and positional information (i.e. bounding boxes), which will be useful for paper queries, and accurate text parsing. I have found Docling to be the best choice with these requirements in mind.

Docling is an open-source tool developed by IBM for parsing and extracting structured information from documents, including text, formulas and layout data. Its formula extraction module has relatively good accuracy, although that comes at a high performance cost, having to run its CodeFormula vision model, so for our initial implementation and for faster iteration we have decided to leave formula processing aside, but we will come back to it in the future.

We use Docling's hybrid chunker, which first runs hierarchical chunking (following the document structure and splitting the different elements) and then applies tokenization-aware refinements on top based on the provided tokenizer, further splitting chunks if needed or merging them when possible and appropriate (e.g. successive chunks sharing headings).

Note that, for this reason, the hybrid chunker must use the same tokenizer as the embedding model, and it must be configured with the adequate chunk size for the model, as I will explain in the next section.

Scaling ingestion

To initially process the high number of existing papers, we use k8s indexed jobs. The job controller first fetches the control numbers of the papers to be processed and splits them among the jobs. I have found that, at least for the kinds of documents we process and the Docling features we have enabled it is more resource efficient to run multiple workers per job. As an example, one worker takes 5 GB of RAM while we have managed to run up to 40 workers with 32 GB of RAM without considerable slowdowns, provided we allocate at least one cpu/thread per worker.

Docling must be configured in each worker to use one single thread via AcceleratorOptions. Note that, since our documents are digitized, we can also disable OCR, which reduces processing times. With this setup, it takes an average of 60s to process one document, but with the upside of not requiring any GPU resources which would be much more expensive to source. Thankfully, we can scale up and reduce the total time to embed the full document set to just a couple of days thanks to the parallelization we have discussed.

The ingestion process starts with fetching the range of papers we want to embed and retrieving the PDFs either from INSPIRE, if we are hosting a copy, or directly from arXiv. The retrieved paper is then fed to Docling. Due to the scale of this process, it is a good idea to implement some sort of checkpointing, to make sure we can resume the processing if something breaks without losing data. In our case, simply filtering out the already embedded control numbers from the range of documents to process does the job.

This pipeline (albeit downscaled) will be run periodically as a workflow on newly harvested documents.

Embedding generation

Once the text chunks are extracted by Docling, we embed them using AccPhysBert (Hellert_2025). This is a specialized embedding model fine-tuned for accelerator physics on top of PhysBert (10.1063/5.0238090) which in turn is a BERT-based model fine-tuned for physics. The fine-tuning has in fact been done using INSPIRE records making it a perfect fit.

The maximum sequence length for AccPhysBert is 512, meaning we can pass 512 tokens as input to the embedding model. However, Docling seems to slightly exceed the token size sometimes. This problem is less prevalent when disabling the contextualization step (a process where Docling adds the section heading at the beginning of each chunk, not very useful as we can store that as metadata), but even then it is still present in certain cases. Reducing the chunk size to 510 fixed this issue.

Our embedding model is running in CERN's Kubeflow infrastructure.

Vector storage

Once the embeddings are generated, they have to be stored in a vector DB. In this case, OpenSearch Vector. There is some configuration to be made when creating the index: "index.knn": "true" needs to be added to advanced settings in order to enable vector search, and "vector_field": { "dimension": 768, "type": "knn_vector" } needs to be included in mappings.properties in order to configure the index to store vectors of 768 dimensions, the usual vector size for BERT-based models.

A long paper can yield a large number of chunks and, by default, OpenSearch allows a bulk size of 500 (hard limit in our deployment), making it necessary to split the calls into multiple batches for documents with over 500 chunks.

Model

Of course, we can't build a RAG system without a LLM. The INSPIRE AI project aims to follow the same principles of INSPIRE: open source software for the global scientific community developed, hosted and managed responsibly at CERN. This naturally requires us to self-host our models rather than simply querying a commercial API.

For our tests I opted for Llama3.1-8B, which, although limited in capabilities compared to more modern, bigger alternatives, is a good, small, fast, easy to host model, useful for prototyping while putting all pieces of the puzzle together. We will replace this model in the future but, other than some prompt adjustments, changing the model will be a simple procedure once everything else is set up. We are serving the model via Kubeflow, using the vLLM inference engine.

Limitations

I have found the limited context length of the model to be a challenge when it comes to questions about a paper, as it is often not possible to attach the full paper content to the model context. This requires either the use of a model with a larger context size and/or a more advanced agentic RAG system. This and other future work ideas will be discussed at the end of this post.

Llama3.1-8b has also proven to be quite erratic when following advanced instructions. Once the prompt becomes moderately big or complex the model tends to ignore most of the directions. Time needed to be invested into prompt engineering and context management to try to work around this limitation. As mentioned above, switching to a more capable model is in our roadmap.

If you are using LangChain in a similar setup, save yourself a not so fun debugging session by making sure you are using the VLLMOpenAI client and not the OpenAI one. The latter will seem to work correctly at first, but after a while will cause the scheduler to start stacking requests in waiting state and make the model eventually stop responding.

Orchestration

We use LangChain, a Python framework for building applications powered by language models, providing tools for chaining together LLM calls, integrating with external data sources and managing workflows, which was particularly appealing for rapidly developing our initial system and allowing for easy integration with services like LangFuse.

While features such as traces, clients, and streamlined chain creation offer clear advantages in certain scenarios, we also observed challenges related to documentation clarity and API consistency which led to specific debugging efforts, as encountered with the OpenAI client issue. Nevertheless, LangChain was a pragmatic choice for quick iteration and development at the start, and we anticipate to extract more benefit from it as our workflows get more intricate and we scale up our infrastructure.

We have different chains for general questions (e.g. "What is the higgs boson?") and for questions related to a paper (e.g. "What are the conclusions of this paper?"), with different prompts and different queries to the vector DB (plain similarity search vs also filtering by vectors belonging to a given paper). We fetch the top 25 docs from the vector store and then rerank them using Jina Reranker v2 (a cross-encoder model to judge query–document relevance more precisely), keeping the top 10 results.

We use structured outputs to validate generation results and we attach extra metadata (e.g. ids, positional info) to the response to be send back to the frontend.

Observability

LangFuse is an open-source observability and analytics platform for LLM applications, enabling tracing, prompt version management, and user feedback collection. We use it for all those purposes.

The integration with LangChain is moderately simple, but becomes a bit tricky when it comes to including metadata (e.g. langfuse_prompt needs to be passed as metadata to the RunnableConfig in order to properly link generations to prompts).

Regarding user feedback, our system allows users to change the feedback submitted for a response and to later add a comment. This required manually creating a langfuse config object for each query, passing a run_id and metadata.langfuse_session_id. This same run_id is returned in the query response and is sent back by the frontend when submitting the feedback. Feedback submissions return the score_id, which is passed down with feedback modifications in order to prevent creating multiple unnecessary score entries for the same user and query.

Prompt management in LangFuse allows us to quickly try different prompts across environments, track changes and roll back at any moment. The chains fetch the corresponding prompts from LangFuse via langfuse.get_prompt, which also allows to configure a cache TTL for a balance between latency and recency.

UI

The last part of the puzzle is the UI. Of course we can simply create a chat window where the user asks a question and receives an answer, but this has several limitations: it is not the most visually structured way to retrieve information and it lacks any visual grounding and references to further information from INSPIRE. Therefore, we decided to create a completely new frontend.

In the main page, users will be able to ask questions and receive both a brief answer and a more detailed explanation, containing references to the text snippets used to sustain the response, as well as a series of cards displaying basic information of the most relevant papers associated with the user's query.

When clicking on a paper or on a reference, the paper page opens, displaying a split screen view with the PDF on the left, the chat window on the right for further questions, some basic information and useful links on the top (including metadata extracted from INSPIRE) and the list of relevant papers in the bottom for easy back-and-forth navigation.

The PDFs are again fetched from INSPIRE when possible (faster), or from arXiv otherwise, and are all pre-fetched when a response is received and locally cached as blobs. When opening a paper, the PDF viewer will load all the PDFs from the cache in parallel into multiple PDF viewers, only one of them being visible. This method avoids PDF rendering times when switching back and forth between papers and allows for an almost-instant navigation. The performance impact is contained as the number of papers listed for each response is limited.

When a paper is opened by clicking on a reference, the PDF viewer will scroll to the corresponding snippet and highlight its bounding box for quick visual grounding. This feature is essential for a tool where accuracy, correctness and easy verification of the information are paramount.

Future work

We are planning and currently working on some of the following features:

• Profiles: enable users to save the conversation history and notes, to share responses and conversations and to customize their experience.
• Infrastructure scaling: scale our infrastructure with more resources to make the platform faster, serve more users and deploy better and bigger models.
• Agentic RAG: the system will route each question appropriately to different pipelines, with different model calls and context fetched and attached. For example, when asking for the conclusion of a paper, we should add the relevant sections of the document to the context and, when asking for a summary, we can follow a map-reduce approach to utilize, summarize and attach the full paper content without exceeding the model's context window size. This would translate into an increase in the number of requests sent to the model, and requires previous infrastructure scaling.
• Graph RAG: What if we combined the power of embeddings with the power of graph databases like Neo4j? That's what Graph RAG consists of. We can consolidate structured (e.g. paper metadata: author, institution, journals, references...) and unstructured information (the ingested paper content) in a graph DB and extract relevant entities from the document bodies (e.g. boson, gluon, radiation, entanglement...), connecting everything together and allowing us to first find a starting node via vector similarity search and then intelligently traverse some neighboring edges to obtain more context. This can be used for certain queries, hard to handle by a plain RAG system, such as "Who is the author with the most publications about quantum entanglement on the Nature journal?", where we can generate a Cypher query to fetch this information. It would also facilitate clustering and topic modeling.

Conclusion and lessons learned

Building effective RAG systems is a complex task, consisting of multiple steps and spanning the entire stack and thus needing:

• Careful consideration of infrastructure needs
• The necessary computing resources and a platform for model serving
• Developing a backend to orchestrate all operations
• Presenting the information to the user in a way that makes their interactions with the system as seamless and effective as possible

Throughout this development, testing has been a crucial constant: evaluating different PDF processing tools and parameters, embedding models, LLMs, prompts and UI/UX ideas. But above all, users themselves are often the best testers. The most effective way to improve such a system is to prototype, release it to a group of users, gather feedback, engage with them, and iterate quickly based on what you learn.

While having team members who are highly specialized in AI/ML is very valuable for this task, it is fundamental to have engineers with a broad understanding of the full stack and application architecture. In many cases, engineers with solid software development skills and a curiosity for AI can contribute significantly to these projects, especially when resources are limited. It’s not about reinventing the wheel, but about understanding which tools are available, which are best suited for the use case, how they work and interact, and how to integrate them effectively. Ideally, a combination of both profiles leads to the best results, but a small team of versatile engineers with initiative can be enough to successfully kickstart a project of this kind.

I hope this post has been interesting or even useful for anyone working on a similar system. Feel free to browse around my GitHub for more details or to send me an email at grcmiguel@outlook.com if you have any comment or question. See you next time!

#ai #llms #rag #software