Reformulating User Queries for Improved Retrieval in RAG Systems

Co-Supervised by: Tobias Steiner

If you are interested in this topic or have further questions, do not hesitate to contact tobias.steiner@unibe.ch.

Background / Context

Retrieval-Augmented Generation (RAG) is widely used to provide language models with additional information from external document collections. By retrieving relevant documents and adding them to the model context, RAG systems can improve the factuality and reliability of generated answers.

However, retrieval quality strongly depends on how well the user query matches the documents in the collection. This can be particularly challenging for historical archives, where users may formulate queries in modern language while the underlying documents use different terminology, spelling, languages, or bureaucratic writing styles.

Query rewriting or query expansion can be used to address this problem. A language model reformulates the original user query before it is passed to the retrieval system, with the goal of making the query more suitable for the underlying retrieval method. For example, a query could be expanded with historical terminology for keyword retrieval such as BM25, or reformulated to better match the representation learned by an embedding model.

This project investigates different approaches for reformulating user queries and evaluates whether query rewriting can consistently improve retrieval performance in RAG systems operating on historical documents.

Possible Research Questions

Can query rewriting consistently improve retrieval performance compared to using the original user query?
Can user queries be reformulated specifically for BM25 keyword retrieval, for example by generating additional keywords, historical terminology, or relevant entities?
Can query rewriting improve retrieval performance when using neural embedding-based retrievers?
Can a model learn when to rewrite or expand a query and when it is better to use the original query unchanged?
What are the common failure modes and limitations of query rewriting and query expansion?
How do different rewriting strategies affect retrieval across different languages, query types, and document characteristics?

Approach / Methods

The student will implement and experimentally evaluate different approaches for reformulating user queries to improve document retrieval.

Possible approaches include:

Training and/or evaluating language models that transform natural-language queries into BM25-oriented keyword queries.
Training and/or evaluating models that reformulate queries to better align them with an existing embedding-based retrieval model.
Comparing different query expansion and rewriting strategies, including prompting, rule-based approaches, and learned models where appropriate.
Investigating whether a model can learn to select between the original query and a rewritten query.
Evaluating the approaches using an existing retrieval benchmark and comparing them against the original queries.
Analyzing successful and unsuccessful retrieval cases to identify failure modes of query rewriting.

The target domain consists of historical documents from the Swiss Federal Archives, focusing on Swiss foreign policy between 1848 and 1993. The extracted document text is noisy and contains OCR and layout artefacts. The documents are written in German, French, Italian, and English, and their language and terminology are influenced by the historical period and bureaucratic writing styles.

An existing retrieval system and evaluation framework are available as a starting point for the project. The benchmark consist of synthetic Question-Answer-Document triplets (with their own limitation). Initial experiments have shown that query rewriting is not straightforward and that improvements are not guaranteed. The project therefore involves both implementation and experimental investigation of why and when query reformulation works. An alternative dataset in the historical domain is also possible, but the student must propose it.

Expected Contributions / Outcomes

The project is expected to provide:

A systematic benchmark of at least four different query rewriting or query expansion approaches on the given historical document collection.
An experimental evaluation of how query reformulation affects retrieval performance for different retrieval methods, such as BM25 and embedding-based retrieval.
An analysis of the strengths, weaknesses, and failure modes of query rewriting for historical and multilingual documents.
An investigation of when and why query expansion helps retrieval and when the original query performs better.
An analysis of factors that influence successful query reformulation, such as query type, language, terminology, OCR noise, and document characteristics.
An open implementation and reproducible experimental framework for query rewriting and retrieval evaluation.

Required Skills