What this project is about
RAGTrust MiniBench examines a failure before a language model writes an answer. Systems that retrieve supporting documents can rank an attacker-controlled document highly because it repeats the user's search terms. Here, the attacker may add a document through a low-trust community source but cannot change trusted documents, user queries or source trust scores. The goal is to get malicious instructions included in the material supplied to the model.
The local test contains 18 synthetic documents and 12 queries. Six queries have a targeted document containing attack instructions; six test normal retrieval. The instructions request harmless marker text, and no external model is called. The project measures exposure to malicious documents, not whether a language model would follow their instructions.
How it works
- Trusted + untrusted documents
- Keyword ranking / trust-aware reranking
- Attack exposure + useful retrieval
Make ranking transparent
The retriever splits titles and document bodies into search terms, removes common filler words and counts matches with the query. Each term contributes at most three occurrences, limiting the effect of repetition. Results retain both relevance and final scores so a reviewer can inspect why a document moved.
Account for source trust
The defence multiplies relevance by a supplied trust score for the document's source. Text matching instruction-like phrases receives a further 0.25 multiplier, ignoring capitalization. This provenance-aware re-ranking uses explicit rules; it is not a learned classifier or a reliable detector of every hidden instruction.
Check all returned documents
Evaluation measures how often an attack document ranks first, whether one appears anywhere in the top three, and whether the expected trusted document is retained. It also reports average source trust. An application may pass all retrieved documents to a model, so a safer first result can still leave malicious content in its input.
The important design choices
Keep the mechanism inspectable
Use capped keyword counts rather than numerical text representations that estimate similarity in meaning.
The trade-offThe ranking and attack are easy to inspect, but this simple method is not representative of all modern retrieval systems.
Control the source scores
Give each source a supplied trust score that the attacker cannot alter, then add a penalty for suspicious phrases.
The trade-offThis isolates source-aware ranking but does not test forged trust information, unknown sources or attacks that reword their instructions.
Look beyond the first result
A low-trust document repeats search terms and instructs the model to output harmless marker text.
- Inspect its position under the ordinary keyword ranking.
- Apply source-trust weighting and the suspicious-phrase penalty.
- Review all three returned documents, not only the first.
What this showsThe review separates better ordering from removal of malicious material, which require different measurements and potentially different controls.
Illustrative example, not an additional measured result.
How it was tested—and what it shows
The checks
The saved results come from a local experiment that returns the same output for the same inputs. The dashboard compares ordinary and defended rankings on identical documents and queries. Clean recall—whether the expected trusted document is retained—is checked alongside attack exposure. Demoting malicious content is not useful if the relevant trusted material also disappears.
What to take from the results
For attacked queries, the proportion with a malicious document ranked first falls from 100% to 0%. The expected trusted document remains in the top three for 100% of queries. However, a malicious document still appears somewhere in the top three for 100% of attacked queries. The defence changes ordering, not complete removal. A model reading all three results remains exposed; answer-level safety was not measured.
What to keep in mind
- The documents and attack markers are synthetic.
- Only a narrow keyword-stuffing strategy is tested; this is not complete protection for an AI search system.
Technical notes & source
Built with
Python · retrieval · provenance signals · pytest
Terms used in this project
- RAG
- A system that retrieves documents to help an AI answer.
- Provenance
- Information about where a document came from.