What this project is about
AgentGuard Lab investigates indirect prompt injection: instructions hidden in external content that try to redirect a model away from the user's task. The model receives a legitimate request and less-trusted content, then returns an answer and optionally one local tool action. The attacker controls that content, not the user's request or the application's permission rules. The experiment asks what each defence blocks and what useful behaviour it also prevents.
This is a single-step educational study, not a test of an autonomous agent completing long workflows. Its fixed challenge contains 12 synthetic attacks and ten harmless comparison cases, including legitimate calendar tasks. Attacks impersonate trusted roles, insert unwanted answers, encode or translate instructions, redirect workflows, or try to break out of content boundaries. The cases were written by the same author with knowledge of the system, not independently designed without that knowledge.
How it works
- Compare each defence
- Keep tool effects isolated
- Show more than success rates
Compare each defence
Five configurations compare no defence, prompts marking content boundaries, a detector that removes suspicious content, application-side tool permissions, and all three together. This reveals individual mechanisms instead of reporting only an all-controls-on result. Because not every possible combination is tested, the study cannot identify every interaction between controls.
Keep tool effects isolated
Every case starts with fresh fictional calendar data in a temporary environment. Answer attacks are scored by an exact text match, ignoring capitalization; tool attacks are scored by their local effect. Model responses and action parameters are checked for valid structure. Invalid decisions remain failed measurements, not successful defences.
Show more than success rates
The runner saves each attempt, raw decision and a record of model versions, software versions and file hashes that identify the code and dataset. Analysis counts detector outcomes separately and compares cases with valid responses from both configurations. It also estimates variation by repeatedly resampling whole cases and shows best/worst outcomes for missing responses.
The important design choices
Keep attacks contained
Use synthetic content and fictional calendar actions.
The trade-offThis permits inspectable experiments without touching real accounts, but cannot establish resistance to real data theft or complex multi-step workflows.
Report missing responses
Separate failures to obtain a valid model response from attack outcomes, showing best- and worst-case assumptions for missing results.
The trade-offThe result is less visually simple than one percentage, but avoids treating a model that failed to answer as a secure model.
An unwanted calendar change
A synthetic document embeds instructions that conflict with the user's requested task.
- Run the same case with fresh fictional calendar data in each defence configuration.
- Inspect the returned answer and requested action separately.
- Compare whether the action was blocked and whether the legitimate task still completed.
What this showsThe comparison separates unwanted answer changes, blocked tool actions and interference with legitimate work instead of collapsing them into one label.
Illustrative example, not an additional measured result.
How it was tested—and what it shows
The checks
The hosted run scheduled 660 requests: two models, five configurations, 22 cases and three repetitions. Ministral produced 324 of 330 valid decisions; Qwen produced 221 of 330. Repeating one attack does not create new independent attack cases, so uncertainty calculations keep repetitions grouped by case. Runs with a simulated model check the software workflow, not real language-model behaviour.
What to take from the results
Among valid responses, Ministral attack success fell from 9.1% without defence to 0% with combined controls; Qwen fell from 31.6% to 4.0%. Qwen had many missing or invalid responses, so these percentages use different numbers of successful measurements and cannot be read alone. Evaluated separately, the detector caught five of 12 attacks and flagged two of ten harmless cases. These small synthetic results do not provide a security guarantee.
What to keep in mind
- The cases are synthetic and small, and the author knows how the attacks were constructed.
- A result in this lab is not a guarantee that an assistant is secure in a real deployment.
Technical notes & source
Built with
Python · Streamlit · pandas · pytest
Terms used in this project
- Prompt injection
- An attempt to make an AI follow instructions hidden in material it was meant to read.