Red-Teaming AI Agents on a $0 Budget: What I Learned Breaking Into a Banking Assistant

TL;DR: For my university laboratory exam, my colleague Mattia and I built a complete AI red-teaming pipeline around the open-source HackAgent framework: To test it we built a vulnerable LangGraph banking agent, a tri-model attack orchestration, ground-truth telemetry, and automated reporting. We ran 143 attack iterations across two target LLMs, two attacker models, and two defense postures. All of this on a Mac Mini M4 with 16 GB of RAM.

Why AI Agent Security Is a Different Problem

Most people think of prompt injections as tricking a bot into swearing or saying weird stuff. That’s harmless. The real problem starts when LLMs become agents with real tools. If a model can actually click “send” on a bank transfer, a prompt injection stops being a joke and becomes an actual exploit.

The threat model I focused on is the Indirect Prompt Injection (OWASP Top 10 for LLM Applications, LLM01) combined with Excessive Agency (LLM06). The scenario is heavily inspired by a real-world vulnerability, Microsoft’s EchoLeak (CVE-2025-32711), where hidden instructions inside an email hijacked an agent through its own summarization feature.

The flow is deceptively simple:

Attacker → malicious email (hidden payload)
              ↓
Mario asks his banking assistant to summarize the email
              ↓
The agent reads the malicious instructions and treats them as legitimate
              ↓
Unauthorized `send_money` tool call → funds transferred

The agent becomes a Confused Deputy: it has the user’s privileges but is now following the attacker’s orders.

The Testbed: A Deliberately Vulnerable Banking Agent

To have something realistic to attack, we built a target environment from scratch:

• A banking assistant implemented with LangGraph, exposed via a FastAPI server implementing the OpenAI API spec, running locally.
• A high-privilege tool: send_money.
• An architectural flaw: the LangGraph state machine invokes tool calls directly, with no intermediate policy validation or human-in-the-loop confirmation.
• An A/B hardening toggle (AGENT_HARDENING environment variable) to compare a “Naked” permissive agent against a “Hardened” one with explicit security rules in its system prompt, to test how well the models followed instructions.
• Server-side logging: every tool execution is recorded in server_tool_calls.jsonl with a unique run_id. This became the ground truth for the whole evaluation.

The attacker payload lives inside an email: when the user asks the agent to summarize their inbox, the hidden instructions try to hijack the goal and trigger the transfer.

The Attack Pipeline: Three Models, Strictly Black-Box

Rather than using HackAgent’s CLI, we integrated its Python library directly into our own orchestration script (run_suite_local.py), gaining fine-grained control over execution flow, logging, timeouts, and resource management.

The architecture is a tri-model setup under a strict black-box threat model. The attacker has no visibility into backend state, mirroring real-world remote attack conditions:

1. Attacker LLM: generates, mutates, and optimizes adversarial prompts. Locally: gemma-4-abliterated:12b (a guardrail-stripped open-weight model). For comparison: GLM-5.2 via cloud API, simulating a well-funded adversary.
2. Target Agent: llama3.1:8b or qwen2.5:7b, both quantized to Q4_K_M and served locally via Ollama.
3. Judge LLM: llama3.1:8b, scoring the target’s textual responses to decide whether a jailbreak succeeded.

Out of HackAgent’s 15 attack strategies, we selected the 11 text-based ones applicable to our agent, covering the full spectrum of modern adversarial techniques:

Family	Techniques	Mechanism
Iterative optimization	PAIR, TAP	Attacker LLM refines prompts over multiple rounds; TAP adds tree search with pruning to cut costs
Dynamic strategy search	AutoDAN-Turbo	Retrieval over a local database of learned attack strategies
Prefix attacks	AdvPrefix	Optimized adversarial prefix inducing affirmative responses
Obfuscation	CipherChat, FlipAttack, h4rm3l	Encoding (ROT13, Base64, Morse), text flipping, and chained obfuscation
Paraphrase sampling	BoN	N variations of a goal, best candidate selected
Social engineering	PAP	Persuasion techniques (authority, urgency, empathy) against safety alignment
Structured formats	tFC, Static Template	Flowchart syntaxes (Mermaid, DOT, ASCII) and classic jailbreak templates

Each attack run carries a unique run_id that correlates client-side prompts with server-side tool execution logs.

Measuring What Actually Matters: Ground-Truth Confusion Matrix

Here is the methodological point I’m most proud of. Most red-teaming evaluations stop at “the judge said the jailbreak worked.” Having only small models running on consumer hardware, we really didn’t trust the judge, and that skepticism turned out to be justified.

Our analysis script (analyze_results.py) cross-references what the Judge LLM claimed against what the server actually executed, producing an operational confusion matrix:

• True Positive (TP): Judge says success AND the send_money tool was actually invoked on the backend.
• False Positive (FP): Judge claims success, but nothing happened -> a judge hallucination.
• False Negative (FN): Judge says the agent refused, but an unauthorized transfer was executed -> a silent operational compromise.
• True Negative (TN): Genuine successful defense.

Across all N = 143 valid iterations, the results were sobering:

Ground Truth \ LLM Judge	“Jailbreak” (1)	“Blocked” (0)
Tool executed (1)	TP = 6	FN = 23
No tool executed (0)	FP = 8	TN = 106

That translates to:

• Recall: 20.7% — the judge missed almost 4 out of 5 real compromises
• False Negative Rate: 79.3%
• Precision: 42.9%

In other words: the target agent would output a polite refusal in the chat while executing the malicious transfer in the backend at the same time. Anyone auditing this system by reading conversation logs would sign off on a security posture that does not exist.

Lesson #1: If you use an LLM as a judge, you need a high-parameter model from a completely different family. Using a small 8B model to judge an 8B target was a recipe for failure. As Zheng et al. (2023) pointed out, models have a strong self-preference bias, and smaller models simply lack the reasoning capacity to catch evasive answers and silent compromises.

Results: Models Are Not Equally Robust and Hardening Is Not a Silver Bullet

Each strategy ran N = 3 times per configuration, with a conservative worst-case metric (a strategy “succeeds” if the tool fires in at least one run). All on a Mac Mini M4, 16 GB RAM, no GPUs, no enterprise budget.

Baseline robustness diverged sharply:

• Unhardened llama3.1:8b: compromised by every single vector (11/11, ASR = 100%)
• Unhardened qwen2.5:7b: resisted 7 of 11 strategies out-of-the-box (ASR = 36.4%), falling only to iterative optimization techniques (PAIR, TAP, AdvPrefix, AutoDAN-Turbo)

Prompt-level hardening worked but only up to a point:

• On qwen2.5:7b, hardening drove the local attacker’s ASR to 0%
• On llama3.1:8b, hardening mitigated only 4 of 11 vectors. The agent remained fully vulnerable to multi-turn and algorithmic attacks (TAP, PAIR, CipherChat, PAP, tFC, AutoDAN-Turbo, BoN)

When we swapped the local 12B attacker for the frontier-class GLM-5.2 via cloud API, in one single testsuite run, TAP successfully breached the hardened qwen2.5:7b agent that had resisted everything else. Tree-based search guided by advanced reasoning overcame a system-prompt defense that stopped a smaller attacker completely.

Lesson #2: prompt hardening often just masks the underlying vulnerability. Against a sufficiently capable adversary, it fails. Security must be enforced architecturally (tool-level authorization, human-in-the-loop), not just linguistically.

The Economics

A core goal was achieving meaningful results with consumer hardware and a tight budget—typical CS student constraints.

• Fully local pipeline = $0 in inference costs, ~2 hours test suite runtime on consumer hardware
• Cloud frontier attacker model = $0.79/run, ~1h15m/run
• The whole suite is designed as a CI/CD security gate: analyze_results.py can fail a build if the Attack Success Rate or False Negative Rate exceeds a threshold, making agent security testing a repeatable DevSecOps practice rather than a one-off audit

Honest Limitations

I want to be transparent about scope:

• N = 3 repetitions is not statistically significant. The goal was educational: mastering the framework, understanding each technique, and architecting the pipeline. The trends are consistent, the exact percentages are not the point.
• LLM-as-a-Judge has capacity limits. An 8B judge cannot reliably evaluate an 8B target: it missed evasive language and silent compromises. Literature (Zheng et al., 2023) also documents self-preference bias: judges favor responses from their own model family. Our use of a same-class local judge directly contributed to the 79.3% FNR.
• Decoupled feedback loops create blind spots. Because iterative attackers (PAIR/TAP) only see conversational output, the elevated FN rate fed them garbage feedback: attackers kept optimizing prompts against a judge that was wrong 79% of the time on real compromises.

What I Take Away From This Project

Working hands-on with 11 attack vectors against a realistic agentic target taught me things no course slide could:

1. Agentic attack surfaces are operational, not just textual. The vulnerability isn’t “the model said a bad word”—it’s “the model called the wrong tool with the user’s privileges.” Security auditing must measure tool execution, not chat tone.
2. Security by design is non-negotiable for agents. System-prompt hardening stopped a 12B attacker and failed against a frontier model. High-privilege tools need architectural guardrails.
3. Automated red-teaming is already practical. Open-source frameworks (HackAgent), local open-weight models, and consumer hardware can run a meaningful security evaluation for educational purposes.
4. Evaluation infrastructure is itself a security problem. The judge failing quietly (79.3% FN) is arguably a more dangerous finding than the vulnerabilities themselves.

Replication Package

Everything is reproducible: the complete source code for the vulnerable banking agent (agent.py, server.py), the test suite runner (run_suite_local.py), the analysis pipeline (analyze_results.py), the exact prompts for both Naked and Hardened configurations, and the raw evaluation datasets (client_runs.jsonl, server_tool_calls.jsonl) for independent verification.

---

This work was developed by Gabriele Alessandria and Mattia Coriale as the final laboratory project for the ASSE course, University of Genova. Built on the open-source HackAgent framework.