I Left an AI Security Agent Running Overnight. Then I Read the Report.

I left an AI pentesting agent running overnight against OWASP Juice Shop, powered by a local model. By morning, I had a report and a more basic question: was this actually a report about my target?

I wanted to see how far a local AI agent could get through a pentest without me babysitting it.

I looked for an AI pentesting agent and found Strix, which seems like the perfect candidate for the job.

Eventually, I want this to support my pentesting work: run assessments regularly, then verify the findings myself.

That future seems so close, yet so far.

01 The setup

In my local lab I have an AI rig consisting of:

• an older Intel i7
• 32 GB of RAM
• SATA SSD & HDD to store the models and VMs
• RTX 5060 Ti 16 GB

The machine is an old video-editing PC with a recently upgraded GPU. It runs Proxmox, with three VMs involved in this experiment:

• holocron provides the models
• nazgul runs the hacker agent
• The OWASP machine features a vulnerable web app for my AI to hack

The names are dramatic, but the hardware budget isn't.

02 Picking the brain

Before letting Strix loose on Juice Shop, I tested the models on holocron in isolation. I wanted to compare speed, VRAM use and if they would support tool calling reliably.

So I ran some benchmarks against the models (more detailed benchmark results can be found here):

Based on those results, I started with Qwen3.8-27B-Uncensored at IQ2_M quantization, served locally through Ollama. Even though the benchmark showed the IQ3_M quantization as the winner, I wanted a faster model to run a quick test.

Now I was able to let nazgul tear apart the Juice Shop!

Or at least, I wanted to...

Failing upwards

I connected Strix on nazgul to Ollama on holocron through an SSH tunnel. But it immediately started to fail with litellm.APIConnectionError. At first glance, it looked like a connection problem: a refused connection, a timeout or maybe a stalled request.

After inspecting the full traceback, I was able to identify the more specific JSONDecodeError: Extra data.

LiteLLM was failing while converting the request for Ollama. That narrowed the problem down, but it didn't prove where the invalid data had originally come from. Luckily it had an easy fix: pointing Strix at Ollama's OpenAI-compatible endpoint got me past that failure. But it didn't take long for the agent to find a new way to fail on me: Agent exhausted recovery attempts without calling finish_scan or agent_finish.

The agent would announce what it intended to do, without actually doing it. After 50 recovery attempts it gave up with the aforementioned message.

I tried switching to a less lobotomized model variation (IQ3_M) and three hours later, it was still running without another crash. I left it overnight, hoping for a comprehensive report in the morning.

03 The pentest report

The next morning, coffee in hand, I opened the report:

23 findings. An old Django application. References to source files and work supposedly performed by subagents.

But...

My target was Juice Shop. I hadn't provided a source tree to analyse. My machine can't handle subagents so I disabled them.

So what the heck did the agent do?

I compared the report with the scan archive.

The configured target was correct. The findings in the archive contained zero vulnerabilities and zero recorded surfaces. There was only evidence of one (root) agent. Some requests even went outside of the intended lab scope. The data did not at all support the claim of 23 findings.

Separately, the logs showed a curl request to a public Vercel-hosted address outside my lab. The response was HTTP/1.1 404 Not Found, with Server: Vercel and a DEPLOYMENT_NOT_FOUND error.

That doesn't demonstrate a successful scan or exploit. It does show that an actual request went out. My target configuration hadn't confined the agent's network activity to Juice Shop.

I couldn't use the report. And the unexpected network activity gave me a second problem to fix.

04 Takeaway

This run told me that this particular setup is not working at all like intended. The report looked convincing, but the data didn't support it.

I'm not sure what the cause of this is and it could range from the selected model and quantization, a too-short context length, how the agent integrates with the model or maybe I need to run a bigger/different model altogether.

I learned that a running agent isn't necessarily proof of progress and a completed scan isn't proof of an actually performed and completed pentest.

This experiment gave me a concrete question for my next one: what evidence would I need before putting my name on an AI-assisted security report?

05 Next experiment

While this time I was mainly focused on getting the system to work, for my next experiment, I'll keep the target fixed and change one component at a time.

First, a short test: can the agent reach Juice Shop, identify one vulnerability, and produce evidence I can use to reproduce it?

I'll also define and enforce which network destinations it can reach, instead of relying on the target setting alone.

If it can get through that short "smoke test", it has earned another all-nighter of hacking into Juice Shop.