Designing a fair benchmark for local security agents
A practical test plan for comparing local models on instruction following, tools, scope control and repeatability.
A model that answers security questions well is not automatically a useful security agent. The benchmark has to measure the entire loop: instructions, tools, evidence and restraint.
I am currently comparing local models for a self-hosted security workflow. Early tests already showed why a single aggregate score would be misleading: some candidates follow instructions cleanly but fail native tool calls; others call tools successfully and then lose track of scope.
01 The question
The useful question is not which model is smartest in the abstract. It is which model can complete a bounded security task reliably on the hardware and context window I can actually operate.
A useful agent completes the assigned task, uses tools predictably, preserves evidence, respects scope and makes failure visible.
02 Four layers to measure
- Instruction following: does the model preserve the objective and constraints?
- Tool use: can it select tools, construct valid arguments and recover from errors?
- Security workflow: can it form hypotheses, gather evidence and distinguish findings from noise?
- Operational cost: how much time, memory and manual intervention does a run require?
A benchmark should make failure modes comparable—not hide them behind a winner.
03 Why use two vulnerable targets
OWASP Juice Shop and WebGoat exercise different parts of the workflow. Juice Shop is useful for broad, application-shaped exploration. WebGoat is better for controlled lesson-level checks. Running both helps separate genuine agent capability from familiarity with one target.
04 What every run should record
model: candidate-name
target: juice-shop
objective: bounded-task-id
tool_calls: []
evidence: []
violations: []
manual_interventions: 0
elapsed_seconds: 0The next step is to run the same task set through the strongest inference candidates, preserve every transcript and score only claims supported by evidence. I will publish the method and the ugly failures alongside the winners.