EVALUATION GUIDE
How to evaluate an LLM firewall.
A field guide for security and AI platform teams: define the boundary, test the behavior, and measure the operational result.
DEFINE A DECISION, NOT A DEMO
Evaluate the boundary
your applications actually use.
A useful evaluation connects detection results to the workflow, data, and operational constraints your organization needs to protect.
Start with one representative AI workflow and write down the question the evaluation must answer. Examples include reducing provider-bound secrets, identifying agent drift, or reducing input tokens without harming task completion.
Choose success criteria before running the test. Use approved benign examples alongside malicious or out-of-scope cases, and reserve examples that are not used to tune the configuration.
Keep a record of model versions, client versions, detector configuration, enabled transformations, and traffic routing. Without those details, comparisons are difficult to reproduce.
FIVE EVALUATION STAGES
A practical sequence.
Confirm visibility and compatibility
Exercise normal prompts, multi-turn conversations, tool loops, streaming, provider errors, and timeouts. Confirm that the session contains the evidence needed to assess each case. Record traffic or modalities that bypass inspection.
Review detection in shadow mode
Replay approved attack examples and ordinary work. Count false positives as well as detected threats. Separate unscanned or failed-open requests from genuine clean results; a scan that did not run is not a detector decision.
Test the actual enforcement point
For each blocking or masking policy, record when the detection becomes available, which bytes or actions are affected, and what the client receives. Test live session-correlation enforcement and historical replay findings separately.
Measure optimization and operational cost
Compare token usage, task correctness, error rate, throughput, and latency with and without selected transformations. Include retries, output tokens, provider caching, and any AI-review inference when interpreting economics.
Practice ownership and recovery
Confirm who can inspect original content, change policies, authorize MCP clients, and promote detector updates. Exercise failure behavior, revocation, backup and restore, and rollback before increasing scope.
Keep a scorecard that explains the result.
| Dimension | Evidence to capture | Decision it informs |
|---|---|---|
| Coverage | Inspected requests, complete tool loops, unsupported paths, and scan failures. | Whether the workflow is actually inside the security boundary. |
| Detection quality | True and false positives, missed cases, severity, and matched evidence. | Which signals are ready for enforcement and which need tuning. |
| Policy timing | Detection timestamp, forwarding or streaming behavior, and visible client result. | Whether the configured action can prevent the specific exposure. |
| Data handling | Original and transformed content, evidence access, retention, and deletion paths. | Whether the deployment meets the organization’s data requirements. |
| Task quality & cost | Input tokens, task checks, retries, latency, and full provider charges. | Whether optimization improves economics without unacceptable quality loss. |
| Operations | Load results, dependency failures, recovery tests, and update rollback. | Whether the team can operate the chosen topology. |
COMMON MISREADINGS
Be careful what
a successful test proves.
A replay is evidence for the tested reconstruction.
It is not automatically a measure of recall across all attacks, proof of live incident detection, or evidence that the original action would have been blocked.
A blocked example is not universal coverage.
Validate the formats, directions, encodings, and client behaviors in scope. Include benign near-misses so accuracy is not improved by simply blocking legitimate work.
Fewer tokens are not the complete business result.
Check task success and the final bill. A change that saves input tokens but creates retries or incorrect outputs can lose its apparent benefit.
See how we frame replay evidenceINVITE-ONLY BETA
Bring your workflow. Define your evaluation.
Tell us what you use, what you need to protect, and where Milgram would run.