Two Security Reviews Caught What One Would Miss

The most dangerous finding on a production AI platform was not an exploit. It was a validation path that could fail without raising anything. Authentication functions returned errors as values, callers checked the return type instead of the auth claim, and the exception handlers downgraded security-relevant failures into log lines that nothing was alerting on. Nothing crashed. Nothing paged. Downstream processing simply continued in an unauthenticated context, and the only trace was an auth-success log with no matching token-issued record — a signal nobody was looking for, in a system where the auth layer was assumed to be working precisely because it never complained.

That is the class of defect a penetration test is not built to find and a code review is. The platform in question was a regulated system with real workloads, real users, and real data moving through AI inference pipelines — the kind where a silent failure is not an outage but a compliance event that has already happened. The client engaged an independent penetration-testing firm for an external penetration test and ML LABS for an independent AI-augmented security review and the remediation that followed.

Two tracks, run in parallel, against the same system. Each one found things the other did not, and the interesting part is not that they overlapped. It is where they did not.

Two Tracks, Different Blind Spots

The findings landed in six categories: authentication, input handling, session management, data redaction, network configuration, and infrastructure. ML LABS triaged every finding from both tracks by blast radius, remediated in priority order, and verified each fix individually. Nothing was marked closed until a test passed that had been written to fail against the original vulnerability and to pass only with the fix in place — which is a slower way to close a finding and the only way to know it is closed.

graph TD
    A1["Dual-track<br/>assessment"]
    B1["Triage by blast<br/>radius"]
    C1["Remediate in<br/>priority order"]
    D1["Per-fix<br/>verification test"]
    E1["Verified and<br/>hardened platform"]

    A1 --> B1
    B1 --> C1
    C1 --> D1
    D1 --> E1

    style A1 fill:#1a1a2e,stroke:#e94560,color:#fff
    style B1 fill:#1a1a2e,stroke:#ffd700,color:#fff
    style C1 fill:#1a1a2e,stroke:#0f3460,color:#fff
    style D1 fill:#1a1a2e,stroke:#0f3460,color:#fff
    style E1 fill:#1a1a2e,stroke:#16c79a,color:#fff

The AI-augmented track scanned every file for dangerous patterns, mapped every authentication surface, traced attack chains through the code, checked dependency vulnerabilities, and mapped findings against ISO-27001 and SOC-2 control families. It surfaced a code injection path, the error-as-value authentication problem above, and exception handling that swallowed security-relevant failures — the kind of defect exhaustive scanning is good at and human attention is bad at, because it is boring and it is everywhere.

The penetration test found what scanning cannot: authorization bypass chains that required composing several legitimate operations in an illegitimate order, and protocol-level session forgery that took creative adversarial thinking to even attempt. Neither track was a subset of the other, and the overlap between them produced findings that came from the comparison rather than from either list.

The AI agents caught critical vulnerabilities the human testers missed. The human testers caught critical vulnerabilities the AI agents missed. Neither alone would have been enough.

The split follows a real fault line rather than a marketing one. The Top 10 for LLM Applications (OWASP, 2025) names prompt injection, sensitive information disclosure, and excessive agency as the categories most likely to escape generic security testing — and those are precisely the categories where exhaustive scanning over inference paths beats manual probing. The AI track traced every prompt-construction site, every place a user-controlled string entered model context, and every output-rendering boundary, then cross-referenced each surface against the redaction rules. The pen-test track attacked the same surfaces with input crafted to slip past pattern matchers. The classic web vulnerability classes in the OWASP Top Ten (OWASP) still applied underneath all of it, because an AI platform is a web platform with a model attached.

Fix Authentication Before Everything

Ordering is not a matter of taste when the findings are ranked by blast radius. Authentication gates everything else: a system with flawless input validation is still compromised if an attacker can authenticate as a legitimate user. Data handling determines what leaks when other defenses fail. Network configuration locks the perimeter after the interior is sound. Work it in any other order and you spend the sprint hardening the walls of a house whose front door does not lock.

Credential storage moved to memorized-secret verifiers that resist offline attack. Session exposure was cut to the minimum necessary window. Rate limiting was enforced across every authentication endpoint, and per-organization authentication isolation ensured automated credentials could not be replayed across tenants. The Digital Identity Guidelines (NIST, 2020) treat salted key-stretching, bounded reauthentication windows, and throttled authentication attempts as baseline controls rather than optional hardening — each remediation was aligned to a specific clause, which is what makes a fix auditable instead of merely done.

The silent-failure fix was structural. Every validation path was rewritten to fail explicitly, raise on the failing branch, and force the caller to handle the unauthenticated case. Ambiguity about authentication state was eliminated as a category, not patched as a bug — because the bug was never any single call site. It was a convention that allowed the failure to be quiet.

Invert The Redaction Model

The redaction overhaul was the most consequential change in the engagement, and it is the one most transferable to any other AI platform. The original model was allow-by-default: fields flowed through unless something filtered them. That design fails silently every single time the schema grows — a new field carrying sensitive data ships, nothing blocks it, nothing alerts, and it appears in output surfaces nobody re-audited.

ML LABS inverted the model. Nothing leaves unless it is explicitly marked safe for that output context, and a new field is blocked until someone reviews it. The failure mode flips from silent data leakage to overly aggressive filtering, which is the trade every regulated system should take: the second failure is visible within an hour of shipping, and the first is discoverable only by an auditor or an attacker.

Around it, the perimeter work was ordinary and necessary. Security headers across all API responses. CORS narrowed to specific production origins. Open redirects closed. Infrastructure configuration consolidated into a single source of truth with automated drift detection, so that manual changes cannot quietly weaken the perimeter after the hardening ends — the control families a regulator asks about (NIST, 2024) assume exactly that kind of enforcement, and a WAF in front of it all is the outer layer, not the strategy.

What A Sprint Cannot Fix

A hardening sprint is a point-in-time pass, and its results decay from the day it ends. Every new endpoint is an authorization surface nobody has reviewed. Every schema change is a redaction decision made by whoever wrote the migration. Every infrastructure edit made outside the pipeline is a hole in a perimeter that was verified once, under different assumptions, by someone who has moved on. Systems do not fail because a control was missing on day one — they drift into failure (Dekker, 2011) through a series of individually reasonable decisions, none of which looked like a security decision at the time.

That is the boundary condition on this entire engagement, and it is worth stating plainly rather than selling around: the sprint closed the findings, and the sprint cannot keep them closed. What keeps them closed is someone who holds the whole security picture and reviews every change against it — the same argument that governs oversight of agents already running in production, and the reason regulated AI delivery is a discipline rather than a phase.

First Steps

1. Run the two tracks in parallel, not in sequence. Exhaustive code scanning and adversarial human testing fail in opposite directions. Running them together is what surfaces the findings in the gap between them; running one is a coverage decision you are making without meaning to.
2. Grep for the quiet failure. Find every place authentication or authorization returns a value the caller can ignore, and make it raise. Then find every exception handler that logs and continues, and ask what it is allowed to swallow.
3. Invert one redaction surface this week. Pick your highest-exposure output context and switch it to deny-by-default. The noise it makes on day one is the leakage you could not see on day zero.

Keep The Security Picture Owned

The vulnerability classes that break production AI platforms are known, and so are their fixes. The gap is execution discipline, not knowledge — and the structure that produces it is concentrated ownership: one operator holding the whole picture, rather than findings distributed across a feature team where nobody sees the shape they make together. Every fix ships with the test that would have caught the original defect. Every finding is mapped to a control family. The specialist auditor and the AI-augmented operator each do what they are actually good at, and the path from pilot to production runs through both of them.

Somebody has to hold that picture after the sprint ends, and that is what managed AI operations is: an accountable owner for a defined scope of live systems, month over month, cancellable on 30 days' notice — so the arrangement survives on the strength of the work rather than on the length of the contract. If your platform handles sensitive data, has an AI inference path, and has never had a structured security pass, the finding you should worry about is the one that has been sitting quietly in production, not raising anything, since the day it shipped.

References

1. National Institute of Standards and Technology. Digital Identity Guidelines: Authentication and Lifecycle Management (SP 800-63B). NIST, 2020.
2. OWASP Foundation. Top 10 for LLM Applications. OWASP, 2025.
3. Open Worldwide Application Security Project. OWASP Top Ten. OWASP.
4. National Institute of Standards and Technology. Security and Privacy Controls for Information Systems and Organizations (SP 800-53 Rev. 5). NIST, 2024.
5. Dekker, S. Drift into Failure. Ashgate/Routledge, 2011.