OpenAI's Cyber-Eval Disclosure Named the Thing We've All Been Ignoring
OpenAI's disclosure of third-party cyber-evaluation incidents isn't a scandal about frontier models—it's proof of a gap we're all facing. Authorized evaluators with legitimate access extracted capabilities and discovered jailbreaks during "safe" security testing. The pattern is clear: when you give testers production-adjacent infrastructure and credentials, you're handing them the keys to the kingdom, contract or not. And if it happens at a lab with resources to monitor millions of tokens, it's happening faster at shops like ours shipping agents into production.
The generalization is straightforward. Every team outsourcing red-teams to Scale AI, academic partners, or bug-bounty firms is running this same risk. Your evaluators—authorized or otherwise—are now part of your attack surface. We've gotten this wrong before. The instinct is to treat evaluation like QA: a necessary chore you outsource to keep costs down. But evaluation infrastructure is a hostile-tenant security boundary, whether you call it that or not.
The Attack Surface Isn't the Model — It's Everything Around the Prompt
Model weights matter less than what surrounds them. An evaluator with access to your staging agent can extract capabilities through leakage points you probably haven't secured. System prompts and tool schemas appear in error messages, in 4xx responses, sometimes in the agent's own reasoning traces. That's passive extraction. Active extraction is worse: the evaluator probes your retrieval indexes, your Postgres replica, your HTTP egress, your sandboxed code execution—everything an agent can touch, an evaluator can touch, and everything they touch is a potential exfil vector.
Think through your own stack. RAG corpora, vector stores, Postgres schemas, Redis keys, API credentials—if an evaluator has a tool-call surface that reaches them, they can use your agent as a probe. Logs and observability dashboards like LangSmith, Langfuse, or Braintrust become secondary exfil paths. We've seen teams share prod API keys across staging and eval environments, reuse credentials with prod scopes, and point eval runs at live retrieval indexes. The OWASP LLM Top 10 categories that matter here are prompt injection, sensitive information disclosure, and excessive agency. All three are baked into the eval infrastructure most teams run today.
Separate Eval Tenants Aggressively — Down to the Credential
Isolation starts with hard boundaries. One AWS or GCP project per evaluator engagement, never shared with staging or prod. Separate OpenAI and Anthropic org IDs and API keys for each evaluator, with per-key spend caps and rate limits that'll fire an alarm if a jailbroken agent tries to blow through the budget in an hour. This matters because the first sign of a successful jailbreak is often a cost anomaly.
Tool scopes get downgraded aggressively. Postgres read-only replicas instead of prod databases. Mocked HTTP egress so a jailbroken agent can't call out to an attacker-controlled webhook. Sandboxed code execution via Modal, E2B, or Fly Machines—never shared filesystem, never persistent state across runs. Vector store namespaces per tenant; never point an eval at your prod Pinecone or pgvector index. Network-level egress allowlists so even if the agent breaks containment, it can only reach services you've pre-approved. When the engagement closes, rotate everything—treat it like an incident response. New keys, new projects, new credentials, from end to end.
Instrument the Eval Loop Like You'd Instrument a Breach
Observability in an eval loop has to be adversary-aware, not just accuracy-aware. Log every eval prompt and every tool call to an append-only audit stream separate from prod telemetry. Detection signals matter: sudden shifts in prompt-token distribution, repeated near-duplicate probe sequences, tool-call patterns that don't match the user personas you're testing. We've shipped per-decision safety filtering on Nemotron; the same discipline applies here.
Plant canary tokens in system prompts and RAG documents so you can detect extraction after the fact, even if you didn't catch it in real time. Budget alarms—sudden cost spikes are a jailbreak signal. Every eval run must be reproducible from audit logs alone, no gaps. And when a canary fires or an alarm goes off, page your team, not the eval vendor. Make the incident yours to own.
Rewrite the Evaluator Contract Before You Sign It
The legal and scoping work is half the defense. Scope documents should enumerate allowed tool surfaces, not just allowed prompts. Borrow rules of engagement from pentest contracts: disclosure windows, no lateral movement, no data retention past engagement close. Put in an explicit clause that extracted capabilities and successful jailbreaks are confidential findings, not publishable artifacts without your review and approval.
Nail down data handling: where do eval transcripts live, who has access, who trains on them, how and when are they destroyed. Before you sign with any eval vendor, ask these questions: Can you guarantee this eval tenant is isolated from your other client work? What's your incident response if we find unauthorized access? Do you run canary tokens or pattern detection on your eval runs? What's your data retention policy? Who owns the rights to jailbreaks we find? These aren't nice-to-haves. As more of the industry outsources evals to third parties, they're table stakes.
What This Changes About How You Ship the Next Agent
Eval-as-attack-surface reshapes your pre-launch checklist. Add a hard gate: Can you point an adversarial evaluator at this agent without touching prod infrastructure? If the answer is no, don't ship yet. Capability minimization is your friend—every tool you don't expose is a jailbreak surface you don't have to defend. We've found the cost of retrofitting isolation two weeks before launch is brutal. Build it in from day one.
This sits alongside RAG failure modes in production and per-decision safety filtering as part of your broader agent reliability story. We help teams stand up isolated eval tenants with separate credentials, instrument audit streams, and red-team harnesses before external evaluators get access. If you're shipping an agent and want to walk through your eval setup before your next red-team engagement, reach out to us at /contact.
First Week: The Checklist
Spin up an isolated eval tenant with its own API keys and project IDs. Scrub all prod tool credentials from eval scopes—downgrade to read-only, sandboxed, mocked. Log every eval prompt and tool call to a separate append-only audit stream. Plant canaries in system prompts. Set budget alarms. If you're running external evals in the next month, audit your contracts against the questions above and close the gaps before signatures. The goal isn't perfect security—it's making sure that when (not if) an evaluator finds something, you find it first.