KushoAI, an AI-native API testing released APIEval-20, an open benchmark for evaluating whether AI agents can generate tests that catch real bugs in APIs given only a request schema and sample payload: no source code, no documentation, no additional context.

Analysis of 1.4 million AI-driven test executions across 2,616 organizations shows that authentication failures account for 34% of API outages and 41% of APIs experience undocumented schema changes within 30 days, yet no standard existed for measuring whether AI agents could detect these failures systematically. APIEval-20 extends the benchmark tradition established by HumanEval for code generation and SWE-bench for bug fixing, applying the same rigour to API testing.

Abhishek Saikia, Co-Founder & CEO, KushoAI, said, "Every vendor selling AI-powered API testing uses the same language: schema validation, payload fuzzing, bug detection. There has been no shared reference point for what any of that means in practice. APIEval-20 gives the field a concrete, reproducible measure of whether an AI agent thinks like a QA engineer."

A Head of Engineering at a Fortune 500 financial services company noted in feedback to KushoAI that they had been evaluating AI testing tools for the past year and consistently ran into the challenge of comparing them objectively. They highlighted that APIEval-20 is the first framework they have seen that directly addresses this gap, surfacing shortcomings in agent reasoning that are not visible in demo environments.

Key Benchmark Details

20 scenarios across payments, authentication, e-commerce, scheduling, user management, notifications, and search. Each contains 3 to 8 planted bugs across simple, moderate, and complex tiers.
Binary evaluation against live reference implementations. Scoring weights bug detection at 70%, coverage at 20%, and efficiency at 10%.