Skip to content
integration bench from HeyMilo

The benchmark for AI coding agents on API integrations.

HeyMilo connects hiring teams to their ATS, HRIS, CRM, ERP, SORs, and legacy systems, including software that shipped in 1993.

Integration Bench is one test for AI coding agents: can it ship one of 1M integrations that survives the real vendor?

Source

~1M: round figure for the integrations an agent might be asked to write, given thousands of business apps with a public API. 1993: UKG Pro's first release, UltiPro for LAN (Ultimate Software S-1, 1998).

See the leaderboard Open a trajectory

In collaboration with experts from

  • Tesla
  • OpenAI
  • Meta
  • PwC
  • Tableau

How this started

We did not set out to build a benchmark. We set out to stop rewriting connectors.

The long version

  1. Nov 2023
    First connector ships.
    A customer's screening results need to land in their ATS. One engineer, one vendor, one week of reading documentation and finding out which parts were true. Tooling: ChatGPT in a browser tab (GPT-4 Turbo shipped that month), code pasted into the connector by hand.
  2. 2024
    The staffing and recruiting stack, one vendor at a time.
    Bullhorn, Greenhouse, Lever, Avionté, JobDiva, Ceipal and the rest. Every connector follows the same pattern: the docs are wrong somewhere, and the only place you can see it is the vendor's request log. Tooling: GPT-4o (May) and Claude 3.5 Sonnet (June) move into the editor. They explain the docs and fill in boilerplate; a human still writes the connector.
  3. 2025
    Enterprise HRIS, and the software nobody wants to integrate with.
    Workday, SuccessFactors, UKG Pro, iCIMS. Then the long tail: SOAP endpoints, cookie-session auth, CSV drops over SFTP, pagination that breaks on page two. The oldest lineage is UKG Pro: it shipped in 1993 as a DOS payroll product and still runs payroll today. Tooling: Claude Code (Feb), then GPT-5 (Aug) and Claude Sonnet 4.5 (Sep) write whole files from a ticket. Every line still reviewed by a person.
  4. Early 2026
    Agents ship connectors end to end.
    Claude Opus 4.6 (Feb) takes a ticket and returns a working connector, and review moves from reading the diff to reading the vendor's log. That is where the failures are: right-looking code that pulls the wrong rows, hammers a rate limit, or puts a credential in a query string. So we build vendors that do not exist, with request logs we control, and grade from the other side of the wire.
  5. Sept 2026
    Integration Bench rev 01. 100+ integrations shipped.
    15 synthetic vendors, 17 models, one official attempt per task. Scores are read from the vendor's log, not the agent's code. 50 public tasks published in full: every tool call, vendor request, graded check and diff for 850 attempts is on this site. The rest of the suite is private. Open a trajectory

of HTTP requests to web content now come from automated clients, not people.

Cloudflare Radar recorded the crossover in June 2026. A growing share of that traffic is agents reading docs, calling APIs and moving records between systems. The integration engineer is increasingly a model. Whether it integrates correctly is now the load-bearing question for any product that only works once connected.

Source: Cloudflare Radar, bot vs. human traffic, June 2026

Graded from the vendor's request log

Most coding benchmarks grade a repo against its own tests, which miss how integration work fails:

  1. The documentation is wrong somewhere.
  2. The vendor enforces limits and signatures the code never mentions.
  3. A connector can return the right rows and still get your API key revoked.

None of that shows in the diff, but all of it shows in the traffic, which is where we read the score.

15
Synthetic vendors that reflect real behavior.
Each vendor is generated from a single declaration into a live API simulator, with its own auth, pagination, webhooks, rate limits and a budget of deliberate documentation errors. No SDK or forum thread about them exists anywhere, so contamination is ruled out by construction.
22.6
What a submission that does nothing scored under our first partial-credit design.
"Never put a credential in a query string" passes trivially if you make no requests. We measured that floor, then switched to mandatory-gated scoring: one failed hard check zeroes the task. Every published number uses that scorer.
850
Attempts published in full.
Every tool call, vendor request, graded check and the final diff, for 17 models on 50 public tasks. The rest of the suite is private. The score is a function of the recorded run and anyone can recompute it.
Browse trajectories
Integration Bench · rev 01 · preliminary

Leaderboard

Mean score over the public task set, pass 1, one official attempt per model per task. Development-set numbers: we built the instrument against these tasks, so they describe the benchmark as much as the models.

#ModelMean scoreScore barResolvedStatus
1 Claude Fable 567.7±2.668%
2 GPT-5.6 Sol63.6±2.764%
3 DeepSeek V4 Pro (max)63.4±2.864%
4 Qwen 3.8 2.4T A95B61.7±2.962%
5 Grok 4.661.2±3.062%
6 Claude Opus 559.3±3.160%
7 Qwen 3.8 27B (xhigh)57.7±3.258%
8 Kimi K357.5±3.358%
Mean score

All 17 models, per-task breakdown and method

The benchmarks

Each benchmark covers a different part of the stack and uses the same scoring method. Fifty tasks are public; the rest of the suite is private. Every official run on the public set is published.

Now · rev 01

Integration Bench

15 synthetic ATS, HRIS and job-board vendors. Polling, webhooks, writeback. 50 public tasks published in full; the rest of the suite is private.
Leaderboard and method
Next

Legacy Integration Bench

The vendors nobody wants to integrate with: SOAP and XML-RPC, cookie-session auth, CSV over SFTP, undocumented pagination. The 1993-to-2010 slice of what HeyMilo connects to.
What is planned · Collaborate on it

After that: systems of record. Bidirectional CRM sync, where a wrong write is the expensive failure. Not scheduled yet.

Release history and what changed in each revision

Get involved

Request tasks or samples →
Extra tasks, the evidence bundle for any attempt, or access to the private set.
Collaborate or fix a task →
Report a wrong grader, propose a vendor, submit a model or harness.