Build evaluations and regression sets for model applications

As of 2026-10-09, AI does this work at L2 Partial automation. 7 updates bear on it. Strongest evidence: From the publisher only.

Level

L2Partial automation

Updates

7

Companies

5Microsoft · Tencent · OpenAI

Strongest evidence

T3From the publisher only

What moved it

Every update that bears on this work, newest first.

• 2026-10-06Microsoft
  GitHub releases ReviewBench for AI code review agents

  GitHub introduces ReviewBench, an open benchmark for AI code review agents, built from analysis of 103.9M pull requests with 219 PRs across 19 languages.

  L1 AssistedFrom the publisher onlySource ↗
• 2026-09-22Tencent
  Tencent Hunyuan introduces WebCraftBench for agent web-app evaluation

  Tencent Hunyuan announced WebCraftBench, a benchmark where agents use a live app, with coverage-guided exploration and scoring of aesthetics and usability.

  L2 Partial automationFrom the publisher onlySource ↗
• 2026-09-15OpenAI
  Cognition's Devin uses GPT-6 Astra for test generation

  OpenAI Devs states that GPT-6 Astra helps Cognition's Devin back up 'it works' with tests before the team ships.

  L2 Partial automationFrom the publisher onlySource ↗
• 2026-09-14OpenAI
  Perplexity engineer uses GPT-6 Astra in Codex for end-to-end testing

  An engineer at Perplexity reports using GPT-6 Astra in Codex to build test harnesses and mock third-party API responses for end-to-end testing.

  L2 Partial automationFrom the publisher onlySource ↗
• 2026-09-10Google
  Google Research introduces ToolGrad framework for tool-use datasets

  Google Research introduced ToolGrad, an efficient framework for generating tool-use datasets that generates ground-truth tool-use chains before prompts, achieving a nearly 100% pass rate and improved LLM tool-use performance.

  L2 Partial automationClaimed L3 · Accepted L2From the publisher onlySource ↗
• 2026-09-03Alibaba
  Qwen introduces E-Commerce Bench benchmark

  Alibaba's Qwen team presents E-Commerce Bench, a benchmark for long-horizon autonomous business operations where agents run online stores for 365 days with ¥100,000 starting capital.

  L2 Partial automationFrom the publisher onlySource ↗
• 2026-08-26Microsoft
  Microsoft Foundry adds agent cost optimization via routing, caching and evaluation

  Microsoft Foundry describes four ways to optimize agent costs through smarter routing, caching and evaluation.

  L1 AssistedFrom the publisher onlySource ↗

More work in Model applications and agent systems