Benchmarks

Discovery timeline

Items with an evidence label, oldest first (labels: what they mean). Unlabelled items (before the labels began) are listed below by date.

1. 21 OctPreprintFinHopper: A Progressive Multi-Hop Benchmark for Complex Financial Agents with Step-wise Evaluation
2. 21 OctPreprintMoveBench: A Rubric-Based Benchmark for Evaluating Large Language Models in Sports and Health…
3. 21 OctPreprintDeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia

21 October 2026

• FinHopper: A Progressive Multi-Hop Benchmark for Complex Financial Agents with Step-wise Evaluation

  Financial agents are expected to answer factual investment queries through open search, but financial multi-hop retrieval differs from general multi-hop QA in a crucial way: each intermediate answer…

  In the issue · 05 AI Finance · arXiv 2610.26036 · Evidence: Preprint

• MoveBench: A Rubric-Based Benchmark for Evaluating Large Language Models in Sports and Health Coaching

  Despite rapid advances in large language models (LLMs), their deployment in the sports and health domain remains high-risk due to the absence of domain-specific evaluation benchmarks.

  In the issue · 20 AI Sports & Fitness · arXiv 2610.26185 · Evidence: Preprint

• DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia

  While language models have demonstrated remarkable capabilities, their performance remains disproportionately skewed towards high-resource languages.

  In the issue · 25 AI Languages & Translation · arXiv 2610.26155 · Evidence: Preprint

• Inception says Mercury 3 beats GPT-6 Luna and Claude Haiku 5.5 at up to 8× the speed

  In the company's own benchmarks, the new diffusion language model beat both rivals on eight of 12 tests at four to eight times their speed.

  In the issue · 37 AI Foundation Models & Assistants · SiliconANGLE

20 October 2026

• Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations

  We test whether Javo, a sales agent fine-tuned on 4.6 million consented, de-identified commercial conversations, sells better than a frontier general-purpose model and better than human sales associates.

  In the issue · 07 AI Commerce · arXiv 2610.25461

• MUSEBENCH: A Benchmark for AI-Generated Music Detection and Artifact Analysis

  Advances in generative AI have made it possible to produce music that is nearly indistinguishable from human compositions, raising the question of whether such works can be reliably distinguished from human compositions.

  In the issue · 23 AI Music & Arts · arXiv 2610.20310

• FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval

  Evaluating fashion retrieval systems remains challenging due to the limited availability of benchmarks that capture the fine-grained, open-ended queries encountered in real-world search and recommendation.

  In the issue · 26 AI Fashion & Beauty · arXiv 2610.17841

• SWE-Residual: Evaluating Starting-State Robustness of Coding Agents

  Software engineering (SWE) benchmarks are the primary way to measure the capabilities of coding agents: an agent is placed in a code repository or machine environment, given a request, and judged by hidden tests.

  In the issue · 38 AI Coding & Developer Tools · arXiv 2610.25899

• CXBench: A Benchmark for Process-Compliant LLM Agents in Regulated Customer-Experience Workflows

  Tool-using language agents increasingly perform regulated customer-service workflows, where a correct final state is insufficient: agents must also provide disclosures, obtain consent, and follow required procedures.

  In the issue · 43 AI Governance · arXiv 2610.25954

19 October 2026

• Blockchain-forensics agents benchmark

  A verifiable benchmark that tests AI agents tracing laundered crypto through mixers, swaps and bridges.

  In the issue · 06 AI Law & Investigation · ABF-Bench, arXiv

• LexPilot: legal LLMs with laypeople

  Shows that legal LLMs often answer underspecified lay questions irresponsibly, and benchmarks how proactively they ask clarifying questions in Chinese civil cases.

  In the issue · 06 AI Law & Investigation · arXiv 2610.23141

• PremiseCheck: fact-checking peer reviews

  A benchmark for factual errors in human and AI review comments.

  In the issue · 08 AI Journalism · arXiv 2610.21046

• ArchWorld: reconstructing diverse architecture with 3D foundation models

  A geographically broad benchmark of architectural landmarks. It shows that 3D foundation model errors vary by region.

  In the issue · 09 AI Architecture · arXiv 2610.22514

• PanoBench: panoptic occupancy for driving

  An instance-aware 3D occupancy benchmark.

  In the issue · 13 AI Transportation · arXiv 2610.23537

• From adults to children: chest X-ray AI

  Benchmarks how foundation models generalise from adult to paediatric chest X-rays.

  In the issue · 15 AI Kids · arXiv 2610.21809

• MindEval2: clinically grounded patients and rubrics

  A more realistic benchmark for mental-health support chatbots.

  In the issue · 17 AI Psychology · arXiv 2610.22314

• MUB: identity-masked music benchmark

  Separates audio evidence from answers that models could recover from metadata.

  In the issue · 23 AI Music & Arts · arXiv 2610.22989

• MagicGarment: 3D garments from one image

  A benchmark for single-image 3D garment reconstruction.

  In the issue · 26 AI Fashion & Beauty · arXiv 2610.19623

• GCAM-Gym climate-policy benchmark

  A Gymnasium environment for comparing climate-policy search.

  In the issue · 34 AI Environment · arXiv 2610.23569

• DeepSeek-V4.1-Flash-1017

  An MIT-licensed update with gains in reasoning, coding, agentic and visual benchmarks, and shorter outputs.

  In the issue · 37 AI Foundation Models & Assistants · DeepSeek (Hugging Face)

18 October 2026

• Citation-grounded earnings-call benchmark

  A benchmark that tests whether LLM analysis of earnings-call transcripts is trustworthy and properly cited.

  In the issue · 05 AI Finance · arXiv

• When Citations Mislead? (PARCEL)

  A benchmark of 3,396 claims for detecting legal hallucinations claim by claim.

  In the issue · 06 AI Law & Investigation · arXiv

• Get In, Decide, Prove

  A benchmark of how ready merchants are for agentic commerce.

  In the issue · 07 AI Commerce · arXiv

• DistressBench

  A benchmark that evaluates crisis support from large language models across several dimensions.

  In the issue · 17 AI Psychology · arXiv

• Can LLMs Tell Stories Out of Order?

  A corpus and benchmark for generating nonlinear narratives.

  In the issue · 24 AI Literature · arXiv

• UTP-Bench: uncertainty-aware travel planning

  A large benchmark testing whether LLM itineraries survive transport delays and crowd fluctuations.

  In the issue · 27 AI Travel & Hospitality · arXiv

• AdSpark

  A 300K-sample dataset and benchmark for product-advertisement video generation.

  In the issue · 28 AI Marketing & Advertising · arXiv

• Blinded, prospective benchmark of in-silico antibody discovery

  A benchmark anchored to experimentally measured affinity and developability.

  In the issue · 32 AI Biotechnology · Nature Biotechnology

• SWE-Journey

  A benchmark of long-horizon, multi-turn coding tasks closer to real use of coding assistants.

  In the issue · 38 AI Coding & Developer Tools · arXiv

• CyberArena

  Benchmarks AI agents in live, symmetric attack-and-defence combat.

  In the issue · 42 AI Cybersecurity · arXiv