AA-Briefcase: Agentic Knowledge Work Benchmark
[](https://artificialanalysis.ai/login)
AA-Briefcase: Agentic Knowledge Work Benchmark
A private evaluation developed by Artificial Analysis for frontier agentic capability in long-horizon knowledge work, testing agents on realistic business workflows that require deliverables such as spreadsheets, presentations, and memos.
Related Links
AA-Briefcase Launch ArticleAA-Briefcase Methodology
ArtificialAnalysis/AA-Briefcase-Lite
ArtificialAnalysis/Stirrup
1Structure2Submission3Grading and Scoring
AA-Briefcase evaluates models across four multi-week knowledge work projects, comprising thousands of input files and 91 tasks in total. Across the scenarios, models must complete realistic professional workflows in fields such as data science, product management, and corporate strategy. Each scenario is a multi-week workflow that the agent works through in sequence, each week holding several tasks. Every task is a deliverable graded against a rubric of checks. Although tasks within a scenario share files and context across weeks, models currently complete each task in an independent run, without carrying over their own prior submissions.
AA-BriefcaseDataScienceProductManagementBankingOperationsHeavy IndustryStrategyWeek 1Week 2Week 3Week 4Week 5Week 6Task 1Task 2Task 3Task 4EVALUATIONScenariosAA-Briefcase is built around four major knowledge workareas, with each structured as a separate projectscenarioWeeksEach scenario is made up of a number of weeks designedto simulate a professional workflowTasksEach week has up to five distinct tasks, which theagent has to complete within the scenario and weekcontextChecksEach task is graded across three dimensions: objectiverubric checks, analytical quality, and presentationquality, with task-specific criteria for eachcomparisonRubricbinary pass or fail across criteriaAnalytical Qualitypairwise vs. another modelPresentationpairwise vs. another model
Each task is graded against three types of checks:
Rubric
Binary pass or fail per check
Did the model follow the task instructions, identify requirements hidden across source files, use the correct evidence, and reach the right conclusions?
Analytical Quality
Pairwise comparison
Compared against another model's submission, which deliverable is more thorough, analytically rigorous, and well-supported?
Presentation
Pairwise comparison
Compared against another model's submission, which one is more professionally presented?
A public fifth scenario has been released via Hugging Face as a representation of scenario structure, submission, and grading. This does not count toward official AA-Briefcase results, and is demonstrative only.
Results
AA-Briefcase EloAA-Briefcase Rubric Score (%)Analytical Quality & Presentation Elo
AA-Briefcase Elo
AA-Briefcase is an agentic knowledge work benchmark developed by Artificial Analysis. AA-Briefcase Elo is a combined metric that aggregates rubric pass rate, analytical quality Elo and presentation Elo · Higher is better
25 of 37 models
Add model from specific provider
Not currently available
Reasoning models are indicated by a lightbulb icon
###
AA-Briefcase Elo
AA-Briefcase Elo is a combined metric that aggregates analytical quality Elo, presentation Elo, and rubric pass rate, with rubric performance converted into Elo via synthetic head-to-head matches. Elo and 95% confidence interval bounds are clamped at 0.
Cost
Cost per TaskAA-Briefcase Elo vs. Cost per TaskCost by File TypeTotal Cost to RunAA-Briefcase Elo vs. Total Cost
AA-Briefcase Cost per Task
Mean cost (USD) per task to run AA-Briefcase, calculated from token usage and model pricing including representative cache hit rates
25 of 37 models
AnswerReasoningCache WriteCache HitInput
Reasoning models are indicated by a lightbulb icon
###
Cost per Task
The total cost to run AA-Briefcase divided by the number of tasks (91 for full submission of tasks). Cost is calculated from token usage and model pricing, split across input, cache hit, cache write, reasoning, and answer token prices, including representative cache hit rates.
Example Task, Submissions, and Grading
Explore a representative AA-Briefcase week from the public Due Diligence scenario available via Hugging Face. The outputs and grading shown here illustrate what AA-Briefcase evaluates. Scores are shown for a representative model set. Submissions and verdicts in this representative scenario do not contribute to a model's AA-Briefcase Elo or other benchmark scores.
Scenario overview
Week 1 overview(Note: the public scenario only has 1 week for illustrative purposes)
Task 1Market Structure MapTask 2Market ModelTask 3Target AssessmentTask 4Briefing Video
Task brief
Model
Claude Fable 5
Compare models
Model submission
market\_overview.pdf
market\_overview.tex
Grading summary
Source files32 of 67 relevant · same for all models
Score Comparisons
AA-Briefcase Elo vs. Intelligence IndexAnalytical Quality vs. Presentation EloAnalytical Quality Elo vs. Rubric Score (%)Presentation Elo vs. Rubric Score (%)
AA-Briefcase Elo vs. Artificial Analysis Intelligence Index
AA-Briefcase Elo · Artificial Analysis Intelligence Index
25 of 37 models
Add model from specific provider
Most attractive quadrant
OpenAIMetaGoogleAnthropicMistralDeepSeekxAIUpstageMiniMaxNVIDIAKimiXiaomiMBZUAI Institute of Foundation ModelsZ AIAlibaba
###
AA-Briefcase Elo
AA-Briefcase Elo is a combined metric that aggregates analytical quality Elo, presentation Elo, and rubric pass rate, with rubric performance converted into Elo via synthetic head-to-head matches. Elo and 95% confidence interval bounds are clamped at 0.
###
Artificial Analysis Intelligence Index
Artificial Analysis Intelligence Index v4.1 includes: GDPval-AA v2, 𝜏³-Banking, Terminal-Bench v2.1, SciCode, Humanity's Last Exam, GPQA Diamond, CritPt, AA-Omniscience, AA-LCR. See Intelligence Index methodology for further details, including a breakdown of each evaluation and how we run them.
File Type Results
AA-Briefcase performance broken out by the file type of the deliverable (Excel, PowerPoint, PDF, Word, Other).
Rubric Score by File Type (Normalized)Analytical Quality Elo by File Type (Normalized)Presentation Elo by File Type (Normalized)
AA-Briefcase Rubric Pass Rate by File Type (Normalized)
Rubric pass rate by deliverable file type · Scores are normalized per file type across all models tested, where green represents the highest score for that file type and red represents the lowest score for that file type
25 of 37 models
Add model from specific provider
Reasoning models are indicated by a lightbulb icon
###
File Types
File types are categorized by the required submission format, with “Other” covering formats such as HTML and LaTeX.
###
AA-Briefcase Rubric Score
The share of binary rubric checks the submission passed (passed checks divided by total checks), aggregated across all AA-Briefcase tasks. Rubric checks are pass/fail criteria covering whether the deliverable includes required content and cites sources correctly, and whether it resolves planted cross-source conflicts.
Token Usage
Output Tokens per TaskOutput Token UsageAA-Briefcase Elo vs. Output Token UsageAA-Briefcase Token BreakdownToken Usage by File Type
AA-Briefcase Output Tokens per Task
Mean reasoning and answer tokens consumed per AA-Briefcase task
25 of 37 models
Answer tokensReasoning tokens
Reasoning models are indicated by a lightbulb icon
###
Evaluation Output Tokens
The number of output tokens used to run the evaluation, including visible answer tokens and reasoning tokens where reported by reasoning models.
Speed
Time per TaskAA-Briefcase Elo vs. Time per Task
Time per Task
Wall-clock time (minutes) per task: answer and reasoning generation plus tool execution time · Lower is better
25 of 37 models
Reasoning models are indicated by a lightbulb icon
###
AA-Briefcase Time per Task
Estimated wall-clock time per task: the sum of answer and reasoning tokens per task divided by the model’s canonical answer output speed, plus mean tool execution time per task. Lower is better.
Turns
Mean Turns per TaskTurns per Task (Range)AA-Briefcase Elo vs. Mean Turns
Mean Turns per Task
Average number of model turns per AA-Briefcase task · Lower is better
25 of 37 models
Not currently available
Reasoning models are indicated by a lightbulb icon
###
What Turns Is Measuring
This chart shows the mean number of turns the agent takes per task. It is a rough proxy for how many actions, tool calls, and iteration cycles an agent is using to complete benchmark tasks.
Tool Usage
Tool invocations issued by each agent during AA-Briefcase: counts by tool category, mean tool calls per turn, and source-pool exploration coverage.
Tool Calls BreakdownMean Tool Calls per TurnFile Exploration Coverage
AA-Briefcase Tool Calls Breakdown, Avg per Task
Average tool invocations per AA-Briefcase task, bucketed by intent
25 of 37 models
Add model from specific provider
ExploreView ImageReadComputeWriteOther
Reasoning models are indicated by a lightbulb icon
###
AA-Briefcase Tool Calls Breakdown
Agent tool calls are grouped into six categories: explore (navigating and searching the workspace), read (reading file contents), write (creating or editing files), compute (running code or calculations), view image (visual inspection of files), and other (anything else).
Model Size (Open Weights Models Only)
AA-Briefcase Elo vs. Total ParametersAA-Briefcase Elo vs. Active ParametersAA-Briefcase Elo vs. Compute Proxy
AA-Briefcase Elo vs. Total Parameters
AA-Briefcase Elo · Size in parameters (billions) · Open weights models only
25 of 37 models
Add model from specific provider
Most attractive quadrant
OpenAIGoogleMistralDeepSeekUpstageMiniMaxNVIDIAKimiXiaomiMBZUAI Institute of Foundation ModelsZ AIAlibaba
###
AA-Briefcase Elo
AA-Briefcase Elo is a combined metric that aggregates analytical quality Elo, presentation Elo, and rubric pass rate, with rubric performance converted into Elo via synthetic head-to-head matches. Elo and 95% confidence interval bounds are clamped at 0.
###
Total Parameters
The total number of trainable weights and biases in the model, expressed in billions. These parameters are learned during training and determine the model's ability to process and generate responses.
Score vs. Release Date
AA-Briefcase Elo vs. Release Date
AA-Briefcase Elo · Model release date
25 of 37 models
Add model from specific provider
Most attractive region
OpenAIMetaGoogleAnthropicMistralDeepSeekxAIUpstageMiniMaxNVIDIAKimiXiaomiMBZUAI Institute of Foundation ModelsZ AIAlibaba
###
AA-Briefcase Elo
AA-Briefcase Elo is a combined metric that aggregates analytical quality Elo, presentation Elo, and rubric pass rate, with rubric performance converted into Elo via synthetic head-to-head matches. Elo and 95% confidence interval bounds are clamped at 0.
Leaderboard
Expand score breakdown
| Creator | Name | Elo | CI | Release Date | |
| ------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | ---- | ------------ | -------- |
| 1 | Anthropic | Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) | 1586 | \-15 / +15 | Jun 2026 |
| 2 | Anthropic | Claude Opus 4.8 (Adaptive Reasoning, Max Effort) | 1353 | \-10 / +11 | May 2026 |
| 3 | Anthropic | Claude Opus 4.7 (Adaptive Reasoning, Max Effort) | 1291 | \-11 / +11 | Apr 2026 |
| 4 | 
Z AI | GLM-5.2 (max) | 1261 | \-10 / +11 | Jun 2026 |
| 5 | 
OpenAI | GPT-5.5 (xhigh) | 1159 | \-10 / +10 | Apr 2026 |
| 6 | 
MiniMax | MiniMax-M3 | 1113 | \-9 / +10 | Jun 2026 |
| 7 | OpenAI | GPT-5.5 (high) | 1102 | \-10 / +10 | Apr 2026 |
| 8 | Anthropic | Claude Opus 4.7 (Non-reasoning, High Effort) | 1092 | \-10 / +10 | Apr 2026 |
| 9 | Anthropic | Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort) | 1079 | \-9 / +9 | Feb 2026 |
| 10 | OpenAI | GPT-5.5 (medium) | 1000 | \-0 / +0 | Apr 2026 |
| 11 | Z AI | GLM-5.1 (Reasoning) | 975 | \-10 / +10 | Apr 2026 |
| 12 | 
DeepSeek | DeepSeek V4 Pro (Reasoning, Max Effort) | 932 | \-9 / +9 | Apr 2026 |
| 13 | 
Alibaba | Qwen3.7 Max | 907 | \-9 / +9 | May 2026 |
| 14 | 
Google | Gemini 3.5 Flash (medium) | 871 | \-10 / +10 | May 2026 |
| 15 | 
Xiaomi | MiMo-V2.5-Pro | 871 | \-10 / +10 | Apr 2026 |
| 16 | 
NVIDIA | Nemotron 3 Ultra 550B A55B (Reasoning) | 870 | \-10 / +11 | Jun 2026 |
| 17 | OpenAI | GPT-5.3 Codex (xhigh) | 867 | \-10 / +10 | Feb 2026 |
| 18 | Google | Gemini 3.5 Flash (high) | 866 | \-10 / +9 | May 2026 |
| 19 | DeepSeek | DeepSeek V4 Flash (Reasoning, Max Effort) | 830 | \-10 / +10 | Apr 2026 |
| 20 | 
Kimi | Kimi K2.6 | 810 | \-10 / +10 | Apr 2026 |
| 21 | Alibaba | Qwen3.6 27B (Reasoning) | 807 | \-11 / +11 | Apr 2026 |
| 22 | 
xAI | Grok 4.3 (high) | 750 | \-10 / +10 | Apr 2026 |
| 23 | OpenAI | GPT-5.4 mini (xhigh) | 707 | \-10 / +11 | Mar 2026 |
| 24 | 
Meta | Muse Spark | 633 | \-12 / +12 | Apr 2026 |
| 25 | Anthropic | Claude 4.5 Haiku (Reasoning) | 604 | \-12 / +11 | Oct 2025 |
| 26 | 
KwaiKAT | KAT-Coder-Pro V1 | 589 | \-13 / +13 | Nov 2025 |
| 27 | Alibaba | Qwen3.5 397B A17B (Reasoning) | 545 | \-12 / +12 | Feb 2026 |
| 28 | 
Mistral | Mistral Medium 3.5 | 505 | \-12 / +11 | Apr 2026 |
| 29 | Google | Gemini 3.1 Pro Preview | 447 | \-12 / +12 | Feb 2026 |
| 30 | Google | Gemma 4 31B (Reasoning) | 351 | \-14 / +14 | Apr 2026 |
| 31 | Google | Gemini 3.1 Flash-Lite | 214 | \-16 / +14 | Mar 2026 |
| 32 | 
Upstage | Solar Pro 3 | 116 | \-16 / +15 | Apr 2026 |
| 33 | 
MBZUAI Institute of Foundation Models | K2 Think V2 | 62 | \-18 / +18 | Dec 2025 |
| 34 | OpenAI | gpt-oss-120b (high) | 0 | \-0 / +5 | Aug 2025 |
| 35 | OpenAI | gpt-oss-20B (high) | 0 | \-0 / +0 | Aug 2025 |
| 36 | Meta | Llama 4 Maverick | 0 | \-0 / +0 | Apr 2025 |
| 37 | NVIDIA | NVIDIA Nemotron 3 Super 120B A12B (Reasoning) | 0 | \-0 / +0 | Mar 2026 |
Explore Evaluations
Artificial Analysis Intelligence IndexA composite benchmark aggregating nine challenging evaluations to provide a holistic measure of AI capabilities across mathematics, science, coding, and reasoning.AA-Briefcase: Agentic Knowledge Work BenchmarkA private evaluation developed by Artificial Analysis for frontier agentic capability in long-horizon knowledge work, testing agents on realistic business workflows that require deliverables such as spreadsheets, presentations, and memos.GDPval-AA v2 LeaderboardGDPval-AA v2 is Artificial Analysis' evaluation framework for OpenAI's GDPval dataset. It tests AI models on real-world tasks across 44 occupations and 9 major industries. Models are given shell access and web browsing capabilities in an agentic loop via Stirrup to solve tasks, with Elo ratings derived from blind pairwise comparisons.
APEX-Agents-AA Benchmark LeaderboardArtificial Analysis' implementation of the APEX-Agents benchmark, testing AI agents on long-horizon, cross-application tasks in professional-services environments with realistic application tooling.
𝜏²-Bench Telecom Benchmark LeaderboardA dual-control conversational AI benchmark simulating technical support scenarios where both agent and user must coordinate actions to resolve telecom service issues.𝜏³-Banking Benchmark LeaderboardA fintech customer-support benchmark from the 𝜏-Knowledge framework that tests whether agents can navigate a large unstructured knowledge base and execute multi-step tool calls to resolve realistic banking workflows.
Terminal-Bench Hard Benchmark LeaderboardAn agentic benchmark evaluating AI capabilities in terminal environments through software engineering, system administration, and data processing tasks.Terminal-Bench v2.1 Benchmark LeaderboardA verified refresh of Terminal-Bench v2.0 — 89 curated tasks across software engineering, system administration, data processing, model training, and security, with environment and instruction fixes so scores reflect agent capability rather than environment gaps.
SciCode Benchmark LeaderboardA scientist-curated coding benchmark featuring 288 test set subproblems from 80 laboratory problems across 16 scientific disciplines.Artificial Analysis Long Context Reasoning Benchmark LeaderboardA challenging benchmark measuring language models' ability to extract, reason about, and synthesize information from long-form documents ranging from 10k to 100k tokens (measured using the cl100k\_base tokenizer).AA-Omniscience: Knowledge and Hallucination BenchmarkA benchmark measuring factual recall and hallucination across various economically relevant domains.
IFBench Benchmark LeaderboardA benchmark evaluating precise instruction-following generalization on 58 diverse, verifiable out-of-domain constraints that test models' ability to follow specific output requirements.
Humanity's Last Exam Benchmark LeaderboardA frontier-level benchmark with 2,500 expert-vetted questions across mathematics, sciences, and humanities, designed to be the final closed-ended academic evaluation.GPQA Diamond Benchmark LeaderboardThe most challenging 198 questions from GPQA, where PhD experts achieve 65% accuracy but skilled non-experts only reach 34% despite web access.
CritPt Benchmark LeaderboardA benchmark designed to test LLMs on research-level physics reasoning tasks, featuring 71 composite research challenges.
ITBench-AA Benchmark LeaderboardArtificial Analysis' implementation of IBM's ITBench benchmark, testing AI agents on Kubernetes incident root-cause analysis from offline incident snapshots. The agent inspects alerts, events, traces, and topology and identifies the contributing-factor entities (deployments, pods, namespaces, network policies, etc.) responsible for the failure.Artificial Analysis Openness IndexA composite measure providing an industry standard to communicate model openness for users and developers.
MMLU-Pro Benchmark LeaderboardAn enhanced version of MMLU with 12,000 graduate-level questions across 14 subject areas, featuring ten answer options and deeper reasoning requirements.
Global-MMLU-Lite Benchmark LeaderboardA lightweight, multilingual version of MMLU, designed to evaluate knowledge and reasoning skills across a diverse range of languages and cultural contexts.
LiveCodeBench Benchmark LeaderboardA contamination-free coding benchmark that continuously harvests fresh competitive programming problems from LeetCode, AtCoder, and CodeForces, evaluating code generation, self-repair, and execution.MATH-500 Benchmark LeaderboardA 500-problem subset from the MATH dataset, featuring competition-level mathematics across six domains including algebra, geometry, and number theory.
AIME 2025 Benchmark LeaderboardAll 30 problems from the 2025 American Invitational Mathematics Examination, testing olympiad-level mathematical reasoning with integer answers from 000-999.
MMMU-Pro Benchmark LeaderboardAn enhanced MMMU benchmark that eliminates shortcuts and guessing strategies to more rigorously test multimodal models across 30 academic disciplines.