Announcing Harvey LAB-AA: evaluating AI agents on real-world legal work

Artificial Analysis

[](https://artificialanalysis.ai/login)K

All articles

July 7, 2026

Announcing Harvey LAB-AA: evaluating AI agents on real-world legal work

Harvey LAB-AA (Legal Agent Benchmark) is our implementation of Harvey's new agentic legal benchmark, evaluating language models on real-world legal work across 24 practice areas.

Models are tested on a private set of 120 legal tasks built by the team at Harvey, spanning practice areas from corporate M&A and capital markets to tax, litigation, and bankruptcy. Models work to create the legal outputs specified in each task, and each task is graded against a rubric of binary criteria. The primary metric we present is the all-pass rate: the share of tasks where all criteria in the rubric are satisfied, reflecting the high standard of real-world professional legal deliverables.

Score

All-pass RateCriterion Pass Rate

Harvey LAB-AA: All-pass Rate

Share of tasks where every rubric criterion passes (Harvey's all-pass grading, no partial credit) · Independently benchmarked by Artificial Analysis

Reasoning models are indicated by a lightbulb icon

Claude Fable 5 (max, with Opus 4.8 fallback) leads Harvey LAB-AA with a 14.2% all-pass rate, after falling back to Claude Opus 4.8 on only one task. This is almost double the next best models, Claude Opus 4.8 (max) and GLM-5.2 (max), which tie at 7.5%, followed by MiniMax-M3 at 6.7% and Claude Sonnet 5 at 5.0%.

Frontier legal work is far from solved: most models pass a majority of individual rubric criteria but fully satisfy the requirements of very few tasks. The best model still leaves \~86% of professional legal deliverables incomplete, and 13 of the 28 models evaluated at launch fully pass zero tasks. Only four models score above 90% on criterion pass rate: Claude Fable 5 (93.6%), Claude Opus 4.8 (91.1%), GLM-5.2 (91.0%), and Claude Sonnet 5 (90.1%).

For up-to-date results see the Harvey LAB-AA evaluation page. Charts show data as at 7 July 2026.

Cost

Topping the leaderboard is expensive. Claude Fable 5 leads all-pass at 14.2% and is the most expensive model we have run at \~$18.9 per task, ahead of Claude Sonnet 5 at \~$11.8 per task. Claude Opus 4.8 costs \~$8.2 per task, while GLM-5.2 (max) matches Opus 4.8 at just \~15% of the cost (\~$1.3 per task). Across all models evaluated, cost per task spans \~950x: Gemini 3.1 Flash-Lite passes 31.1% of criteria for \~$0.02 per task.

Cost per TaskScore vs. Cost per TaskTotal Cost to RunScore vs. Compute

Harvey LAB-AA: Cost per Task

Average cost per task (USD), broken down by input, cache hit, cache write, reasoning, and answer tokens

AnswerReasoningCache WriteCache HitInput

Reasoning models are indicated by a lightbulb icon

###

Evaluation Cost per Task

Average cost per task in the evaluation. Costs are split by input, cache hit, cache write, reasoning, and answer token pricing where canonical token counts are available.

Token Usage

All-pass rate is generally tied to how much work a model does per task. Claude Fable 5 generates \~117k output tokens per task to reach 14.2% all-pass, while Claude Opus 4.8 and GLM-5.2 generate \~111k and \~78k output tokens per task respectively to reach 7.5%. Claude Sonnet 5 generates \~179k output tokens per task, the most of any model evaluated at launch, to reach 5.0%.

Output Tokens per TaskScore vs. Output Tokens per TaskToken BreakdownScore vs. Output Tokens

Harvey LAB-AA: Output Tokens per Task

Output tokens used to run one task, broken down by reasoning and answer tokens

AnswerReasoning

Reasoning models are indicated by a lightbulb icon

###

Evaluation Output Tokens per Task

The average number of answer and reasoning tokens produced per benchmark task in this evaluation.

Speed

Stronger models tend to spend longer per task. Claude Fable 5 averages \~16.9 minutes per task, Claude Opus 4.8 \~18.5 minutes, and Claude Sonnet 5 \~22.8 minutes. GLM-5.2 is the exception, matching Claude Opus 4.8's all-pass rate at only \~5.0 minutes per task. DeepSeek V4 Flash was the fastest model to achieve an above 0% all-pass score, averaging \~4.4 minutes per task.

Time per TaskScore vs. Time per Task

Harvey LAB-AA: Time per Task

Weighted average decode time (minutes) per task; excludes TTFT and overhead time · Lower is better

Reasoning models are indicated by a lightbulb icon

###

Evaluation Time per Task

The weighted average time (seconds) per evaluation task. This is calculated by dividing output tokens per task by output speed, weighted by the relative weights of each benchmark in the evaluation.

Turns

The leaders work in long agentic loops. Claude Fable 5 averages \~64 turns per task for its 14.2% all-pass, with Claude Opus 4.8, GLM-5.2, and MiniMax-M3 all in the 56-64 turn range. Claude Sonnet 5 runs the longest loops of any model at \~161 turns per task, 2.5x Claude Fable 5.

Average Turns per TaskTurns per Task (Range)Score vs. Average Turns

Harvey LAB-AA Benchmark Leaderboard: Average Turns per Task

Average number of model turns per Harvey LAB-AA task · Lower is better

Reasoning models are indicated by a lightbulb icon

###

What Turns Is Measuring

This chart shows the average number of turns the agent takes per task. It is a rough proxy for how many actions, tool calls, and iteration cycles an agent is using to complete benchmark tasks.

Score vs. Release Date

Harvey LAB-AA: All-pass Rate vs. Release Date

Most attractive region

OpenAIGoogleAnthropicMistralDeepSeekxAIMiniMaxNVIDIAKimiStepFunXiaomiZ AIAlibaba

Example Tasks & Submissions

View task set on GitHub

Browse representative Harvey LAB tasks from the public task set, the reference files each model was given, and the deliverables it produced.

Task 1M&A change-of-control analysisTask 2Deposition outlineTask 3Arbitration agreement redlineTask 4M&A disclosure schedulesTask 5Commercial lease review

Mergers & Acquisitions

Instructions

Review the attached acquisition data room contracts and internal memo for change of control and assignment provisions, and prepare a comprehensive deal team report.

Output: `coc-analysis-report.docx`

Show more

Deliverables

Expected outputs the model must produce

• coc-analysis-report.docxA comprehensive deal team report analyzing change of control and assignment provisions across the target’s material contracts.

Reference files

Provided to the model

apex-distribution-agreement.docxOpen

apex-msa.docxOpen

deal-overview-memo.docxOpen

deal-summary-memo.docxOpen

first-continental-credit-agreement.docxOpen

great-lakes-credit-agreement.docxOpen

meridian-dpa.docxOpen

meridian-license-agreement.docxOpen

novabridge-partnership.docxOpen

novabridge-supply-agreement.docxOpen

orion-subscription-renewal.docxOpen

pinnacle-ecommerce-agreement.docxOpen

pinnacle-license.docxOpen

ridgeline-10k-excerpt.docxOpen

solara-deferred-comp-plan.docxOpen

terranode-isa.docxOpen

terraverde-lease-agreement.docxOpen

webb-employment-agreement.docxOpen

wellstone-manufacturing-agreement.docxOpen

Model submissions

Deliverables produced by each model

Claude Fable 5 (with fallback) - coc-analysis-report.docxGemini 3.1 Pro Preview - coc-analysis-report.docxGPT-5.5 (xhigh) - coc-analysis-report.docxGLM-5.2 - coc-analysis-report.docxGrok 4.3 (high) - coc-analysis-report.docx

Claude Fable 5 (with fallback) - coc-analysis-report.docx

Open

How Harvey LAB-AA differs from Harvey's LAB

Harvey LAB-AA is our independent reimplementation of Harvey's evaluation, and there are several key differences to the original version:

• Models are run on our Stirrup agent harness, enabling features such as context compaction rather than failure when reaching context limits, with simplified Artificial Analysis-authored agent and judge prompts
• We do not include Harvey's custom tools and document-generation skill scripts (e.g. pptx, docx), instead providing a simple code execution tool to reflect raw model ability
• Deliverables must match the exact filename specified, rather than fuzzy matching when models produce incorrect filenames
• Grading uses a single Gemini 3.1 Pro judge, tested to be well-calibrated against a frontier panel

Harvey LAB-AA resources

• The leaderboard and full results live on the Harvey LAB-AA evaluation page, updated as new models are released
• The methodology page documents the full implementation, including the agent and grading prompts
• Harvey's original LAB announcement introduces the benchmark and its design
• A public set of representative tasks is available on GitHub
• Harvey LAB-AA runs on Stirrup, our open-source agent framework

#### Read the latest

How GPT-5.6 Sol, Terra, Luna compare on intelligence vs costGPT-5.6 Sol and Luna are ahead of Terra at every point on the Intelligence vs Cost per Task chart. GPT-5.6 Luna stands out as a particularly cost efficient modelJuly 13, 2026Muse Spark 1.1: Meta gains 8 Intelligence Index points in three monthsMeta's Muse Spark 1.1 scores 51 on the Artificial Analysis Intelligence Index and is cost and token efficient compared to its peersJuly 10, 2026GPT-5.6 benchmarks across Intelligence, Speed and CostGPT-5.6 Sol comes close second to Claude Fable 5 in the Artificial Analysis Intelligence Index at one third of the cost, and leads the Artificial Analysis Coding Agent Index in OpenAI’s Codex harnessJuly 9, 2026