🔍 Read the full analysis: Z.ai’s Legal AI Clears About 1 In 5 Benchmark Tasks. How Does It Rank So Highly? on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models in the archived version. It passed 92.5% of individual rubric criteria, but a task counted as complete only when every criterion passed; the benchmark result does not establish that the model can reliably produce legal work without human review.
Z.ai’s GLM-5.3-FlashX-Legal achieved a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, placing seventh among 81 models in the archived results. The score measures how often the model met every requirement for a task—not how often it earned partial credit—and comes as Z.ai promotes the system for professional legal workflows.
Vals AI’s archived leaderboard lists the legal model at 19.17%, rounded to 19.2% in Z.ai’s documentation. It also reports a 92.5% pass rate across individual criteria. These figures describe different measures: a task fails the all-pass test if even one required detail is wrong or missing, while the criteria rate counts performance across the individual rubric items.
The evaluation used a private holdout of 120 tasks, according to Vals AI. The benchmark’s public release contains 1,250 tasks across 24 practice areas, but internet access is disabled for the leaderboard evaluation. Two AI judges grade each submission, and Vals averages their task pass rates. Vals lists the GLM result at $0.15 per test; Z.ai’s pricing page separately lists token charges of $0.37 per million input tokens and $1.25 per million output tokens.
Z.ai says the legal model improved by 12.5 percentage points over its GLM-5.3-Flash base model. That difference matches the archived table’s 19.17% legal score and 6.67% base-model score. Z.ai also calls the result state of the art among open-source models, but the source material does not identify a comparison set for that claim, and it reports no published weights for the legal model.
A High Rank With Limited Task Completion
Seventh place can sound like evidence of broad legal competence, but the benchmark’s strict scoring rule changes what the rank means. The model completed all required elements on roughly one in five tasks; on the other tasks, it missed at least one criterion. A missed deadline, citation, or key recommendation can matter even if the rest of a work product is strong.
At the same time, the 92.5% individual-criteria score suggests the model often satisfied many components within tasks that did not pass in full. Those results are relevant to developers and legal teams comparing systems, but neither metric alone establishes whether outputs are safe for client use. Firms considering deployment would still need to examine accuracy, citation checking, data handling, and the human review process.
Leaderboard performance also does not settle procurement questions. Z.ai recommends connecting the model to services for legal materials and company data, but the available source material does not specify how client information is handled in each deployment arrangement. The cost per test is a benchmark figure, not a complete estimate of the expense of integrating, supervising, or correcting the system.
Products Worth Considering
LG 24G414B-B 24-Inch UltraGear Full HD IPS Gaming Monitor
LG UltraGear 32GX870A-B 32-inch 4K OLED Gaming Monitor
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Work
Harvey released its Legal Agent Benchmark as an open-source evaluation on May 6, 2026. Each task gives an agent an instruction and a client matter with relevant materials, then asks it to produce a work product for review. Harvey says the benchmark uses more than 75,000 expert-written, binary rubric criteria covering matters such as facts, conclusions, citations, recommendations, deadlines, and formatting.
Harvey calls its scoring method all-pass grading: a task counts as complete only if every criterion passes. A task can therefore receive substantial partial credit but still register as a failure in the headline completion rate. In a May 26 baseline, Harvey reported that the best all-pass rate among the frontier models it tested was 7.1%. That was a different evaluation run, so it should not be treated as a direct comparison with Z.ai’s leaderboard result.
Z.ai’s developer documentation describes GLM-5.3-FlashX-Legal as optimized for document understanding, legal reasoning, and task execution, including research, contract review, and drafting initial work products. The company says it used legal-domain training and reinforcement learning across tens of thousands of task environments modeled on legal workflows. Its October 13 AutoClaw blog post announced an application-based beta and said the model is built on GLM-5.3-Flash.
“An all-criteria pass rate of 19.2%”
— Z.ai, in its model documentation
What The Leaderboard Cannot Show
The leaderboard shows performance on a defined test set, not accuracy across all legal work, jurisdictions, or client matters. The source material does not provide enough detail to determine how representative the 120 private tasks are of real-world use, or how performance varies across the benchmark’s practice areas. The score also depends on two AI judges; no human-review result for this model is supplied here.
A separate four-task spot check described in the source article reported citations to rulings that the author could not locate. That small check is not a benchmark result and does not establish a general error rate, but it highlights why citations require verification. The source material also does not establish whether the legal model’s weights are publicly available, what data protections apply across deployment options, or whether Z.ai’s state-of-the-art claim uses a complete, clearly defined model set.
Beta Access And Independent Checks
Z.ai’s AutoClaw post dated October 13, 2026, opened a beta by application. The documentation page described in the source material does not carry a date, while Vals lists the model’s release as October 12. Further public leaderboard updates could show whether the score changes or how the model compares with additional submissions, but no upcoming evaluation or release milestone is specified in the available material.
For readers evaluating the announcement, the next useful evidence would include fuller documentation on data handling, model availability, and performance by task type, along with results from further independent testing. Legal professionals considering trials would need to check citations and substantive work before use and confirm how any client data is processed. The benchmark does not remove the risk of errors or the need for qualified human review.
Key Questions
What does the 19.17% score mean?
It is the share of evaluated tasks on which the model passed every required rubric criterion, according to Vals AI’s archived leaderboard. A task with even one failed criterion does not count as a pass.
Why is the criteria score much higher?
The 92.5% figure measures individual rubric criteria, while 19.17% measures complete tasks. A model may meet most requirements in a task and still fail its all-criteria test because one item is missing or incorrect.
How did GLM Legal rank on the leaderboard?
It ranked seventh among 81 models on Vals AI’s archived version of the Harvey Legal Agent Benchmark leaderboard. That rank applies to the cited archive, not necessarily to later leaderboard versions.
Does the result prove the model is safe to use for legal advice?
No. The result measures performance on a specific benchmark and does not establish reliability across legal matters. Outputs need qualified human review, including verification of citations and client-specific details.
Is the model’s state-of-the-art claim independently established?
Vals AI independently evaluated and listed the score, but Z.ai’s claim that it is state of the art among open-source models is not fully verifiable from the supplied material: the comparison set is not named, and published weights for the legal model are not identified.
Source: ThorstenMeyerAI.com
