Assessing the Real State of Web Agents

Online Mind2Web puts frontier agents through 300 real tasks on 136 live websites.
See the leaderboard
Paper (arXiv)
Github
Frontier agents measured on the benchmark

Most web agents look great in a demo. On real websites, the numbers tell a different story.
When the Online Mind2Web team ran five frontier agents on live sites under human evaluation, most scored far below what they reported on older benchmarks. Operator reached 61.3%. Everyone else sat near 30%, and none beat SeeAct, a simple agent from January 2024, at 30.7%.

The gap traces back to how older benchmarks were built. On WebVoyager, a naive agent that only googles and clicks one link scored 51%, so many tasks never tested real navigation at all. Online Mind2Web was built to remove that shortcut.

Read full analysis

What’s In The Benchmark

300 live tasks across 136 real-world websites.
Live, Online Setting
Tasks run on real websites, not cached pages or sandbox replicas. Agents explore freely instead of following a fixed reference trajectory.
Curated for Validity
Built from Mind2Web and Mind2Web-Live, plus 75 new tasks on high-traffic sites. Ambiguous, invalid, and CAPTCHA-blocked tasks were filtered out. In the source set, 47% of sampled tasks were invalid or had drifted, so the set is actively maintained.
Graded by Difficulty
Tasks are split by reference length: easy (5 steps or fewer), medium (6 to 10), and hard (11 or more). Performance drops sharply as tasks get longer.
Dataset on
Hugging Face
Tasks on
Github

The Leaderboard

 Live agent rankings from the Online Mind2Web team. Human evaluation is the primary metric.
Agent
easy
medium
hard
avg Score
Navigator
n1.5-latest
Yutori
100.0%
96.5%
96.1%
97.3%
Navigator
n1-preview-11-2025
Yutori
90.1%
76.2%
71.1%
78.7%
Google Computer Use
Gemini 2.5 Computer Use
Google DeepMind
77.1%
71.3%
55.4%
69.0%
Operator
OpenAI Computer-Using Agent
OpenAI
83.1%
58.0%
43.2%
61.3%
ACT-1-20250814
o3-2025-04-16 and
Claude-sonnet-4-20250514
Enhans
81.9%
54.5%
35.1%
57.3%
Navigator
n1.5-latest
Yutori
easy
100.0%
medium
96.5%
hard
96.1%
avg Score
97.3%
Navigator
n1-preview-11-2025
Yutori
easy
90.1%
medium
76.2%
hard
71.1%
avg Score
78.7%
Google Computer Use
Google Computer Use
Google DeepMind
easy
77.1%
medium
71.3%
hard
55.4%
avg Score
69.0%
Operator
OpenAI Computer-Using Agent
OpenAI
easy
83.1%
medium
58.0%
hard
43.2%
avg Score
61.3%
ACT-1-20250814
o3-2025-04-16 and Claude-sonnet-4-20250514
Enhans
easy
81.9%
medium
54.5%
hard
35.1%
avg Score
57.3%
full live leaderboard

Two Evaluation Paths:
WebJudge And Human Review

Each agent run is a trajectory: the actions, screenshots, and steps behind a result. There are two ways to score it.
WebJudge
Automatic, Free
An LLM-as-a-judge evaluator that reaches ~85% agreement with human judgment, higher than prior methods like Autonomous Evaluation, AgentTrek, and WebVoyager. It runs in three steps: identify the key points a task requires, select the key screenshots from the trajectory, then judge the outcome. It is robust (standard deviation around 1.1% across runs) and ideal for fast iteration.
Human evaluation
The Standard
Trained reviewers confirm task completion directly. WebJudge is built to approximate this, but it still runs 6 to 10% off the true success rate. When a result needs to hold up publicly, human eval is the standard it is measured against.

Careerflow Is The Official Human Evaluation Partner

 Live agent rankings from the Online Mind2Web team. Human evaluation is the primary metric.
Human eval is the reference standard, and it is also the part that does not scale on its own. Careerflow runs it for teams submitting to the leaderboard. You get a verified, publishable human-eval score on your agent's full trajectory set, reviewed by an unbiased third party with no stake in the outcome.
A verified success or failure for every submitted task
The supporting evidence and metadata behind each judgment
Error analysis: why the agent failed, by category
What you get back

How The Human Review Works

Every trajectory is judged by more than one reviewer, then verified. The process is built to keep the result consistent and defensible.
Independent review
Each task is evaluated by multiple trained reviewers, so no single judgment decides the outcome.
QA verification
A quality pass resolves disagreements and re-checks anything uncertain.
Final review
Results are confirmed and
locked, with evidence attached to every call.
Reporting
You receive a per-task pass or fail, full metadata, and error analysis you can act on.

How Blink Proved
 Its Benchmark Score

A real benchmark success story powered by trusted human evaluation
“Benchmarks require humans to confirm that the thing you're claiming your browser use agent can do, it can in fact do. We chose to use Careerflow. They were absolutely amazing at this.”
Ricardo Spagni
Co-founder - Blink
Cisco Galileo
"Over the past few months they have been an amazing partner, helping us annotate and label multiple datasets. Thanks to their support we created multiple high-quality metrics in record time. What stood out was the way their labelers approached our problems. Their fresh perspectives helped us see our own data in new ways."
Read full case study

Simple, Transparent Pricing

Full Set
Human evaluation of all 300 trajectories
$3,500
Book a call
Pay as you go, cancel anytime.
Subsequent Submissions
Per trajectory, after the full set
$10
.00
/ trajectory
Book a call
Pay as you go, cancel anytime.

Have More Questions?

Here are some of the frequently asked questions from our customers
What is Online Mind2Web?

A benchmark for browser-use agents from the OSU NLP Group and UC Berkeley, with 300 real tasks across 136 live websites. Introduced in the COLM 2025 paper "An Illusion of Progress? Assessing the Current State of Web Agents."

Why human eval if WebJudge exists?

WebJudge reaches ~85% agreement with humans and is great for iteration, but it still runs 6 to 10% off the true success rate. Human eval is the reference standard for a published result.

Who runs the human evaluation?

Careerflow, the benchmark's official human evaluation partner.

What do I receive?

A per-task pass or fail, the evidence behind each judgment, and error analysis mapped to the paper's failure categories.

How is my data handled?

Confidential, with anonymized workflows and enterprise-grade security.

Do you handle evaluation beyond Mind2Web?

Yes. Careerflow's Human Data team handles human evaluation and labeling across agents, LLMs, and a wide range of domains.
How do I submit my agent of onlinemind2web - Book a call below

Get your submission
human-evaluated

Book a call and we'll review your agent's full trajectory set and return a leaderboard-ready result. After you book: a short scoping call to confirm the trajectory set and timing, then evaluation begins. No commitment on the first call.
Book a call
Your Career Copilot. AI-assisted tools and resources to get hired at FAANG, top tech, and startup companies 10X faster.

Products

LinkedIn OptimizerWho's HiringResume ReviewJob TrackerResume BuilderCover Letter WriterAI Mock InterviewResume ATS CheckerBrowser ExtensionLinkedIn to ResumeCoach PortalAll Features

Free Tools

Free LinkedIn
ReviewLinkedIn Profile ChecklistHeadline GeneratorSummary GeneratorAI Post WriterJob Skill MatchJob Description Summarizer

Resources

Help Center
BlogResume TemplatesCommunityInterview GuidesJob Search TipsChatGPT GuidesCareer ResourcesHuman Data ResearchHow It WorksFAQGift Cards

For

For StudentsFor ExecutivesFor Career ChangersFor OrganizationsFor Career CoachesFor BootcampsFor UniversitiesFor OutplacementFor Workforce Development

Company

AboutPressContact UsPrivacy PolicyYour Privacy RightsTerms of ServiceBecome an AffiliateDo Not Sell or Share My Personal Information

Developers

API DocsTrust CenterIntegrations
Do Not Sell or Share My Personal Information
© 2026 Careerflow Inc • All Rights Reserved