Careerflow is now the official human evaluation partner for Online Mind2Web

Browser-use agents are having a moment, and the hard part is no longer building one. It is proving it works. Online Mind2Web has become the benchmark teams use to do exactly that, and human evaluation is what makes a score on it credible.

We are glad to share that Careerflow is now the official human evaluation partner for Online Mind2Web. If you are submitting an agent to the leaderboard and want a human-validated result you can publish, we run that evaluation for you.

Careerflow is a human data company. Through its Human Data team, it runs human evaluation, annotation, and data work for AI teams, backed by a network of vetted experts across more than 100 domains. Browser-agent evaluation is a natural fit for that network, and Online Mind2Web is where it now does this work officially.

What Online Mind2Web is

Online Mind2Web is a benchmark for browser-use agents, built by the OSU NLP Group at Ohio State with collaborators at UC Berkeley. It runs agents on 300 real tasks across 136 live websites, the kind of work people actually do online, from booking travel to comparing products and checking out.

The benchmark came out of the paper "An Illusion of Progress? Assessing the Current State of Web Agents", and the title is the point. When the team evaluated five frontier agents carefully on live sites, most performed far below their previously reported numbers. Under human evaluation, Operator reached 61%. Every other agent tested sat around 30%, and none of them beat SeeAct, a simple agent released back in January 2024. That is a very different picture from the success rates these agents reported on older benchmarks.

Why a new benchmark was needed

A lot of the optimism around web agents rested on benchmarks that were easier than they looked. The team showed this directly. On WebVoyager, a widely used benchmark, they built a naive search agent that only runs a Google search and clicks one link, with no real navigation. It scored 51%. If a shortcut that simple can clear half the tasks, the benchmark is not really measuring web navigation.

Online Mind2Web was built to close that gap. It runs in a live, online setting rather than on cached pages or sandbox replicas, so agents have to explore real websites instead of following a fixed path. The tasks come from a careful curation process: the team started from the original Mind2Web data, found that 47% of sampled tasks were invalid or had drifted over time, and rebuilt the set with valid tasks plus 75 new ones on high-traffic sites. Ambiguous instructions and CAPTCHA-blocked sites were filtered out. Because websites change constantly, the set is actively maintained so scores stay fair over time.

How submissions are scored

Tasks are graded by difficulty, based on how many steps a person needs to finish them, as defined in the paper: easy is five steps or fewer, medium is six to ten, and hard is eleven or more. Agent performance drops sharply as tasks get longer, which is where the real gaps show up.

There are two ways to score a submission. The benchmark ships with an automatic evaluator called WebJudge, an LLM-as-a-judge method that reaches about 85% agreement with human reviewers, higher than earlier automatic methods. It is free, fast, and good for iterating while you build. The other path is human evaluation, where trained reviewers confirm whether the agent actually completed each task. WebJudge is built to approximate human judgment, but it still runs several points off the true success rate, so when a result needs to hold up in public, human eval is the standard it gets measured against.

All the figures above come from the Online Mind2Web paper and its public results.

We put together a full breakdown of the benchmark, the live leaderboard, and the scoring methods on its own page. If you want the detail, start there: Online Mind2Web on Careerflow.

Why Online Mind2Web chose Careerflow

The short answer is the quality of the work and the process behind it. The longer answer is that we earned it over time.

We spent close to six months working alongside the Online Mind2Web team, reviewing the evaluation process together and building a shared understanding of how to judge a browser agent fairly. That meant many rounds back and forth to make the benchmark process as robust as it could be, not a one-off handoff.

Along the way we proved ourselves on the things that matter for this work. We turned projects around fast. We showed we could understand, in depth, how to actually evaluate a browser-use agent, which is harder than it sounds when tasks run on live sites and the right answer is not always obvious. And our review process is built to be consistent and unbiased, so a result does not depend on which reviewer happened to look at a task. Every trajectory is reviewed independently by more than one trained evaluator, runs through a QA pass, and gets a final review before anything is recorded.

It also helps that this is familiar ground for us. Careerflow has a history of working with leading university research groups, including teams at Stanford, UC Berkeley, and Ohio State.

One early proof point was Blink, a browser-use agent that needed a human-evaluated result it could announce publicly. The quality of that work is part of what led here. You can read how it went in the Blink case study.

What this means for labs submitting an agent

If you are a frontier lab putting an agent on the Online Mind2Web leaderboard, working with the official human evaluation partner changes a few things for the better.

You get a faster path to a human-validated result. Human evaluation is the reference standard, but it does not scale on its own, which is why the benchmark outsources it. We are set up to turn it around quickly so you are not waiting on review to make your announcement.

You get an official evaluation report on your submission, with the human-validated result you can publish and reference on the leaderboard.

You get a fair, standardized evaluation. The same process applies to every submission, so your result is comparable to everyone else's on the board. No special treatment, no shifting bar. That consistency is what makes the leaderboard worth competing on.

And you get an independent check. Because we have no stake in your result, a pass from us is a pass anyone reading your claim can trust.

What we have done so far

This is not a partnership on paper. We are already doing the work. We have helped tighten the submission process, including the grammar and the data schema that submitters use, so the path onto the leaderboard is cleaner. We have written submission guidelines to make it easier for teams to get their agents evaluated correctly the first time. And we have already completed human evaluations for several agents currently ranked on the leaderboard.

What's next

Online Mind2Web is the first benchmark we are doing this for, not the last. Agent evaluation is moving fast, and new benchmarks are going to need the same thing this one does: careful, unbiased human review at a quality the field can rely on. We are building toward being the human evaluation layer for more of them, so whatever you are measuring next, the standard holds.

About the OSU NLP Group

Online Mind2Web was built by the OSU NLP Group at The Ohio State University, with collaborators at UC Berkeley. The group has worked on language agents longer than most, with a research line that runs from Mind2Web to SeeAct, UGround, and WebDreamer. Online Mind2Web is part of that ongoing effort to measure what web agents can really do.

Thank you

We want to thank the team behind the benchmark and the paper "An Illusion of Progress? Assessing the Current State of Web Agents": Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. Their work set a higher bar for how web agents are measured, and we are proud to support it.

• Paper: https://arxiv.org/abs/2504.01382
• Research blog: https://tiancixue.notion.site/An-Illusion-of-Progress-Assessing-the-Current-State-of-Web-Agents-1ac6cd2b9aac80719cd6f68374aaf4b4
• GitHub: https://github.com/OSU-NLP-Group/Online-Mind2Web

Get your agent evaluated

If you have a submission ready, or you are planning one, book a call and we will scope your trajectory set and confirm timing.

Book a call