.webp)
Galileo is an AI evaluation and observability platform, now part of Cisco. AI teams use Galileo's metrics to measure how their agents and models behave, then turn those evaluations into live guardrails that catch failures before they reach users.
Most of those metrics are powered by language models acting as judges. Instead of a person reading every output, a model scores it: did the agent advance the user's goal, is this answer grounded in the source, is this response toxic. That is what makes evaluation possible at production scale, where teams check outputs continuously rather than in occasional spot checks.
It also creates a hard problem. A judge is itself a model, and a model can be wrong. If the judge is unreliable, every decision built on top of it is unreliable too. Before Galileo ships a metric, the team has to prove the judge is right, and that proof comes from human-labeled evaluation data.
Galileo's data science team owns every metric on the platform. They build new metrics, improve existing ones, and are responsible for their accuracy. Their job is to make sure that when a customer trusts a Galileo metric to guardrail an agent in production, that trust is earned.
You cannot take a judge's word for it. An evaluation model has to be measured against a source of truth known to be correct, and the only reliable source of that truth is careful human judgment. For each metric, the team needs an evaluation set where humans have labeled every example by hand. Those labels are the benchmark the judge is held against. The bar is high: judges have to reach at least 90 to 95% accuracy against the human labels before a metric ships to customers.
Building those eval sets was the bottleneck. The data had to be labeled by people, accurately and consistently, at a quality high enough to validate a metric that would then judge other companies' agents. Galileo's eval data had to be more reliable than the production systems it measured. Each metric needed its own set, the labeling process had to be built and ramped quickly, and the hard edge cases had to be resolved before a metric shipped, not after a customer hit them.
Careerflow's work came in three parts: sourcing the right people, building and iterating the labeling approach for each metric, and delivering reviewed, high-quality eval sets.
Careerflow staffed the work from its global expert network, and every person who touched the data passed through multiple layers of vetting: an AI interview, a human interview, skills assessments, and final selection onto the labeling team. That filtering is what produces a high quality bar, and it is the same sourcing, vetting, assessing, and training pipeline Careerflow runs across its data business. The result was a stable, expert team that has delivered consistent annotations for more than a year.
For each new metric, Careerflow moved fast. The team quickly stood up a new labeling tool and data pipeline, then iterated on the metric's definition and its edge cases until the task was airtight.
The working rhythm was tight and collaborative. Galileo would send a Google doc describing a metric, and Careerflow would come back with recommendations on how to handle the tricky cases. There were times Careerflow improved the definition of a metric itself, with clear reasoning for where Galileo's original definition would not hold. That is the kind of input that only comes from a team that understands the evaluation problem, not just the labeling task.
An example: Action Advancement. One of the metrics Careerflow built eval sets for was Action Advancement, which measures whether an agent actually makes progress toward what the user asked for, rather than just producing a response. Labeling it well means judging, case by case, whether a given agent action genuinely moved the user's goal forward. Getting that right at scale, consistently, across thousands of examples, is exactly the kind of judgment the vetting process is built to find.
Quality control was built into every set. Careerflow used a three-opinion model on the data, plus a dedicated QA layer, so each label reflected multiple expert judgments rather than one person's call. Across the engagement, the team produced more than 10,000 unique samples spanning Galileo's agent evaluations.
Throughout, Careerflow operated at speed, held a high quality bar, and kept open two-way communication with Galileo's team. In practice, Careerflow worked as an extension of Galileo's own data science team: Galileo's scientists stayed focused on building metrics, while the evaluation data underneath them was handled end to end.
Careerflow's labeled eval sets supported metrics across Galileo's suite, spanning agent behavior, retrieval quality, and safety. Galileo used that ground truth to validate and ship more than ten metrics to production. Some of the metrics include:
The eval sets Careerflow builds do double duty. Galileo uses them to benchmark its own internal evaluation models, and it benchmarks frontier models against the very same sets. In other words, Careerflow's labels are the yardstick Galileo holds both its own metrics and the leading models in the field against.
With Careerflow's evaluation sets, Galileo validated its judges against a strict accuracy bar and shipped more than ten metrics to production. Those metrics now run for Galileo's own customers, who use them to guardrail and observe agents in production. Galileo's evaluation work is part of what made the platform successful enough to be acquired by Cisco, a Fortune 500 company.
The partnership continues today, including after the Cisco acquisition. Careerflow keeps building eval sets as Galileo expands its metric coverage, and the scope is now growing into multimodal metrics. For Galileo's data science team, the value is not just the labels. It is a data partner that moves at their speed, communicates openly in both directions, and takes ownership of the evaluation pipeline so the team can stay focused on building metrics.
