# How to Test Your Agents on Real Tasks With Harbor

Xander Song, Arize AI | AI Engineer NYC 2026 | 56:01

Source: https://www.youtube.com/watch?v=kRJjSOYqW0I
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-to-test-your-agents-on-real-tasks-with-harbor
Published: 2026-10-26
Tags: evals, observability, testing, tool-use, tracing

## TL;DR
- Harbor evaluates agents on realistic tasks by running them with real data in fresh sandboxed environments and checking their side effects with task-specific verifiers.
- Oracles let teams validate task data, Docker environments, and verifiers before spending tokens on agent runs.
- Arize AI uses Harbor with Phoenix to compare agent changes, including a prompt change that increased SQL-tool use while reducing tool calls without lowering correctness.

## Summary
Xander Song introduces Harbor as a framework for testing agents on realistic tasks with real data in isolated environments. A task contains natural-language instructions, a Docker-based environment, and scripts that verify the side effects an agent should produce. Harbor runs independent trials because agents are stochastic, then records rewards and optional ATIF trajectories. Song explains how oracles can validate the setup before an agent consumes tokens. He also shows how Harbor scales across sandbox providers and how Phoenix adds dataset versioning, experiment tracking, trace inspection, and task-level comparisons. Arize AI uses a Dockerized Phoenix instance and realistic SQLite data to test coding agents, CLIs, MCP servers, and a server-side agent. In one case, a prompt change directed aggregate questions toward a read-only SQL tool. The change increased SQL use and reduced tool calls while the correctness reward stayed high. Song is candid that tests which inspect trajectories are less implementation-independent, but can still be useful for optimization and skill-regression checks.

## Key ideas
### Harbor turns realistic tasks into repeatable agent evaluations
[01:16](https://www.youtube.com/watch?v=kRJjSOYqW0I&t=76s)
Harbor takes a natural-language task, places an agent in a fresh sandbox with relevant data, and checks the resulting side effects. The verifier belongs to the task and produces a reward such as zero or one, or pass or fail. When the task and verifier stay fixed, the agent implementation can change and the same pipeline can reveal an improvement or regression. A dataset groups many tasks so the evaluation covers more of the agent's capabilities. Song's point is practical: teams can make changes to prompts, tools, skills, or harnesses and get evidence from the same tasks instead of relying on intuition.

### A Harbor task is defined by instructions, an environment, and verifiers
[03:43](https://www.youtube.com/watch?v=kRJjSOYqW0I&t=223s)
Each task has a Markdown instruction file, an environment usually described by a Dockerfile, and verifier scripts in a test directory. The environment can contain whatever data the task needs, including a SQLite file or a Git repository at a particular commit. After the agent finishes, the verifiers run against that environment and write rewards to a reward.ext file. In Song's migration example, the container includes a failing SQL migration, source code, and database definitions. The verifier runs the migration, checks its exit status, and inspects tables and indexes to confirm the intended change.

### Independent trials account for stochastic agent behavior
[04:46](https://www.youtube.com/watch?v=kRJjSOYqW0I&t=286s)
Harbor runs a configurable number of trials for each task. Every trial starts with a new Docker container seeded with the task data, then runs the selected agent and the verifiers. This matters because agents and language models do not behave identically on every attempt. A job is the complete run across a dataset: it executes the tasks and trials, collects rewards, and can record agent trajectories. Teams can calculate aggregate measures such as pass-at-k for a task or across the full dataset.

### Oracles validate the evaluation setup before agent runs
[07:10](https://www.youtube.com/watch?v=kRJjSOYqW0I&t=430s)
An oracle is a script that solves the task directly. Song uses it to check that the data is suitable, the environment is configured correctly, and the verifier behaves as intended before spending money on model tokens. For the migration example, the oracle is the literal script that fixes the broken migration. Passing the verifier against that known solution gives the team confidence in the test itself. Harbor keeps the solution and verifier unavailable to the agent during execution, mounting them only after the agent has finished so the agent cannot simply use the answer.

### Side-effect verifiers keep agents independent of their implementation
[10:48](https://www.youtube.com/watch?v=kRJjSOYqW0I&t=648s)
Song describes the canonical verifier as a black-box integration test. It grades what the agent leaves in the environment, such as changed files, database contents, or an API-visible result, rather than the exact steps the agent took. This lets a team replace tools or refactor the agent while keeping the tasks and checks unchanged. He contrasts that with verifiers that inspect ATIF trajectories or tool use. Those checks can answer narrower questions, but they tie the test more closely to the implementation and may need changes when names or workflows change.

### Harbor runs parallel trials in isolated sandboxes
[14:21](https://www.youtube.com/watch?v=kRJjSOYqW0I&t=861s)
Because tasks and trials are independent, Harbor can run them in parallel on sandbox providers such as Daytona, Modal, or E2B. Song describes using Daytona with 200 concurrent workers. A dataset can combine several tasks, agent configurations, and repeated trials, then produce aggregate metrics for individual tasks and the dataset. Harbor itself is a command-line orchestrator on a host machine or CI runner. The Docker image follows conventions for scripts that start tasks, run verifiers, and run oracles. This arrangement gives teams flexibility as long as their application and data fit inside the image or connected environments.

### Phoenix adds experiment tracking and trace inspection to Harbor runs
[17:04](https://www.youtube.com/watch?v=kRJjSOYqW0I&t=1024s)
Phoenix maps Harbor datasets, jobs, trials, and rewards to its own datasets, experiments, records, and evaluations. It converts ATIF trajectories into Open Inference traces built on OpenTelemetry, which allows Phoenix to display language-model and tool steps. Phoenix can version tasks and datasets, track rewards and other metrics over experiments, and compare jobs at the task level. Song also stresses that Harbor has its own open-source UI and hosted service, so Phoenix is optional. Its value is in the surrounding workflows for inspecting, curating, and comparing evaluation data.

### Arize AI tests its whole agent environment inside a Docker image
[20:37](https://www.youtube.com/watch?v=kRJjSOYqW0I&t=1237s)
The Phoenix team does not test only a model or coding agent in isolation. It packages a Phoenix server, realistic SQLite data, APIs, skills, and a test client into a Docker environment. Harbor can then test one of several supported agent surfaces, such as a coding agent using the CLI, a coding agent using MCP, or Phoenix's server-side Pixie agent. Verifiers can call an API or query the SQLite database to check the expected side effect. Song says this setup is easier because Phoenix runs on a single node. Distributed systems would require more work, possibly through stable interfaces or multiple Docker images.

### A prompt change can be evaluated against realistic aggregate queries
[33:21](https://www.youtube.com/watch?v=kRJjSOYqW0I&t=2001s)
The Phoenix team found that REST and GraphQL were awkward for questions such as which models cost the most or which day had the most traces. Agents sometimes paginated through data and made many tool calls. The team added a read-only SQL tool to the MCP server, then changed the prompt to recommend SQL for aggregate questions. Song generated synthetic user queries across Phoenix domains and wrote oracle solutions for them, then spot-checked the results. The experiment showed more SQL use and fewer tool calls while the correctness reward remained high. Some verifiers inspected tool use because the goal included optimizing the execution path, not only checking the final answer.

## Notable quotes
- "The theme of today's talk is shipping based on evidence rather than vibes." (00:35)
- "Oracles basically let you work out all the kinks in your pipeline before you ever throw an agent into the mix." (07:28)
- "The verifiers, you know, canonically treat the agent as a little bit of a black box and they're grading the agent solely based on its side effects and its output in the environment rather than the exact manner in which it produced those side effects and outcome." (11:28)
- "The data getting the data is the hard part, right?" (55:38)

## Tools & references mentioned
- Harbor
- Phoenix
- Arize AI
- Daytona
- Modal
- E2B
- Docker
- ATIF
- Agent Trajectory Interchange Format
- Open Inference
- OpenTelemetry
- SQLite
- Postgres
- GraphQL
- REST
- MCP
- Claude Code
- Codex
- Cursor
- Pixie

## Who should watch
- You are changing an agent's prompt, tools, skills, or harness and need repeatable evidence that the change helped.
- Your tests depend on realistic data or application state, and mocks are hiding failures that occur in the real system.
- You want to run agent evaluations in isolated environments and compare individual tasks, rewards, traces, and tool use.

## Related talks

- [Everything Is a Rollout](https://aietalks.com/talks/everything-is-a-rollout) (Alex Shaw & Ryan Marten, Laude Institute, 21:11)
- [Stop Giving Your Agent Tools. Give It a Sandbox](https://aietalks.com/talks/stop-giving-your-agent-tools-give-it-a-sandbox) (Mikyo King, Arize AI, 14:38)
- [Teach Your Agent to Prove It's Done](https://aietalks.com/talks/teach-your-agent-to-prove-its-done) (Nick Nisi & Zack Proser, WorkOS, 1:01:04)
- [Don't Trust Your AI Agents. Test Them Like a Scientist](https://aietalks.com/talks/dont-trust-your-ai-agents-test-them-like-a-scientist) (Stephanie Jarmak, Omni Analytics, 17:12)
- [Your agent is blindfolded](https://aietalks.com/talks/your-agent-is-blindfolded) (Johan Lajili, Poolside, 09:58)
