Medic diagnoses Spark failures by gathering evidence from logs, metrics, job metadata, and runbooks.
2
A multi-agent design with specialized prompts and tools gave the system more control than one large reasoning prompt.
3
Recorded production fixtures and offline evaluations let the team measure quality and catch regressions without repeatedly testing against live systems.
Summary
Drasko Profirovic describes Medic, an agentic tool that investigates failed Apache Spark jobs and produces evidence-based diagnoses with suggested fixes. The first prototype connected an LLM to internal data through MCP tools, but one large prompt produced inconsistent answers, consumed too much context, and was difficult to test. Pinterest improved it with trace-based observability, a fixture-backed end-to-end test harness, and offline evaluations. The system filters noisy exceptions by learning which ones appear in successful jobs, then ranks the remaining signals. A metrics sub-agent turns long time series into annotated graphs and returns a compact summary. The final architecture uses specialized agents for triage, hypothesis research, remediation, and report assembly. Runbooks in a vector database ground proposed fixes. The team found that multi-agent reasoning gave them more control, while workflow-based determinism was brittle. They are now testing feedback from previous sessions and considering applications to Flink and Trino.
The support burden makes automated Spark diagnosis useful
Profirovic describes data platform support as a continuous stream of questions from teams that depend on shared infrastructure. Spark is difficult to troubleshoot, especially for people who are new to distributed systems. Support engineers also have to choose between helping one team with a failing job and unblocking another team facing a deadline. Medic was designed to scale diagnostic knowledge on demand. A user can ask, "Why did a job fail?" and receive a research-style report with evidence for the root cause and fixes grounded in the job's context. The tool also needs to appear where engineers already work, including Slack and the Airflow UI.
The first MCP prototype depended too much on one prompt
The team first exposed data resources to language models through the Model Context Protocol. An operator could start a conversation with MCP tools enabled and ask the model to reason about a Spark job. The prototype worked, but only with careful prompting. Pinterest then created one reasoning-and-acting agent with a prompt that described its problem-solving method, report format, and examples of common failures. Beta trials exposed the limits of this design. Prompt tuning became unsustainable because adding detail in one area damaged behavior elsewhere. Answers could be shallow or excessively verbose, the team lacked controls over the agent, and large log outputs quickly consumed the context window.
Recorded production state turned testing into measurable evaluation
The team added OpenTelemetry traces sent to LangFuse so they could inspect the agent's execution as a waterfall of steps. They also built an end-to-end harness that snapshots production state. In record mode, the agent calls real downstream systems and saves tool responses as fixtures checked into the codebase. In playback mode, it analyzes those fixtures instead of live data and generates a report. Offline evaluations then grade the report. One example checks that the report contains no more than three suggested fixes. This approach replaced anecdotal manual testing with repeatable measurements and helped the team see whether changes caused regressions.
Spark logs contain many benign exceptions, so investigating only the last exception can point the agent in the wrong direction. The first filter used regular expressions and manually defined heuristics, which did not scale. The replacement classifier learns which exceptions commonly occur in successful jobs and treats them as likely red herrings. Medic fingerprints and clusters exceptions, then ranks them by content relevance and how close they occurred to job termination. The agent no longer reads the entire log directly. It uses one tool to retrieve the top K truncated exceptions and another to retrieve full details for a selected exception. This improved the signal-to-noise ratio and reduced misleading early anchors.
A metrics sub-agent compresses long time series into visual evidence
Raw time-series metrics are expensive to place in an LLM context, especially for long-running jobs. Medic sends metrics to a quarantine sub-agent that converts them into graphs and combines the graphs into one annotated image. The image resembles a Grafana dashboard and includes useful labels such as minimum and maximum values. Image input gave the team a predictable token cost regardless of job duration. The sub-agent can identify signals such as executors dropping to zero or near zero, long plateaus, bottlenecks, and resource behavior that does not match healthy progress. It summarizes those findings for the parent agent so the main context stays manageable.
Specialized agents give Medic control over each diagnostic step
Pinterest replaced the single reasoning-and-acting agent with a multi-agent architecture built on LangGraph's deep agent library. Each agent has its own prompt and a limited set of MCP tools. The library also supplies tools such as a to-do list and virtual file system to keep work organized. The workflow classifies the user's intent, then sends deep diagnostic requests to a triage agent. For a failed job, triage creates failure hypotheses, research agents investigate them in parallel, and each returns a score and root cause. A supervisor selects the highest-confidence cause, a healer agent proposes fixes from runbooks stored in a vector database, and the supervisor assembles the report.
The architecture made new capabilities easier to add
Separating agents made prompts easier to maintain and allowed focused testing with the end-to-end harness. Profirovic says that expanding the project's scope became as simple as adding a new prompt. The team used this pattern to extend Medic from failure diagnosis to Spark SQL optimization. The multi-agent design also gave them more control over system behavior. Better log handling substantially reduced inaccurate root causes. Pinterest tried LangGraph workflows to make the system more deterministic, but those workflows were brittle compared with the reasoning-and-acting approach.
Feedback and support for other distributed systems are the next experiments
The team is experimenting with using feedback from earlier user sessions to improve Medic automatically. Profirovic also sees the same diagnostic pattern applying to other distributed data systems, including Flink and Trino. The talk ends with these possible extensions rather than a claim that the current system has solved every production failure. The existing design depends on evidence selection, specialized reasoning, evaluation fixtures, and human-facing recommendations, so extending it will require adapting those parts to each system's logs, metrics, metadata, and runbooks.
"Our vision for a diagnostics agent was to ask it simply, "Why did a job fail?" and get back a deep research document which provides evidence on the root cause of the failure."01:28
Who should watch
You maintain Spark infrastructure and need a way to reduce the time engineers spend investigating failed jobs.
You are building an LLM agent that must reason over large logs and metrics without losing control of context or output quality.
Your team relies on manual production tests and wants a fixture-based evaluation loop for agent changes.
