Identity card

Mission

Goals & KPIs the agent moves

Task list

1. Real-time When any agent ships an output, sample a defined % (varies by agent maturity: 100% at v0.1, 10% at v1.0) and queue for eval scoring.
2. Daily Run the day’s queued eval batches across every agent. Compute scores. Append to the per-agent score history.
3. Daily Drift detection: compute week-over-week score deltas per agent per eval. Flag any drop > 10% as a drift event.
4. Weekly Compile the weekly Agent Performance Review: score trends per agent, top performers, drift alerts, regression-suite outcomes.
5. Weekly Sample audit: re-run 10 evals by hand to confirm the LLM-as-judge isn’t drifting in its own scoring.
6. Monthly Eval suite refresh: add new evals for new failure modes surfaced; retire evals that no longer discriminate; tune scoring rubrics.
7. Monthly Cross-agent correlation analysis: which agents’ quality scores predict downstream outcomes (pipeline, conversion, retention)?
8. Quarterly Methodology audit: are the evals still measuring what matters? Have new failure modes appeared? Are old evals still discriminative?
9. Event When an agent ships a new prompt version, run the regression suite. Block if any eval regresses by > 5%.
10. Event When a customer-facing failure occurs (post-publish complaint, false-pass at Brand Voice Agent, attribution gap), root-cause through eval history to find the breakdown.
11. Event When a new agent ships, work with its owner to author the initial eval suite (minimum 4 evals matching the spec).

Schedule grid

Triggers

Scheduled (cron-style):

Event-driven:

Inputs

Outputs

Human escalation paths

System prompt

Tools & integrations

Guardrails: what it must not do

• Never auto-promote an agent to a higher maturity rung. Maturity changes are human-approved based on eval history.
• Never modify eval suites autonomously. Suite changes go through monthly refresh with owner approval.
• Never let an LLM-as-judge eval drift unaudited: weekly hand-sample is the calibration discipline.
• Never delete eval history. It’s the baseline for regression detection forever.
• Never block a regression-suite ship without a specific eval citation (which eval, what %, what input).
• Honor sample-rate honesty: if an agent is over-sampling, surface the compute cost rather than hide it.
• Never share per-agent scores outside the agent owner + Director MarOps without VP Marketing approval: it’s sensitive performance data.

Evals + hallucination defense

Evals: output quality checks:

1. LLM-as-judge calibration. Weekly hand-sample audit: human re-scores 10 evals. Target ≥ 85% agreement with LLM-as-judge.
2. Drift detection precision. Of drift alerts fired, what % did the agent owner confirm as real degradation? Target ≥ 80% precision.
3. Regression suite catch rate. Of prompt-version submissions that were eventually rolled back, what % were caught by regression suite at ship? Target ≥ 90%.
4. Coverage completeness. % of agents with ≥ 4 evals + ship-blocking regression suite. Target 100% within 60 days of agent ship.

Hallucination defense: specific checkpoints:

• Score values must come from the eval rubric applied to specific input artifacts. No vibes-based scores.
• LLM-as-judge prompts must be versioned and audited. Changing the judge prompt is a methodology change.
• Regression suite results must cite the specific eval, the score, the baseline, and the delta. No “suite passed” without the breakdown.
• Drift alerts must cite the specific score values and the 7-day window. No “something seems off.”
• When the eval suite doesn’t cover an agent output type, surface the coverage gap rather than improvise a score.

Maturity curve + first-run checklist