The key idea
Evaluation compares outputs against defined task requirements. It should include realistic failures rather than only examples that make the tool look good.
Our method
Collect representative and difficult inputs, define a scoring rubric and keep a reference set separate from prompt tuning. Have reviewers assess correctness, completeness and brand fit. Record model and prompt versions, costs and intervention points.
How to assess progress
Monitor disagreement between reviewers and repeated failure types. A single aggregate score can hide an unacceptable error. Re-evaluate when the task, model, sources or workflow changes.
Decision
Decide which errors are unacceptable for this workflow and which imperfections a reviewer can reasonably repair. A wrong product claim may block release even when style and completeness score well. Write those blocking conditions before choosing the evaluation examples or comparing candidate configurations.
Exercise
Assemble twelve ordinary tasks, six difficult tasks and six cases that should trigger clarification or abstention. Have two reviewers independently score a subset without seeing the configuration name. Discuss disagreements and revise ambiguous rubric language before running the full comparison. Keep a separate set untouched until the workflow changes are finished.
Deliverable
Deliver an evaluation pack containing input cases, required evidence, scoring anchors, blocking errors and a results ledger. Record configuration, run date, review effort, task completion and failure category. Summarise both overall performance and the worst relevant failures so the launch decision cannot hide an unacceptable error inside an average.
Pitfall
A model judging its own answer can miss the same assumption that caused the mistake. Use independent checks for facts and calculations, and human review for consequential judgement. If reviewers disagree about the correct outcome, repair the task definition before treating a numerical score as meaningful.
The technology. The practical use.
Chatbots
An interface to ask questions, explore a brief or draft content with an AI model.
Read the guideAgentic AI
An agent chooses steps and uses tools toward a goal, within defined permissions.
Read the guideSkills
Reusable instructions, examples and scripts teach an agent how your team does a task.
Read the guideOfficial sources
Anthropic · Building effective agentsDemos
A synthetic CSV covers four campaigns across two periods. The question: does improving overall return hide a problem?
View detailsWant this workflow built for your team?
We help marketing teams save time, deliver more and improve business results with AI solutions, workflows, tools and practical training.
Discuss my AI workflow