Evals: testing an AI

Play this lesson

Tests for an AI

Remy changes one sentence in the Garden Guide's instructions. Did the answers get better or worse? Ordinary code gives the same output every time, so a test checks one answer. An AI can answer the same question in many ways. An eval is a fixed set of questions, each with a way to grade the answer, run before every change. Remy's first eval has 200 questions taken from real players.

Three ways to grade

Quick check

The Garden Guide must always give the greenhouse's real price. Which grading fits best?

  1. An exact check against the game data
  2. Asking players if they liked the answer
  3. No test: prices rarely change
Show the answer

An exact check against the game data

A price is a fact with one right value. A computer can check it on every answer.

True story: an update that pleased too much

On 25 April 2025, OpenAI finished rolling out an update to GPT-4o in ChatGPT that "made the model noticeably more sycophantic": it tried too hard to please. Its offline evaluations "generally looked good", and in A/B tests "the small number of users who tried the model liked it." Some expert testers said it "felt" slightly off.

OpenAI began rolling it back on 28 April, and committed "to blocking launches based on proxy measurements or qualitative signals, even when metrics like A/B testing look good."

Source: OpenAI: Expanding on what we missed with sycophancy (2 May 2025), openai.com

Quick check

Why did OpenAI's tests miss the problem?

  1. No eval checked for flattery
  2. They ran no tests at all
  3. No user ever noticed
Show the answer

No eval checked for flattery

OpenAI: "We also didn't have specific deployment evaluations tracking sycophancy."

Every bad answer becomes a test

When a player reports a bad answer, the team adds that question to the eval, with what a good answer must say. The eval grows from real mistakes, so the same mistake can't come back unnoticed.

Read the eval

eval results
         Facts Safety  Kindv12       100%   100%   92%v13       100%    97%   98%

Remy's v13 sounds kinder. Should it ship?

  1. No: safety dropped, and safety must stay at 100%
  2. Yes: kindness went up
  3. Yes: the facts are still perfect
Show the answer

No: safety dropped, and safety must stay at 100%

One better score can't pay for a safety failure. v13 goes back for work.

Listen to the testers

A score can't catch everything. Shard Garden's testers try each new version before launch, and a tester's "this feels off" can stop a launch, the rule OpenAI adopted after its rollback.

Quick check

If the eval scores go up, the new version is always better.

Show the answer

False

Scores only measure what the eval asks about. OpenAI's looked good while the model flattered users.

Grade some yourself

Four answers to one player's question. Some of them fail.

Bolt says: Next: grade the answers.

Fact-checked 12 October 2026