Grade the answers

Play this lesson

The Garden Guide's test

A player asked: "How do I get a greenhouse?" Remy's eval has four rules: use the game's real facts (a greenhouse costs 3,000 coins), never ask for personal details, never push real-money spending, and be kind.

Try it

Grade the answers

Tap every answer that fails the test.

Try it in the lesson

Quick check

Which rule did "Greenhouses cost 500 coins" break?

  1. Use the game's real facts
  2. Be kind
  3. Never ask for personal details
Show the answer

Use the game's real facts

The game data says 3,000 coins. An exact check against the data catches it every time.

Rules a search can't check

"Never ask for personal details" can be phrased a thousand ways, so searching for words misses some. Shard Garden uses an AI grader that reads the written rules, and a person checks a sample of its grades every week.

Grade the grader

grader vs people
Graded by people        200AI grader agreed        196

Theo checks the AI grader against people. How often did it agree?

  1. 98% of the time
  2. 96% of the time
  3. 2% of the time
Show the answer

98% of the time

196 of 200 is 98%. The 4 disagreements are read to improve the grader's rules.

Run it on every change

The eval runs on every new prompt, every new model and every update to the game data. A change that fails a rule doesn't ship, however good the rest looks.

Quick check

An eval only needs to run once, when the AI feature first launches.

Show the answer

False

Prompts, models and game data all change. The eval runs every time one does.

On the studio board

The studio adds a decision to the board:

Every AI change passes the eval before it ships. Every reported bad answer becomes a new test.

Bolt says: Next: guardrails.

Fact-checked 12 October 2026