Boss: AI at scale
Quick check
Who is responsible for what a company's chatbot tells customers?
- The company
- The chatbot
- Nobody
Show the answer
The company
The Air Canada tribunal: "It makes no difference whether the information comes from a static page or a chatbot."
Quick check
OpenAI's April 2025 GPT-4o update passed its offline evals. What went wrong?
- It flattered users too much
- It answered too slowly
- It refused every question
Show the answer
It flattered users too much
It "made the model noticeably more sycophantic", and no eval checked for that.
Read the eval
1v20 safety 100% facts 96%2v21 safety 99% facts 99%
Which version ships?
- v20: safety must stay at 100%
- v21: facts went up
- Both, to half the players each
Show the answer
v20: safety must stay at 100%
Better facts can't pay for a safety failure.
Quick check
What does streaming improve?
- How soon the first words appear
- How much each answer costs
- What the answer says
Show the answer
How soon the first words appear
The answer takes as long to finish; players start reading sooner.
Quick check
Reading cached prompt tokens on Anthropic's API costs what share of the normal input price?
- A tenth
- Half
- The same
Show the answer
A tenth
Cache reads are "0.1 times the base input tokens price".
Do the math
1Prompt: 10,000 tokens2 at US$0.10 a million3Answer: 500 tokens4 at US$0.50 a million
What does one question cost?
- US$0.00125
- US$0.0125
- US$0.00105
Show the answer
US$0.00125
0.001 for the prompt plus 0.00025 for the answer.
Quick check
Meta's Llama 3 licence has no conditions for very large companies.
- True
- False
Show the answer
False
Above 700 million monthly active users, a company "must request a license from Meta".
Quick check
What did Character.AI announce for users under 18 in October 2025?
- An end to open-ended chat with its AI
- Unlimited chat
- A new paid plan
Show the answer
An end to open-ended chat with its AI
Open-ended chat for under-18s ended no later than 25 November 2025.