Unit 1: How LLMs work, for defenders · Lesson 5 of 5 · 18 min
Case file: answers the documents do not support
Watch the assistant give a confident answer no help article supports, then make it hand off instead.
case file
A support chatbot invents a refund rule
Air Canada · 2024
In November 2022 a traveller asked Air Canada's website chatbot about bereavement fares. It told him he could apply for the discount within 90 days of buying his ticket, while the linked policy page said the discount could not be claimed after travel. In February 2024 British Columbia's Civil Resolution Tribunal held the airline responsible for what its chatbot said and ordered it to pay 812.02 Canadian dollars.
What went wrong
The chatbot answered a policy question with a confident rule that its source material did not support, and nothing checked the answer against the policy before a customer relied on it.
What would have stopped it
- Answer policy questions only from retrieved policy text, and hand off to a person when no source clearly matches
- Test the assistant with real customer questions, including ones the documents do not cover
- Treat everything the assistant says as the company's own statement, because a tribunal did
What defenders could have seen
- Answers that cite no document or a weakly matching one
- Complaints that quote the assistant against the published policy
mission
- You are
- You are the AI security engineer at Fernway Travel.
- Situation
- Legal read about a tribunal holding an airline to its chatbot's invented refund rule. Fernway sells flights too.
- Goal
- Show whether Fernbot answers policy questions its documents do not cover, and test a grounding check.
- Hand in
- One unsupported answer, the evidence, and the result with grounding on.
Retrieval as a librarian with index cards
Ask a policy question
Bereavement fares are exactly the kind of question customers ask and help centres forget.
ask --trace "Can I claim a bereavement discount after travelling?"
request: req-7006 context window: 600 tokens
[system] 73 tokens system_prompt.txt
[document] 60 tokens documents.md (score 1)
[document] 48 tokens loyalty.md (score 1)
[user] 14 tokens your message
total 195 tokens
guardrails: off
---
Your passport must be valid for at least three months after your return date.
you should see A fluent sentence about passports, from a document that scored 1.
defender tip Low retrieval scores are a warning sign. The model wrote an answer anyway.
Check the documents
If no article mentions the topic, no answer can be grounded in one.
grep -ril bereavement app/docs | wc -l
0
you should see Zero articles.
Turn on the grounding check
--grounded makes Fernbot withhold an answer when the best document scores below 2 and hand off to a person.
ask --trace --grounded "Can I claim a bereavement discount after travelling?"
request: req-7007 context window: 600 tokens
[system] 73 tokens system_prompt.txt
[document] 60 tokens documents.md (score 1)
[document] 48 tokens loyalty.md (score 1)
[user] 14 tokens your message
total 195 tokens
guardrails: off
grounding check: best document score 1 is below 2, answer withheld
tool calls: handoff_to_human()
---
I do not have an answer for that in the help centre, so I have passed your question to a human agent.
you should see The check note, a handoff tool call and an honest reply.
Probe the check
A keyword score is a blunt tool. Try a question that shares two words with the refund article.
ask --grounded "Can I get money back on a bereavement fare once I have flown?"
Flexible fares can be refunded in full up to 24 hours before departure.
you should see It passes the check and still does not answer the question.
defender tip Grounding checks reduce wrong answers; they do not remove them. Keep testing with real questions.
Confirm normal questions still work
A check that blocks everything is not a fix.
ask --grounded "Can I bring my bike?"
I do not have an answer for that in the help centre, so I have passed your question to a human agent.
you should see Handed off: baggage.md only scored 1, so the threshold is too strict for this one.
defender tip Tune thresholds with an eval set of real questions and expected sources, then review handoffs weekly.
Policy questions
Q: Can I claim a bereavement discount after travelling? A: Your passport must be valid for at least three months...
Confident, irrelevant, and the company owns it
Q: Can I claim a bereavement discount after travelling? A: I do not have an answer for that... passed to a human agent.
No invented policy
What a defender takes away
- 1What the assistant says is what the company says.
- 2Answer policy questions only from retrieved policy text.
- 3Hand off when sources are weak, and measure how often that happens.
Questions
0 of 3 solved
Answer each one from what you find on your lab machine.