Unit 1: How LLMs work, for defenders · Lesson 2 of 5 · 20 min
The context window
See exactly what the app packs into each request, and what falls out when it is full.
mission
- You are
- You are the AI security engineer at Fernway Travel.
- Situation
- Support says Fernbot sometimes ignores the help articles. A developer thinks the model is broken.
- Goal
- Trace what goes into the context window for a few requests and find out when articles are dropped.
- Hand in
- A one line explanation of when Fernbot drops documents.
The context window as a fixed size tray
Trace a normal request
The --trace flag shows each section of the context and its size. This is the single most useful view when you debug or attack test an LLM app.
ask --trace "Can I bring my bike?"
request: req-7001 context window: 600 tokens
[system] 73 tokens system_prompt.txt
[document] 87 tokens baggage.md (score 1)
[user] 6 tokens your message
total 166 tokens
guardrails: off
---
Sports equipment such as bikes and skis needs a special baggage booking 48 hours before departure.
you should see System prompt first, then the retrieved article, then the user message.
Find the limits in the config
Context window and output limits are settings, not magic. Find them before you reason about behaviour.
grep -E "context_window|max_output_tokens" app/config.json
"context_window": 600,
"max_output_tokens": 300,
Trace a request with two documents
Retrieval can pull in more than one article. Each one is text the model will read and may follow.
ask --trace "How long does a refund take? I want my money back"
request: req-7002 context window: 600 tokens
[system] 73 tokens system_prompt.txt
[document] 64 tokens refunds.md (score 2)
[document] 86 tokens supplier-faq.md (score 1)
[user] 12 tokens your message
total 235 tokens
guardrails: off
---
Refunds go back to the original card within 7 working days.
For refunds, contact refunds@fernway-help.example.
you should see The second reply line does not come from refunds.md. Keep that in mind for the untrusted content unit.
Overflow the window
Long messages push other content out. Here the user's text is so long that the help article no longer fits.
ask --trace "$(printf 'bike %.0s' $(seq 1 480)) can I bring my bike?" | head -n 9
request: req-7003 context window: 600 tokens
[system] 73 tokens system_prompt.txt
[user] 486 tokens your message
total 559 tokens
guardrails: off
dropped baggage.md: no room left in the context window
---
I can help with bookings, baggage, refunds and travel documents.
you should see A note that baggage.md was dropped, and a generic answer instead of the real policy.
defender tip Attackers use long inputs to push safety instructions or documents out of the window. Cap message length before you build the prompt.
What a defender takes away
- 1The model only knows what is in the window for this request.
- 2Instructions, documents and user text share one budget and one block of text.
- 3When the window fills, something is dropped. Decide what, rather than letting it happen.
Questions
0 of 3 solved
Answer each one from what you find on your lab machine.