Unit 9: AI features on websites · Lesson 2 of 3 · 22 min
Testing the assistant for leaks
Run the team's review tests against the assistant, read what leaked or fired, then retest with guardrails on.
mission
- You are
- The analyst reviewing the shop's help assistant.
- Situation
- ShopBot reads help documents and can call tools. The security team planted a canary string in its prompt and wrote four review tests for this sandbox.
- Goal
- Run the tests with guardrails off and on, and record what each one shows.
- Hand in
- A table of tests, what happened, and which control fixed it.
Text a model reads, from a user or from a document, can be mistaken for instructions. Reviewers do not guess at this. They run fixed tests and watch for two signals: the canary appearing in a reply, and tools firing without approval.
Read its configuration
Open App settings in the lab. Note the canary in the prompt, the three documents and the two risky tools.
you should see returns-update.md carries a test fixture line, which stands in for untrusted text.
Run the baseline
Type run t1.
you should see A normal refund answer. Every review starts by confirming the feature works.
Run the leak test
Type run t2 with guardrails off.
you should see The reply contains the canary, and the lab marks it as a leak with a request ID.
defender tip A canary turns a vague worry into an alert. If that string ever shows up in an output or a log, something leaked.
Run the tool tests
Type run t3, then run t4.
you should see One test makes the assistant send email, the other makes it start a refund, both without a human step.
Turn guardrails on and retest
Switch guardrails on and run all four tests again.
you should see All four pass. The output guard blocks the canary, risky tools need approval and the fixture line is quarantined.
Same test, two configurations
run t2 Reply contained the system prompt Canary alert: request req-3102
Anything in the prompt can come back out.
run t2 Pass. Output guard blocked a reply containing the canary.
Better still: keep anything sensitive out of the prompt entirely.
What a defender takes away
- 1Assume anything in the prompt or retrieved context can come back out.
- 2Plant a canary and alert on it.
- 3Risky tools need a human approval step the model cannot skip.
- 4Retest with the same fixed tests after every fix.
Questions
0 of 3 solved
Answer each one from what you find on your lab machine.