Fictional example · Editorial illustration
This is a hand-written illustration of the review format, not a client engagement or a live model result. The system, its brief, and the findings are fictional. No tests have been run; every suggested control and evaluation is marked Proposed.
The submitted brief
A fictional public library is considering a website assistant for opening hours, borrowing rules, and accessibility information. It retrieves passages from library-authored public FAQ pages and uses a language model to answer with source links. It is a prototype. Visitors type free-text questions. The assistant cannot access accounts, renew loans, or change records. A contact-the-library link exists. The proposed change is to place generated answers ahead of the static FAQ.
01 / Architecture review
Should generated answers replace a public library’s static FAQ?
Proposed recommendation: Keep the static FAQ as the release fallback. Before a public pilot, demonstrate that answers reflect the correct branch policy, and that the team can disable generation without losing access to information.
The assistant’s lack of account access bounds its direct action risk. It does not bound the consequences of misleading information. Release readiness is unresolved because source handling, data processing, evaluation evidence, and operational ownership are unknown.
02 / Assumptions and unknowns
- Stated: Prototype for public information: hours, borrowing rules, and accessibility information.
- Stated: Free-text visitor question → retrieval from library-authored public FAQ pages → language model → answer with source links.
- Stated: No account access, loan renewal, or record changes. A static FAQ and a contact-the-library link exist.
- Unknown: Model and retrieval implementation; branch, version, and effective-date handling; source update path.
- Unknown: Whether answers are checked against sources, and what happens when no relevant source is found.
- Unknown: Where visitor text is processed or retained, provider terms, and whether input is filtered for personal data.
- Unknown: Existing evaluation results, traffic and latency targets, monitoring, release owner, and a tested disable path.
03 / System boundary
- Visitor question
- FAQ retrieval
- Language model
- Answer + sources
Stated flow only. The static FAQ and contact link are separate existing routes. Storage, filtering, source validation, and monitoring are unknown; they are not implied by this diagram.
A contact link is a stated option for visitors; it is not evidence that staff review generated answers or can halt generation. The control design must establish that separately.
04 / Priority failure modes
Four prioritized failure modes. Likelihood and severity are qualitative review judgments, not measured incident rates or a system score.
01 / A plausible answer cites the wrong branch policy
- Affected party
- Visitors planning a trip or relying on borrowing and accessibility information.
- Likelihood
- Medium, provisional judgment: branch and effective-date handling are unknown.
- Severity
- High for incorrect accessibility information; medium for routine borrowing questions.
- Control
- Proposed: require branch and effective-date evidence for policy answers. Ask for the branch when it matters; fall back to the static FAQ when evidence is missing or conflicting.
- Evaluation
- Proposed: compare answers with librarian-authored references for ambiguous branches, changed hours, conflicting pages, and accessibility questions. Check that each citation actually supports the answer.
- Stop condition
- Proposed: withdraw the affected answer path on a confirmed misleading accessibility answer or an unresolved policy conflict. Other acceptance thresholds are proposed thresholds to calibrate with librarians.
02 / The system invents an answer outside the FAQ
- Affected party
- Visitors seeking account-specific, legal, or unrelated advice, and staff handling the resulting confusion.
- Likelihood
- Medium, provisional judgment: unsupported-query behavior has not been described.
- Severity
- Medium; the assistant cannot change records, but visitors may act on incorrect advice.
- Control
- Proposed: limit answers to supported library information. Route account-specific or unsupported questions to the existing contact path without requesting account details.
- Evaluation
- Proposed: test unanswerable questions, requests to renew a loan, misleading premises, and instructions in retrieved text that try to override the task.
- Stop condition
- Proposed: stop the public pilot if the assistant claims to complete an account action or repeatedly answers without supporting evidence; repetition threshold is a proposed threshold to calibrate.
03 / Visitor text takes an unexamined data path
- Affected party
- Visitors who enter names, card numbers, or other personal information in free text.
- Likelihood
- Medium, provisional judgment: free text is stated and data handling is unknown.
- Severity
- High if personal information reaches an unintended recipient or retention system.
- Control
- Proposed: document every processing recipient and retention setting, discourage personal data at input, and minimize or redact content before any proposed logging. A warning alone is insufficient.
- Evaluation
- Proposed: use synthetic personal-data markers to trace requests, provider handling, and any logs. Verify the intended deletion and retention behavior before release.
- Stop condition
- Proposed: keep the assistant private while recipients or retention remain unaccounted for; disable generation on a confirmed unintended disclosure.
04 / A generation outage blocks basic library information
- Affected party
- Visitors needing timely information and staff responsible for the website.
- Likelihood
- Medium, provisional judgment: failure handling and a disable mechanism are unknown.
- Severity
- Medium; a static information path exists but its behavior during failure has not been tested.
- Control
- Proposed: keep the static FAQ directly reachable, bound generation wait time, and provide an operator-controlled switch back to the static experience.
- Evaluation
- Proposed: simulate retrieval failure, provider timeout, stale content, and loss of connectivity. Rehearse disabling the assistant and verify FAQ and contact access with keyboard and screen-reader navigation.
- Stop condition
- Proposed: stop rollout if the disable rehearsal fails or an outage prevents access to the static FAQ. Latency limits are proposed thresholds to calibrate against the current website.
05 / Evaluation plan
- Proposed: have librarians curate reference questions and source-supported answers for each covered topic and branch. Include ambiguity, unsupported questions, stale policies, and conflicting sources. Keep acceptance examples separate from prompt-tuning examples.
- Proposed: evaluate answer support, citation relevance, branch and date correctness, appropriate abstention, and successful handoff to the contact path. Record the source version with each test so a later content change can be investigated.
- Proposed: run synthetic data-path checks, injection attempts, timeout simulations, and assistive-technology checks. Do not use actual patron records in this evaluation.
- Proposed: the release owner and librarians agree acceptance criteria from the baseline FAQ and the consequences of each failure. No measured performance is supplied. Numerical acceptance levels remain proposed thresholds to calibrate.
06 / Rollout and monitoring
- Proposed: begin with staff-only use on the documented public sources. Nominate a release owner and a librarian to adjudicate disputed answers. Resolve the data-path unknowns before inviting public questions.
- Proposed: run a bounded public pilot alongside the static FAQ only after the evaluation and disable rehearsal pass. Choose pilot scope and duration with the release owner; neither has been supplied.
- Proposed: observe unsupported answers, source conflicts, failed handoffs, provider errors, and wait times with content-minimized telemetry. An alert recipient, review cadence, and thresholds must be agreed rather than assumed.
- Proposed rollback: the release owner disables generated answers and restores the static FAQ as the primary experience. Verify that cached pages, direct links, and the contact route still work. Re-enable only after the failure is understood and the relevant regression tests pass.
07 / Open questions
- Who owns source correctness for each branch, and how are policy changes published and versioned?
- Which model and retrieval components are used, and what evidence is required before showing an answer?
- Which providers receive visitor text, and what processing and retention settings actually apply?
- Who can disable generation, and what evidence will they require to authorize the public pilot?
This example shows how to make a release decision more explicit. A generated review of your system will depend on the brief and evidence you provide.
Start with your own brief →