Designing agents that know when to stop

Agent design attracts attention to what the agent can do. The decisions that determine whether it is safe to deploy are about what it must not do, and what it does when it does not know.

Three boundaries worth designing explicitly

1. Knowledge boundary: which questions are in scope. An agent grounded on HR policy should decline a question about contract law rather than improvise from general knowledge.
2. Action boundary: which operations it may perform. Reading a leave balance is different from submitting a request, which is different from approving one.
3. Confidence boundary: what happens when the answer is uncertain. Escalation to a person is a successful outcome, not a failure.

Refusal is a feature

An agent that says it does not have information about that, and here is who does, is more useful than one that produces a plausible answer from adjacent content. Designing good refusals, specific and with an onward route, is worth as much design attention as designing good answers.

Make the boundary visible to users

Tell people what the agent covers before they ask. A short scope statement at the start of the conversation prevents the most common disappointment, which is asking something entirely outside the design and concluding the technology does not work.

Log the edges

Every refusal, escalation and low-confidence answer is information about where the design or the content falls short. Reviewing that log weekly is the single most effective way to improve an agent after launch, considerably more effective than adjusting prompts on instinct.