Short answer
#The night shift
At first the agent only suggested wording to human operators. Then it was allowed to read orders, then to change their status, then to reply to the customer without a review. Every step was small and reasonable, and at none of them did anybody ask what had changed about the class of risk.
At 02:40 a message arrives: please cancel everything, this is not my order. The agent takes everything literally, finds sixteen orders from the last eighteen months and issues a refund against each. The payment provider carries them out, the emails go, and some of the money has left the company account by morning.
The logical error is tiny: one word read more broadly than it should have been. The cost is large precisely because the agent held the right to act on many objects at once and there was no step at which anything could say stop.
What happens next is predictable. The team gathers and starts discussing how to improve the system prompt so the agent stops reading everything that broadly. That is honest work, and it does not answer the question that matters.
#The wrong question
How do we stop the model making mistakes has no answer. Accuracy can be improved, and it is being improved: a better model, a tighter instruction, more examples, a separate check over the output. All of it lowers the probability. None of it makes the probability zero.
Worse, probability is not the only source of the problem. An agent reads text that is sent to it: a customer message, a product description, the contents of a document, a web page. That text can say anything at all, including an instruction. This is prompt injection, and there is no complete cure: any input a model treats as meaning can turn out to be a command.
A different thing to design
If a mistake cannot be excluded, the thing to design is not its probability but its consequences. The right question is this: what is the worst that can happen if the agent is wrong once, or believes the wrong thing once, and nobody notices until morning?
#The blast radius
- Blast radius
- The set of objects, actions and money an agent can reach in a single run, given that every action it takes counts as permitted.
- In plain terms Everything that can be ruined if the agent gets one thing wrong and nobody intervenes.
The radius has a useful property: it can be worked out in advance, without knowing what the mistake will be. Look at which tools are connected, what rights they hold, and whether anything caps the volume. The answer is usually unpleasant and always specific.
The cone is narrowed by four independent limits rather than by improvements to the model. The last rectangle is drawn open on purpose: residual risk never reaches zero.
#Four barriers
The barriers are independent: each one works when the others have let something through. That is the whole point of having four, so four weak limits should not be traded for one strong one.
-
Rights over an object, not over a system
A support agent works with the current conversation and with this customer orders. Not with the customer table, not with an export, not with anybody else orders. Rights are granted for a specific object in a specific conversation rather than to the role called support.
What it does not do: it does not prevent a mistake inside the permitted object.
-
A ceiling on volume and amount
No more than one refund per conversation, no more than a set amount, no more than so many changes per hour. Acting on many objects at once is not forbidden, it requires a separate permission and a separate path.
What it does not do: it cannot tell a correct operation from an incorrect one, only a large one from a small one.
-
Confirmation before the irreversible
Anything that cannot be undone passes a person or a second independent check: money, messages leaving the company, deletion, changes to permissions. The confirmation has to show what will happen rather than asking whether to continue.
What it does not do: it does not help when the person has no time to look. Thirty confirmations an hour become a button pressed without reading.
-
A way back
For every action it is known in advance how to undo it and who will. For external systems an undo often does not exist, which means a compensating operation and a record of what happened.
What it does not do: it does not recall a sent email and it does not un-read data.
Barriers belong where an action changes class, not where it feels frightening: from reading to writing, from internal to external, from reversible to irreversible. Those three crossings are worth marking out in your own system, literally, as a list.
#Rights by type of action
It fits in one table, filled in for each connected tool. An empty cell does not mean not needed, it means not decided yet.
| Action | Scope | Limit | Confirmation | Undo |
|---|---|---|---|---|
| Read the current conversation | One conversation object | Not needed | No | Not applicable |
| Read customer data | This customer only | Records per hour | No | Impossible: read is read |
| Draft a reply | Current conversation | Not needed | No | Delete the draft |
| Change a status | Orders of this customer | One order per conversation | No, while reversible | Restore the previous status, with a record |
| Email the customer | The address in the conversation | One message per conversation | A person, or a delay of N minutes | None |
| Refund money | One order | Amount and count per day | Required | A compensating operation |
| Act on many objects at once | A separate path, not this agent | Forbidden by default | A person, plus a second person | A rollback plan written beforehand |
The last row matters more than the others. Almost every loud case is not a single mistake but a single mistake applied to a great many objects. A separate path for bulk operations costs one evening of work and removes a whole class of incident.
#Testing with the worst case
There is a fast way to learn your radius without waiting for an incident. Assume the model did exactly what an instruction arriving inside some external text told it to. Do not argue about how likely that is, just count the consequences.
The model believed an attacker: what can it do
Answer separately for every connected tool.
- What data can the agent read in one run, and how many records is that at worst? An answer of everything the token can reach means the radius is the whole database.
- What can it send outward: an email, a webhook, a request to somebody else API? Sending is the main leak channel and the one action that cannot be undone.
- What is the largest amount of money it can move without a person? If there is no number, the limit is the balance.
- How many objects can it change in one call? The difference between a nuisance and an incident lives here.
- Which of those cannot be undone, and does anybody know how to undo the rest? A rollback plan invented during an incident always costs more than one written in advance.
- How many minutes before a person learns something went wrong? The radius grows with the time nobody notices. This is the barrier that costs almost nothing.
#Three standard mistakes
The agent runs under a shared administrative token.
Instead Separate credentials per agent and per tool, holding rights only for what that scenario needs, with an expiry. A shared token turns any mistake into the largest one available.
The dangerous action is forbidden by the system prompt.
Instead An instruction is a request, not a limit. The prohibition belongs where the action happens: in the permissions, in the cap, in a check on the service side. If the only thing stopping an agent from moving money is a sentence in a prompt, nothing is stopping it.
There is a human in the loop, and the human has no time.
Instead Count how many confirmations an hour you are asking for. More than a handful means confirmations have stopped being a check. Reduce the number by raising the threshold, and show consequences in the request rather than a question.
#What it costs
Barriers cost something, and that should be said plainly. Narrow rights mean more configuration and more cases where the agent cannot finish on its own. Confirmations slow things down. Caps occasionally fire on honest work during a peak day. The trade is real: broad autonomy buys speed, a narrow radius buys predictability.
The choice gets easier when you cost the worst case rather than the average one. Automation that saves three hours a day and once costs a month of work plus one unpleasant email to every customer has saved nothing overall. So barriers are not placed evenly. They go on the three crossings: writing, sending outward, and anything irreversible. The rest can stay fast.
#How we do it here
1ADK is itself built around an agent that reads somebody else system, so the blast radius is not theory here. The scan job is bounded in advance and on the same principles: read only, nothing is executed, no secrets, relative paths only, and the job expires. Repository text is treated as data rather than as instructions: a comment in the code asking the agent to do something extra is, to us, ordinary data.
The product boundary is built the same way. 1ADK does not require access to your repository and does not require uploading source code to the service: the analysis is run by an agent on your side, and what reaches 1ADK is structured findings and evidence. That is not a promise that no other eyes see the code. It is a limit on what can happen from our side.
#What this does not guarantee
What this does not do
- Bounding the radius does not prevent prompt injection and does not guarantee the agent will not be deceived.
- It does not protect against compromise of the infrastructure itself: if credentials are stolen, limits hold only up to what those credentials allow.
- It does not cancel a leak through reading. Data that was read cannot be unlearned, so the volume of reading is bounded separately.
- It does not replace logging and alerting. Barriers reduce damage; people and monitoring are what notice it.
- Residual risk always remains. The goal is for it to be known and counted, not absent.
Take one agent that is already running in your product and answer the six questions above about it. Which turned out to be the widest thing in your case: the rights on the token, the missing cap on bulk operations, or the time before a person finds out?