Short answer
#The day that was not in the calendar
Ask your team when the AI assistant in your product stopped being a chatbot. Usually nobody can name the date, because nobody made a decision that day. There was a ticket: give the agent access to order statuses so it stops asking.
The sequence is almost always the same. First the agent suggests wording to human operators, and its mistakes are seen by a person before anybody receives them. Then it is allowed to read data, so the answers get more accurate. Then to change a status, to save the operator a click. Then to send the message itself, because the operator was only pressing a button anyway. Every step saves time and looks like a continuation of the one before.
But the steps are not of one kind. Between the second and the third there is a boundary, and past it a mistake by the model stops being text and becomes an event in your system. That boundary is crossed by connecting a tool rather than by deciding anything, which is why nobody notices it.
#Five rungs
It helps to lay the permissions out as rungs and see that what grows is not the intelligence of the agent but the cost of one mistake.
Height is not the difficulty of the task. It is how much of a mistake survives being discovered.
| Rung | Worst single mistake | What has to exist before connecting |
|---|---|---|
| Advises a person | An operator sent an inaccurate answer | Nothing special: a person sees the mistake before it is sent |
| Reads data | Showed one customer the details of another | Rights over the conversation object rather than the table; a cap on records |
| Changes a record | Changed a status, a plan or a delivery address for the wrong person | A change log holding the previous value, and a way to put it back |
| Sends outward | A message went to the wrong person, or carried somebody else data | A re-check of the fact by another route, a delay, or confirmation |
| Does the irreversible | A refund, a deletion, a permission change, a publication | Mandatory confirmation, an amount cap, and a compensating operation written in advance |
Notice that the third column contains nothing about the model. Everything in it is built in your application and stays true when the model is replaced by any other.
#Four verbs
The conversation about agent permissions becomes tractable when it is reduced to four verbs. For every connected tool, say what the agent can read, change, send and delete through it. Not what it does in your scenario: what the access technically allows.
The gap between those two questions is the whole risk. A scenario describes intent, access describes capability. A model makes mistakes and is deceived inside the capability, not inside the intent.
A support agent was given CRM access so it could see a customer history of enquiries. The ticket says exactly that: read the history only. Through the API the same token can read the whole contact database, page through an export of it, and change deal fields, because the CRM had one role available, manager, and no narrower one existed.
Nobody intended to give the agent the contact database. It was handed over by the answer to which role shall we give it, decided in twenty seconds late on a Friday.
#Why the list grows on its own
This subject has an unpleasant property: it moves while you are not touching it. Agent permissions are added one at a time, each for a good reason, and almost always by somebody who is not required to think about consequences outside their own ticket.
What gets spent is not budget but three other things. First, knowledge of what the agent can do: it stops existing in one head at around the fifth tool. Second, reversibility: each new permission carries some chance of being irreversible, and nobody writes the compensating operation in advance. Third, time to discovery: the more actions happen without a person, the later a person learns that one of them was wrong.
Three months in it looks like this: twice as many tools, no written list anywhere, and the same rights on the agent token that were granted on the first day, because nobody asked for them to change. A year in, the people who were in the original conversation have moved on. Answering what can our agent do to the customer database becomes a several-day investigation.
There is one cheap way out and it is dull: the permission list is reviewed whenever integrations change, rather than on a schedule. Every connection is a reason to re-read the table of four verbs, and that takes minutes for as long as the table exists.
#An inventory you can finish this week
This is not a quarter of work. The tool list of any agent is short, usually between three and ten entries. Take the four verbs and fill the table in for each. Where you do not know the answer, write a question mark and ask a developer: that is the most valuable part of the exercise.
For each connected tool
Answer from what the access allows, not from what the scenario intends.
- What can the agent read through this tool at worst, and how many records is that? An answer of everything the role can see means the reading volume equals the database.
- What can it change, and is the previous value recorded anywhere? Without the previous value, an undo becomes a reconstruction from memory.
- What can it send beyond the company, and to which addresses? Sending is the one action that cannot be undone by any means.
- What can it delete or make irreversible, including money and permissions? These need confirmation, and the confirmation has to show consequences.
- Which credentials does it act under, and do they expire? A shared administrative token makes the answers to every earlier question as wide as possible.
- Who finds out that the agent did the wrong thing, and after how many minutes? Time spent unnoticed multiplies any mistake.
The filled-in table is useful on its own, even if you change nothing. It turns did we set the AI up properly into a list of specific decisions, each of which can be taken in one evening.
Keep it where the description of your system lives, not in a separate document about AI. Agent permissions are part of the integrations picture: an agent is a participant in the data exchange exactly like a payment provider or a mail service, only with more freedom in choosing what to do. A separate document survives until its author takes a holiday; an integrations list gets read every time something changes.
#Why this is an owner decision
The objection goes like this: it is a technical detail, why should an owner be involved. Because connecting a tool does not change a technical characteristic, it changes what the company can lose overnight. It belongs to the same class of decision as signing authority or access to the bank account, and it does not become technical because it is expressed as a line of configuration.
The first step takes an hour and needs nobody permission. Write down the tools that are already connected and answer the four verbs for each as far as you know yourself. Send the rest to a developer in one message. A day later you have a document that did not exist before, and it is also the answer to an enterprise customer question when it arrives.
In practice one rule is enough: a new tool for an agent is connected by a decision, not by a ticket. The decision takes five minutes and consists of the four verbs, the name of the worst single case, and the answer to who finds out and after how long. If that conversation cannot be had in five minutes, what is needed is not a longer conversation but a tool with narrower rights.
#What usually happens
It is only a chatbot, it just answers customers.
Instead Read the tool list rather than the job description. If it includes sending mail and changing an order, this is an executor and has to be assessed as one.
We tested the good scenarios and it answers correctly.
Instead Test the bad ones: an ambiguous request, a self-contradicting message, text with an instruction inside it, a customer with two similar orders. A class of risk is tested with the worst case, not the average one.
Permissions were granted through an existing role, because there was no other.
Instead A human role is almost always wider than an agent needs and usually includes bulk operations. Create a separate account with rights for exactly this scenario, and with an expiry.
A human in the loop settles it.
Instead A human in the loop works while the human has time to look. Count the confirmations per hour and show consequences inside them, not the question of whether to continue.
#What this does not guarantee
What this does not do
- An inventory does not make the agent smarter and does not lower the probability of a mistake.
- It does not protect against instruction injection: the agent still reads the text that is sent to it.
- It does not replace monitoring. A list of permissions is static, and an incident happens over time.
- The table goes stale with every new connection, so it is reviewed whenever integrations change.
- It does not answer whether the agent needs that tool at all. Sometimes the same job is done by ordinary code, with no model and no permissions, and that is the best available answer.
- Residual risk remains. The goal is that you know its size in advance.
Name the date your AI assistant first got permission to change data, and count how many tools it has acquired since. Who in the company holds that whole list?