Suppose an AI assistant is asked to find a supplier, compare quotations and arrange the purchase of equipment. Finding a supplier is research. Drafting the order is preparation. Actually placing it commits someone else's money. The interface may present those as one smooth conversation, but the backend cannot afford to treat them as the same kind of action.
This is why I find the phrase 'the agent has permission' too vague to be reassuring. Permission from whom, to do what, against which resource and for how long? A model can produce a plausible plan without possessing the authority to carry it out. Once software begins to act in the world, the distinction between intelligence and authorization stops being academic.
Follow the purchase request
- 01The request
A staff member asks an assistant to source replacement equipment. The request establishes a goal, not unlimited purchasing power.
- 02The proposal
The agent finds options and prepares a purchase with supplier, items, amount and delivery details. The proposed payload is still a draft.
- 03The approval
An authorized person approves one specific proposal. A changed recipient, amount or scope requires a new decision.
- 04The execution
The target service checks current permissions and the approved payload, commits at most one authorized operation, and stores evidence of the outcome.
Think about what happens between the request and the final purchase. The assistant may look at a supplier catalogue, a past order and a message from an employee. Each item supplies information, but none necessarily supplies permission. A catalogue can describe price; an employee message can express a preference; a purchase policy controls who is allowed to commit funds. Keeping those categories separate sounds obvious until a system combines them all into one large prompt and asks the model to decide which text is authoritative.
The steps look unremarkable when written down. That is their strength. A financial application already needs separation of duties, transaction identity and limits. Connecting a language model should not make those requirements disappear. The model can help choose an action; the service enforcing that action must remain the final authority.
Delegation is not a password handed to a chatbot
A second trap is confusing the model's record of a plan with the application's record of a decision. The assistant may summarize that 'the user approved the order', but the purchasing service needs its own verifiable approval event, attached to the relevant principal and payload. If the model is restarted, its memory is truncated or the conversation is replayed, that event should not change. This is one reason consequential actions deserve explicit APIs and durable state machines instead of parsing chat transcripts as if they were legal authorization documents.
One tempting implementation gives the assistant a long-lived token belonging to the human user. It is convenient because everything works until something goes wrong. The system may struggle to distinguish the user's direct actions from the agent's automated ones. If the user changes roles, the token is cached or a background job wakes up days later, an old request can outlive its legitimacy.
A safer design gives the automated process its own identity and a narrowly described delegation. Read a particular calendar, draft an invoice, update one approved object, or execute a transaction subject to a limit. The identity of the human principal and that of the executing agent are both recorded. OAuth token exchange provides established concepts for delegated access, but it still takes application-level policy to enforce resource and business constraints. 2
The threat does not always arrive as an obvious instruction to ignore safeguards. A document might say that a particular recipient account is now the company's preferred settlement destination. An agent summarizing the document could mistake the claim for an approved business fact. If changing payment details requires an independent verification procedure, the purchasing backend must continue to require it. The model should be able to surface the discrepancy and ask for review, but not silently rewrite the organization's established authority chain because the untrusted text appeared in a convincing format.
Even a legitimately delegated instruction has a shelf life. A person who could approve an expense on Monday may have lost that role by Thursday. A product may have moved to another tenant. A payment quote may have changed. A queued task, retry or webhook callback must re-evaluate the boundary that matters at execution time. Revocation and changed circumstances have to be operational realities, not promises that the assistant will remember. 4
Approval must attach to the thing being approved
Human approval is often drawn as a simple Approve button. That button is meaningful only if the system knows which immutable action it authorizes. A transfer approved for one recipient is not authority to transfer to another. A purchase approved up to one total is not authority to increase the amount after taxes or fees change. Approval should be bound to the exact relevant fields and expire when those fields change.
An approval interface also has to make the consequences comprehensible to the person clicking it. If the human is asked to approve a vague sentence such as 'complete the order,' the software has failed even when the payload is technically immutable behind the scenes. Show the meaningful details: destination, amount, fees, items and the policy exception if one exists. Once those details are agreed, the operation can be signed or recorded against a stable representation. If one changes materially, the next action is review, not hopeful interpretation of the old approval.
Weak approval
A boolean flag saying 'user clicked approve' with no binding to recipient, amount, item or resource state.
Bound approval
A recorded decision over a specific action payload, delegated scope, approving principal and validity window. The execution service rejects a mismatch.
Idempotency belongs here as well. If the executor sends a purchase request, times out and retries, the retry should not create a second purchase. The operation needs a stable identity whose meaning cannot silently change between attempts. Authorization, transaction semantics and auditable consequences are ordinary systems engineering; agent enthusiasm does not suspend them.
Operational testing should include what happens when an otherwise sensible plan is interrupted. An agent may select a supplier, wait for a review and then resume after the price has changed. It may lose network connectivity just as the destination records success. It may receive the same callback twice, or continue running after access has been revoked. Each case is easier to reason about when proposal, authorization, execution and reconciliation are separate states. Without those boundaries, teams end up asking the model to repair a transactional ambiguity that the underlying systems created.
Audit records also need to say more than 'agent updated record'. Which human request started the work? Which grant applied? What exact operation was approved? What did the target service accept? Where did a rejection occur? The objective is enough evidence to investigate a disputed action without copying sensitive prompts or credentials into every log. NIST's risk and control frameworks offer useful disciplines for this kind of operational accountability. 15
A full audit trail is not a demand to preserve an agent's every intermediate token. That would create privacy, storage and discoverability problems while still not guaranteeing an understandable explanation. The evidence worth keeping is closer to an accounting record: the request identity, authorized actor, approved scope, state transition, relevant policy decision and resulting operation identifier. It should make it possible to distinguish an unauthorized instruction from an authorized attempt that failed because the downstream service was temporarily unavailable.
Autonomy should be earned in small steps
Not every action deserves the same ceremony. Drafting a reminder is unlike transferring money. An assistant can begin with read-only research, then create drafts, then perform low-risk writes under bounded permissions and monitoring. Expand only when the organization has evidence about errors, reversals, policy violations and how quickly staff can intervene. High-impact operations may continue to require explicit authorization regardless of performance.
A system that can explain why it chose an action is useful. A system that can prove it was allowed to take that action is safer.
There is a governance question as well: who can stop the agent when its tools behave as designed but the overall pattern is harmful? Teams need limits on frequency, spend and resource scope, plus monitoring that detects unusual behavior across multiple individually permitted actions. Rate limits alone are not enough if an agent can divide a large consequence into smaller requests. The service should enforce aggregate policies, preserve a route for human interruption and keep the operator's responsibilities clear. Capability can increase over time, but accountability must not be delegated to a paragraph of prompt text.
There is a useful middle ground between forbidding automation and letting an agent act without limits. An organization might allow the assistant to reconcile a report, prepare customer responses or flag anomalous transactions while preventing it from posting irreversible financial changes. The boundaries can be revised as the service demonstrates reliability under adversarial input and ordinary operational failure. What matters is that the rule belongs to a responsible system owner who can inspect and change it, not to an optimistic assumption about how the model will behave next time.
The central question is not whether an AI agent can press a button. It is whether the software around that button can identify the actor, verify the authority, preserve the transaction and explain the result after the conversation is over.
Sources and further reading
