The automation-first mistake
Enterprise AI conversations often jump directly to agents that “run the business”. The model should approve invoices, send messages, update the CRM, move inventory, create journal entries and coordinate employees. That sounds ambitious. It also combines the least deterministic part of the system with the most consequential permissions.
Most organisations have a more immediate problem: people cannot easily understand the state already trapped inside their software. A manager wants to know why gross margin fell in one branch. An accountant wants the transactions behind a receivable balance. An inventory lead wants to know which products are repeatedly adjusted and by whom. A support operator wants a concise timeline of everything that happened to one order.
Those are explanation problems. They can create value before the AI is allowed to mutate a single record. More importantly, they force the organisation to solve the foundations that autonomous action would otherwise hide: data ownership, permission scope, provenance, consistent business definitions and reliable retrieval.
Explanation is a serious product capability
An explanation layer should do more than turn SQL results into friendly prose. It needs to understand the concepts of the business. “Revenue” may exclude cancelled sales. “Available stock” may mean on-hand minus reservations. “Outstanding customer balance” may include posted invoices but not drafts. If the AI uses a different definition from the finance system, fluent language simply makes the wrong answer easier to believe.
The strongest design is to reuse trusted domain services, metrics and reporting queries where they already exist. The model can choose among bounded tools such as “get branch sales summary”, “get invoice timeline” or “get stock movement evidence” rather than inventing unrestricted queries against the production database. The tool returns structured facts; the model explains them.
This architecture makes the AI useful even when it is conservative. A user can ask a natural-language question, receive a concise answer, inspect the supporting records and follow links back into the system of record. The model becomes a navigation and synthesis layer over governed business data.
The AI should inherit the user’s permissions, not escape them
An assistant embedded in enterprise software should never become a shortcut around authorization. If a salesperson cannot open payroll, asking the AI “what does everyone earn?” should not work because the retrieval tool runs with a privileged service account. If a branch manager can only see branch A, the assistant should not summarize branch B merely because both branches live in the same database.
Permission scoping should happen before sensitive context reaches the model. The backend resolves the actor and applies the same resource, tenant, branch and field-level policies used by the rest of the application. The AI receives only the records the user is entitled to inspect. That reduces both accidental leakage and the blast radius of prompt injection.
The model should also know when access was denied. It can explain that the requested information is outside the user’s scope without revealing the information itself. This is more reliable than handing a model all data and asking it to remember a policy written in the system prompt.
Evidence should travel with the answer
Enterprise users should be able to distinguish an AI interpretation from a source record. An answer such as “supplier costs rose sharply last month” is more useful when the interface shows the purchase orders, invoices or report rows that support the claim. If the assistant cannot produce evidence, it should reduce confidence or say the available data is insufficient.
This is partly a retrieval problem and partly a product-design problem. Every tool response should carry stable identifiers, timestamps and source types that the interface can render as citations. For generated summaries, the model can reference those identifiers instead of fabricating URLs. The user can open the underlying invoice, transaction or audit event and verify the interpretation.
NIST’s AI Risk Management Framework emphasizes governance, mapping, measurement and management of AI risk across the lifecycle. In a business assistant, provenance and documentation are practical ways to support those goals. They make evaluation possible because reviewers can compare generated statements against the exact data the system used.
Draft actions are a better bridge to automation than silent autonomy
Once explanation works, the assistant can begin proposing actions without immediately executing them. It can draft a supplier email, prepare a stock-transfer request, assemble a reconciliation note, prefill a journal description or suggest which invoices need follow-up. The user reviews the proposal inside the normal workflow and submits it through existing permissions and approvals.
This preserves human accountability while still removing repetitive work. It also generates valuable evaluation data. Which suggestions are accepted? Which fields are routinely changed before submission? Which recommendations are rejected? Those signals can improve prompts, tools and policies before the organisation grants more autonomy.
For sensitive actions, the approval model should remain explicit even if AI creates the draft. A model-generated payment instruction should not bypass maker-checker because the model is “trusted”. The AI is another actor in the workflow, and its outputs should face at least the same controls as human-entered data.
Tool boundaries matter more than agent branding
Modern AI systems gain power through tools: search, databases, APIs, email, files and workflow actions. OWASP describes excessive agency as risk created by excessive functionality, permissions or autonomy. The practical response is to expose the smallest tool surface needed for the job.
Instead of a generic “execute SQL” function, provide specific read tools with bounded inputs. Instead of “call any internal API”, expose named operations. For write tools, require structured parameters, validate them server-side and route them through the same authorization and business-rule layer used by the ordinary application. High-impact tools can require an approval token that the model itself cannot mint.
Tool responses should also be treated as untrusted input where appropriate. A retrieved document can contain instructions intended to manipulate the model. The system should separate data from control, constrain what downstream tools can do and avoid turning arbitrary retrieved text into permission to act.
Evaluate the business workflow, not just the model
A model can score well on generic benchmarks and still fail inside a company because it uses the wrong data, ignores branch scope or invents explanations for missing records. Enterprise evaluation should therefore use real workflow cases with expected evidence and allowed actions.
For an explanation assistant, test questions such as “why is this invoice overdue?”, “which stock adjustments explain the variance?” and “what changed after the customer complained?”. Review factual correctness, completeness, permission compliance, evidence quality and whether the answer clearly separates observation from inference.
For draft actions, evaluate parameter correctness and policy compliance before language quality. A beautifully written refund note attached to the wrong transaction is a failure. A proposed transfer that exceeds the user’s authority should never reach execution. These tests should run continuously as models, prompts and tools change.
A safer enterprise AI architecture
A practical architecture begins with the existing system of record. Domain services expose trusted, permission-aware read tools. A retrieval layer adds governed documents and operational context. The model receives only the minimum context needed for the question. Its answer carries evidence references. If the user requests an action, the assistant creates a structured draft. The backend validates that draft against current permissions and business rules before the ordinary approval workflow takes over.
Logging should record which model and prompt version were used, which tools were called, which evidence IDs were supplied and which action, if any, was proposed. Sensitive prompts and outputs need retention rules appropriate to the organisation. Observability should make it possible to reconstruct why the assistant produced a consequential recommendation.
This approach may sound less exciting than “the AI runs everything”. It is also much closer to software an organisation can trust. If AI can reliably explain the business, cite the evidence and respect the user’s authority, then the company has built the foundations required for deeper automation. Autonomy should be the result of proven controls, not the starting assumption.
Sources and further reading
