Where AI Systems Break in Real Work

A chat demo can be excellent and still tell you very little about whether an AI system can carry a real company workflow.

The interesting problems begin when a request is not self-contained: it depends on history, must use a tool, changes something outside the chat, or needs another person to take responsibility.

The demo hides the state

In a demo, each prompt arrives with all the context it needs. In a real workflow, the system has to know what has already happened, which version of a document is current, whether a customer has replied, and which step is pending.

That is a state problem, not a writing problem. If state is vague, an agent may produce a plausible answer while repeating work, acting on stale information, or losing a handoff entirely.

Authority cannot be implicit

A useful system needs an explicit answer to a simple question: what may it do on its own?

Drafting a message is different from sending it. Preparing a payment request is different from authorising a payment. Suggesting a reroute is different from changing a customer's booking.

The boundary should be visible to the operator, encoded in the workflow, and easy to change. Human approval is not a failure of automation. It is how an organisation keeps authority legible.

Every action needs a receipt

If a system takes an external action, someone should be able to see what happened, when it happened, and through which provider. A response that says "sent" is not the same as a confirmed message, record update, or transaction.

Receipts make a workflow operable. They let people reconcile a system after a failure, answer questions from colleagues, and distinguish an attempted action from a completed one.

Exceptions are part of the design

The normal path is usually easy. The operating value appears in the exception: missing information, an ambiguous request, a conflicting instruction, a tool outage, or a decision that should not be automated.

A good first system names these cases in advance. It does not pretend they will disappear. It routes them to a person with enough context to decide quickly.

Start with a bounded loop

The right first project is rarely "transform the business with AI." It is one loop with a clear beginning, one useful action, one owner, and a defined escalation path.

Once that loop is trustworthy, a company has something more valuable than a demo: evidence about how AI can work inside its own operations.