How to keep an autonomous system from doing something stupid

At 3:40 in the morning a supplier feed hiccups and reports every open order as delayed. The system does what it was told to do. It emails the customer. Nine thousand times. Forty minutes later the feed corrects itself and the system emails all nine thousand again to say the delay is over.

Nothing crashed. No rule was broken. Every one of those emails was, on its own, the right response to what the system believed it was seeing.

The danger is not the dramatic action

People picture the failure as one enormous wrong move. A refund of four hundred thousand dollars. An order for ten thousand units nobody wanted. Those are the easy ones. Loud, caught within the hour, a single action to reverse.

The failure that hurts is small, quiet and repeated. A discount two percent deeper than intended, applied to every order for six days. A follow up message sent to a group that should have been excluded, once per person per hour, across a long weekend. Each action is beneath notice. The damage is in the multiplication.

A human reviewer cannot catch that. Shown any one of those actions they would approve it, because it is correct. So “we will have someone review it” is the usual answer to the safety question, and the wrong one here. Review is good at judging one unusual case. It is useless against four thousand ordinary ones.

What it may do at all

The first mechanism is the narrowest possible definition of the actions available to it. Not what it should do. What it is able to do at all.

There is a real difference between a system that could refund any amount but is told to stay under two hundred dollars, and one that cannot refund more than two hundred because the ability was never granted. The first fails when the rule is misread or the data behind it is wrong. The second has nothing to misread.

So write the list down. Short and boring. Refund up to this amount on orders that shipped. Reorder these products, up to this quantity. Everything not on the list is absent, not merely discouraged.

Here is the test. Ask what the system could do if every rule inside it were wrong at once. If the honest answer is serious damage, the limits live in the instructions instead of the boundary.

How much it may do in a period

The second mechanism is a ceiling on volume. Per hour, per day, per customer. This is the one people skip, and the one that catches the failure above. A ceiling does not need to know why the system sent five hundred emails in ten minutes. Only that the number is not normal, and stop.

Set ceilings from ordinary behaviour, not from capacity. If your busiest real hour ever produced sixty of something, the ceiling is not ten thousand. It is closer to a hundred and twenty. When one is reached, stop and raise a flag rather than slow down and continue.

In xupsell, our own Shopify app, the limit that matters most is not what an offer may say. It is how many times one shopper may be shown anything at all. That ceiling turns a wrong rule into an annoyance instead of the thing that costs a merchant customers.

What still waits for a person

The third mechanism is a person in the loop for the actions that need one. The mistake is applying it everywhere or nowhere. Everywhere and you have rebuilt the queue you were trying to get rid of. Nowhere and there is no place left for judgement.

Draw the line with two questions. How expensive is this if it is wrong, and can it be undone. Cheap and reversible, let the system act. Expensive or irreversible, a person decides. Money leaving the business, anything sent to many people at once, anything a customer sees that cannot be pulled back, anything touching your largest accounts.

When it escalates, its job is to make that decision fast. Evidence gathered, rule named, recommendation stated. A person deciding in thirty seconds with the full picture is a control. A person facing two hundred items in a queue is not, because by Thursday they approve them all without reading.

The record, which matters more than the limits

The fourth mechanism is a permanent record of every decision, including why it acted. Insist on this above all the others. Not a log of what happened, but a record of what the system saw, which rule matched, what it did, and when, kept so it cannot be quietly changed afterwards.

Limits tell you the damage stopped. Only the record tells you what happened, and that difference decides whether the fix takes an afternoon or three weeks.

Without a record you know nine thousand emails went out and nothing else. You cannot tell whether the software did something it was never meant to do, a fault in the software, or exactly what it was told, a fault in the rule. One is fixed by changing the software. The other by writing one sentence differently. Guessing means changing both, which creates a second problem while chasing the first.

With the record it takes minutes. You watch the signal arrive, the rule match, the decision follow. The system acted correctly on bad information, so the fix belongs at the information.

The record earns its keep again months later, when a customer disputes a charge or somebody asks why this account got that price. Without it you are relying on memory. If a vendor cannot show you the record, the rest of the safety story is decoration.

A way to stop everything

The fifth mechanism is one control that halts autonomous action immediately, for one workflow or all of them, without waiting for a software update and without a phone call to whoever built it.

It should be usable by somebody who is not technical, at two in the morning, without asking permission. And someone in the business should have pulled it once, deliberately, before they ever need it. A stop switch nobody has used is a claim, not a control.

Questions worth asking

Take these to a vendor or an internal proposal.

What can this system do that it should never do? If the answer is that it is configured correctly so it cannot happen, the limits live in the instructions rather than the boundary.

What stops it when the incoming data is wrong but the rules are right?

How many times can it do this in an hour, and what happens at that number?

Show me the record for one action from last week, including the rule it matched and what it saw. This is the best question on the list.

Who can stop it, and when did somebody last do that on purpose?

Which decisions still wait for a person, and how was that line chosen?

The answers tell you more than a demonstration will. Someone who has run a system like this answers quickly and concretely.

If you are working through this for a system of your own, book a call.