All tests green, nine months of selling too cheaply

The total that stood still

A configurator for made-to-measure products. The customer creates a part with dimensions, chooses a material, sees the price. If they create a second part, the total stands still. They can order like that, and the order runs through the shop as if everything were fine.

The operator noticed when he tried for himself what a customer had described. The bug had existed for nine months. It was triggered by a change that was right in itself: a hard-coded default material was removed so that the customer has to choose deliberately.

Three safeguards, all green

The system had what one hopes for. The pricing formula was covered by tests. The server recalculated every price before handing over to the shop, instead of taking it from the browser. A schema checked every configuration for validity.

The cause sat between two steps. The material is written once, in the material step, to all parts that exist at that moment. A part created later got nothing. The pricing function valued a part without material at zero per square metre, calculated that correctly, and the total stood still.

Each of the three safeguards answered the question it was built for, and answered it correctly.

The tests checked the calculation with parts that already had a material in the test. They proved the formula for the inputs the test builds. Whether the application ever produces those inputs, such a test does not say. The bug sat between two modules that were each green on their own.

The recalculation protected against manipulated prices. It answered whether the number matches the input. It did not answer whether the input is one from which a valid price can arise at all.

The schema allowed parts without material. It reported such configurations explicitly as valid, because from the schema's point of view the material was optional.

Nothing in the system asked: can this price be right?

The second case is a sign

Half a year before this report the same thing had happened once before. That time with a fixed-price product that needs no material. It was fixed by giving that one product a fixed price. The symptom was gone. The mechanism stayed.

A case that occurs a second time almost always says the same thing: the first time, the occurrence was repaired and not the rule. The rule here was: a value distributed once to all elements reaches only the elements that exist at that moment. Every new element has to receive it again.

What is different now

A newly created part takes over material and settings from the first. That closes the known path. It does not close the next one, which nobody knows yet.

That is why at the last point before the money there is now a check that does not depend on how the state came about. The server rejects every order containing a part without material, without dimensions, or with a total of zero. Not with a correction, but with an error message and an entry in the error monitoring.

Beneath that lies a second net with different logic. If the price of a part falls below the pure material sum, two emails go out and a warning goes into the monitoring. One email to the shop's order address, one internal. This net never blocks. It catches what the check above it does not know.

A test now runs all 27 material combinations the schema allows against the price tables. And all stored orders were recalculated. Three paid and delivered orders had gone through at a fraction of the price.

Four decisions, and against what

The second net is an alert and not a second block. A second block would have the same logic and the same blind spots as the first. An alert with different logic also catches what the block does not know.

For a part without its own material rate, the net uses the rate of the first part, not zero. Otherwise exactly the reported case produces no shortfall, and the net stays silent in the very moment it was built for.

The minimum price per square metre was not raised as a safety net. It would then be a second price table that can drift apart from the first.

The alert goes by email to the people who hold the order before production, not only into the error monitoring. An alert is as good as the person who reads it.

Three questions that transfer

Which number in the system means money? Price, discount, quantity, weight for shipping. For each of these numbers there is a last point before it takes effect. That is where a check belongs on whether the number is possible at all.

What do the existing safeguards really check? Each answers one question. The gap lies between the questions. A test that sets up the finished state checks the calculation and not the path to it.

Where does the alert land? A warning in a tool nobody opens daily is not an alert. It has to arrive where action is taken, before the machine runs.