When a failure reports success

Four cases in one week

A map on a directory page. Every tile loads, the server answers with status 200, the error log stays empty. Across every tile, in grey letters, it says "API KEY REQUIRED". The map provider had changed its terms and kept delivering, just with a watermark.

A form that sends an alert email to the team. The test email arrives. The sender address is correct in the text. Whoever clicks reply in their mailbox writes to the technical sending address, and nobody reads that.

A test run against the staging environment. All checks green. The database query log later shows: the test had been talking to the tables of the local environment. The environment variable was set, but the database client had been generated before and kept its configuration.

A build step that passes locally and fails in the pipeline. A new page wanted to load data from the database at build time. Locally the connection existed, in the build step it did not. This case is the most harmless, because at least it was loud.

Three of the four were silent. They reported success.

Two channels

All four have the same structure. There is a channel on which the system reports success. A status code, a return value, a green test report, a line of text in an email. And there is a channel on which the result actually arrives. The delivered image. The reply-to field in the email header. The table that was really read.

If both channels are the same, all is well. A test that compares the output value itself cannot report success and be wrong. As soon as the channels are separate, the report becomes a statement about the reporting channel. About the result it says nothing.

Monitoring almost always watches the reporting channel. Status codes, response times, error logs, test reports. That is not negligence, it is the nature of the thing: the reporting channel is machine readable, the result channel often is not. No monitor recognises a watermark in an image. No delivery report checks whether the reply-to field of an email points at the right address.

The check starts at the result

In all four cases the proof did not come from a log. It came from looking at the delivered page. From replying to the test email instead of reading its content. From the database query log that shows the tables actually read, instead of the test report. From a local build on the same path as the pipeline, with the cache emptied, instead of the development server.

That is more effort than a glance at a dashboard. But it is the only check that says anything about the result. Everything else is a check of the report.

For a business this means: there are places where once a quarter a person has to do what the customer does. Open the page, not the status report. Reply to the automatic email, not just check that it arrived. Play an order through to the end, not just read the order confirmation.

A fallback makes it worse

In the same week a decision came up. The alert email to the team needs a target address. What happens if it is missing because someone forgot the configuration?

The convenient answer: store an emergency address to send to when in doubt. Then everything keeps running. That is exactly the problem. The alert looks functional and reaches nobody who is responsible. It served the reporting channel and left the result channel.

The decision went the other way: without a target address the system throws an error. That is inconvenient, and it is the only feedback that cannot be overlooked.

A fallback that silently catches an error turns a loud failure into a silent one. Silent failures are the ones that last nine months. More on that next week.

Three questions that transfer

Which success report does nobody check against the result? Every monitoring setup has places where it trusts the report. The list of those places is short and worth making, because that is where the silent failures live.

Who regularly does what the customer does? Not the status report, but the page. Not the delivery report, but the reply to the email. One appointment per quarter, one person, one list.

Where does a fallback silently catch a failure? Every emergency address, every default value, every swallowed exception is such a place. There the system looks functional while the result is missing.