It is the early hours. A static transfer switch has thrown the load to its alternate source, and a row has dropped in the half second it took. By the time anyone is looking, both sources read clean, the switch reports healthy, and its own event log says it did exactly what it was designed to do: it saw a source go out of tolerance, and it transferred. Facilities confirm the power is fine. The switch vendor confirms the unit behaved to specification. And no one can independently say the thing that actually matters, which is whether the source was genuinely failing, or the switch’s own sensing told it so when it was not.
Within the hour the load is back and the room is stable, but the confidence is not. Days later the review records a probable cause, because the evidence to do better than probable was never captured. Anyone who runs a hall has lived some version of this. The downtime is rarely the worst part; it is brief and contained. The worst part is that the same event could happen again next quarter and the team would be no better placed to explain it.
Both sources read clean. The switch reports healthy. The load still dropped.
The fault sits on a seam no one owns
What makes it so hard to close out is not the technology. It is where the fault sits. The switch, like most of what trips a data hall, lives on a boundary that several teams share and none of them owns end to end. Facilities own the plant. The network team own connectivity. Security own segmentation and hardening, and as the controls network converges with everything else, a change they make can now reach a device that used to sit on its own. The integrator who commissioned the switch owns its logic, and is rarely in the room at three in the morning. The vendor owns the box. Each can see its own side of the line and no further.
So when a fault crosses that line, and it usually does, the investigation becomes a series of boundary disputes. Facilities show the equipment is healthy. The network team show connectivity is clean. Security show the policy is correct. The vendor shows the component is in spec. Every one of them is telling the truth about the part they can see, and not one can answer the only question that matters, which is what the device was actually doing when it acted. The fault lives in the handoff, and the handoff has no owner. The person accountable for the floor is left holding a trip that belongs to everyone, and so to no one.
Every team can prove its own side is fine. None can say what the device did.
Built by hand, a little differently each time
The other half of the problem is how the estate came to be: a hall assembled over a decade from many vendors and tenants, each generation commissioned a little differently from the last. A controller is the unit of trust. It summarises what sits beneath it and reports a clean status upward, and every layer takes that status on faith, because there is no other input. When the status is wrong, when a sensing threshold has drifted, a setting was changed during a past commissioning and never written down, or a firmware revision quietly shifted a transfer threshold so one switch no longer behaves like its supposedly identical neighbour, nothing above can tell. The only thing it can see is the report, and the report says healthy.
This is why these incidents are rarely hardware stories. Trace one back and what you find, more often than not, is configuration, process or human error somewhere in the chain: a change whose blast radius no one could fully see, two machines that were commissioned to be identical and never quite were. A device quietly behaving differently from what everyone assumed is not the edge case. It is the common case, and it stays hidden precisely because everything upstream is trusting the device to describe itself.
What it would take to explain it
The missing thing is narrow. Not more dashboards, which report what the device reports. Not more monitoring, which the building already has and which still said healthy. What is missing is independent evidence of what the device was actually doing, taken from the device itself rather than the layer that summarises it, at the moment the question is live.
That is a diagnostic problem, not a monitoring one, and it is the one Iris is built for. Brought to a specific device, under the operator’s control, it runs only the checks they have authorised and never acts on its own. It establishes how that equipment is genuinely behaving, regardless of what its status page claims. The switch that transferred can be read directly, its actual behaviour and its configuration established independently of the tidy log it reports, so the transfer can be reconstructed and checked against what the sources were doing, rather than read off a status that only says it completed. Transfer and protection logic can be proven to behave as documented, off the live floor or at commissioning, before it is trusted with the real load. And a fleet can be brought up the same, verified way, so the drift that makes identical units diverge is caught before it ships rather than after it bites. The same evidence speaks to everyone at once: facilities learn what the device did, the network and security teams learn whether a change of theirs reached it, and the vendor learns precisely what it must account for.
Find the failure, and prove it.
It is worth being plain about the limits, because that is the point. When the utility supply fails upstream of the building, Iris can do nothing. It reads the controller, not the physics behind it, so it does not measure a battery’s true wear or an engine’s condition; what it can do is catch the controller misreporting them. What it owns is the layer where most of these incidents begin: the controllers, the sensors, the configuration and the behaviour, the part no single team sees whole.
Closing
The trip will happen again. Today it closes as a probable cause and the team moves on, not for want of curiosity but because the evidence to do better was never there to capture. A protective system is only as sound as what the device tells it, and what the device tells it is exactly the thing no one in the room can independently check.
Axions is built to change that: independent evidence from the device itself, brought to the equipment when the question is live, so the floor can prove what tripped it.
A trip you cannot explain is a trip you will see again.