Guarding the keys: hardening the management plane of a US regional water and power utility

The situation

A regional utility in the United States supplied water and electricity to several counties. Its operational technology (OT) team had done solid work separating control systems from the corporate network. The IT side was different. Firewall and VPN consoles, an SD-WAN controller, the hypervisor manager, endpoint management, the IT service platform and the GIS server were all administered from ordinary laptops on the corporate network, several with web consoles reachable from the internet "for vendor support". Many admins had standing domain-wide rights.

Why now. In late October 2026, Lumen's Black Lotus Labs described Volt Typhoon's Silent Train framework: short-lived relay networks used in several zero-day campaigns against centralised management platforms, with victims including energy, water and transport organisations¹². The same week, Help Net Security reported a China-nexus group using a VMware ESX logging flaw as a zero-day to hide its activity⁴. CISA and partners had already warned in 2024 that Volt Typhoon seeks persistent access to US critical infrastructure, living off the land and abusing valid accounts³. The utility's board asked a simple question: if someone took over our management tools, would we know?

What had to be true

• No management interface on the internet, and every one inventoried with an owner.
• Administration only from dedicated, hardened workstations on an isolated network.
• No standing admin rights: access granted for a task, approved and time-limited.
• Logs that survive a compromise: shipped off the device, to storage the attacker cannot edit.
• A practised response for "a management tool is compromised", including OT impact.

Architecture

The data flow, end to end

The hard part: telemetry an attacker cannot erase

Attackers who reach a management tool tend to clean up after themselves, and the ESX case shows they will use logging weaknesses to hide⁴. Logs that stay on the device prove little. The team made off-device telemetry the rule: firewalls, VPN concentrators, hypervisors and consoles send logs to OpenTelemetry Collectors at each site, which forward them over mutually authenticated TLS to a central collector behind a gateway that terminates mTLS⁵. From there, data lands in write-once storage with its own separate admin path.

Two details mattered. First, the logging pipeline had its own credentials and admins, so taking over the firewall console did not also give control of the logs about it. Second, detections focused on behaviour CISA describes for this actor, such as valid accounts used at odd times, new admin accounts and configuration exports, rather than malware signatures³.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: start with the inventory. Several consoles nobody owned turned up, including two still facing the internet. Differently: involve the main vendors in week one. Their support processes, not our technology, were the slowest part to change. For the patching side, see defender-first AI models and vulnerability management, this week's tech news brief, securing a legacy PHP estate and OpenTelemetry in practice.

If you run critical infrastructure in the United States or elsewhere, see the US page, or analyse your challenge and bring the result to a call.