Recovery Preparedness: The Technology and Staging Behind Effective OT Incident Response
Building on last week’s discussion of incident preparedness documentation, the next critical piece is recovery preparedness. Documentation covers only the first half of what an organization needs to respond to a cybersecurity incident. The second half is staging the people, technology, and resources needed to recover, which can shrink mean time to recovery from weeks or months to hours or days.
In operational technology (OT), recovery speed has direct consequences. Every hour a production line, substation, or treatment process stays offline costs revenue and can affect safety, regulatory standing, and public trust. OT recovery also carries constraints that enterprise IT rarely faces. Industrial assets often run for at least fifteen to twenty years, depend on legacy operating systems and vendor-specific software, and in many cases can only be restored by engineers with deep knowledge of the process. Executing a recovery plan quickly depends on four staged capabilities: baselines, golden configurations, asset inventories, and verification testing.
Baselines
Every organization should know what its normal operating state looks like. Responders cannot confidently declare a system clean, or confirm that recovery is complete, without a documented reference point to compare against.
OT baselines should cover several dimensions. Network baselines record which assets communicate with each other, over which industrial protocols, at what intervals, and in what volumes. Because industrial networks are deterministic, deviations such as a new Modbus write source or an engineering workstation talking to a controller outside a maintenance window stand out clearly against a good baseline. A network baseline also becomes a validation tool at the end of an incident. Once systems are restored, traffic should return to the documented pattern, and any communication that persists outside that pattern may point to threat actor persistence or a compromised asset that was missed during eradication.
Host baselines record the expected state of Windows-based assets such as HMIs, historians, and engineering workstations: installed software, running services, local accounts, scheduled tasks, and open ports. Process baselines capture normal setpoints, alarm rates, and operating ranges so engineers can recognize manipulation of the physical process even when the network looks quiet.
Baselines have a shelf life. They should be refreshed after planned changes and stored in a protected location separate from the systems they describe, because a threat actor with access to the environment may also try to alter the reference data.
Golden Configurations
Industrial assets each have unique configurations that represent years of engineering work, tuning, and site-specific knowledge. A PLC’s ladder logic, a safety instrumented system’s configuration, a drive’s parameter set, or a managed switch’s VLAN layout cannot be rebuilt from vendor manuals after an incident. These configurations must be captured, verified, and stored as golden copies before they are needed.
A complete golden configuration library includes controller project files and logic, firmware versions, HMI and SCADA application projects, historian configurations, network device configurations for switches, routers, and firewalls, and full system images of Windows hosts. Responders must ensure the integrity of their backups and configurations before restoration. A backup that may have been tampered with introduces new risk into the recovery.
Golden configurations also extend to the tools required to use them. Restoring a controller project requires the correct version of the vendor’s engineering software, valid license keys or activation files, hardware dongles where applicable, and installation media. These dependencies are frequently overlooked and are a common reason restorations stall during real incidents.
Asset Inventories
Responders cannot recover what they do not know exists. An accurate asset inventory turns a chaotic recovery into an ordered one by defining what needs to be restored, in what sequence, and with what resources.
An OT asset inventory used for recovery should record, at minimum, the manufacturer, model, serial number, physical location, network address, firmware version, logic or configuration version, and the process function the asset supports. It should also record criticality and dependencies, so the team knows which assets must come back first for safe operations to resume. A historian may be valuable, but the controllers and safety systems governing the process take priority.
The inventory should link each asset to its golden configuration, its vendor support contact, and the availability of spare hardware. Staging spares for critical, long-lead-time components is a recovery decision as much as a procurement one. If a compromised controller must be replaced and the lead time is several months, the recovery timeline is set by the supply chain regardless of how good the backups are.
Passive discovery tools can help build and maintain the inventory without disrupting sensitive devices. A newer class of platforms adds carefully engineered active discovery designed to be safe for fragile industrial and embedded devices. These tools send lightweight, protocol-aware queries to fingerprint assets, and some can interrogate devices in their native protocols to retrieve firmware versions, serial numbers, and configuration details that passive monitoring may never observe. Certain platforms extend this into device management, identifying default or weak credentials, flagging outdated firmware, and supporting remediation across large populations of OT and internet of things (IoT) devices. Combining passive and safe active discovery produces a more complete and current inventory than either method alone. Both approaches should still be paired with physical walkdowns, because serial-connected devices, standalone systems, and equipment on isolated networks are often invisible to network-based discovery.
Verification Testing
A backup that has never been restored is an assumption. Verification testing turns stored configurations into a proven recovery capability and produces realistic recovery time estimates for the incident response plan.
Testing should operate at several levels. Integrity checks confirm that backup files are complete, readable, and match their verified originals. Restoration tests load configurations onto representative hardware, ideally in a test bench or lab that mirrors the production environment, and confirm that systems start and behave as expected. Full recovery exercises walk through the restoration of an entire process area in priority order, measuring the time taken at each step.
These exercises consistently expose gaps such as an expired software license, a firmware mismatch between the backup and the spare hardware, a missing password, or an undocumented dependency between systems. Finding these issues during a scheduled test costs time. Finding them during an active incident costs production and can extend an outage by days.
Results from verification testing should feed back into the incident response plan and tabletop exercises. Measured restoration times allow leadership to set achievable recovery time objectives and make informed decisions about where additional investment in spares, redundancy, or staff training will shorten recovery the most.
Bringing It Together
Recovery preparedness is the bridge between a written incident response plan and a facility that is running again. Baselines tell responders what normal looks like. Golden configurations provide trusted material to rebuild from. Asset inventories define what must be restored and in what order. Verification testing proves that all of it works. Organizations that stage these capabilities in advance give their responders the clearest path to safe, verified, and timely recovery when an incident occurs.
