With the benefit of time and commendable candor from investigating authorities, the national security and critical infrastructure operator communities are learning more about how an intruder tried to tamper with chemical levels in the water supply in Pinellas County, Florida. Even in our data breach and cyber attack-saturated headlines, this incident gave everyone pause – particularly similar municipal water and other utilities.
A simple timeline lays out the disturbing chain of events: The Pinellas County Sheriff reported that on Feb. 5 in Oldsmar, Fla., an alert plant employee noticed unusual activity – specifically observing a remote actor spending several minutes trying to adjust the amount of sodium hydroxide, or lye, in the water from 100 parts per million to 11,100. If successful, this action could have led to the water becoming toxic. An advisory from the Water Information Sharing and Analysis Center (WaterISAC) suggests how a copy of TeamViewer remote access software on a targeted Windows 7 computer at the plant played into the attack scenario. Fortunately, the anomalous activity was observed and action was taken to preserve water quality.
Reactions and questions are swift following the attack: “Is remote access a good idea for computers controlling chemicals in our water?” “Should critical control systems run on Windows 7, an operating system Microsoft stopped supporting in 2020?” “How would we spot similar attacks like this at scale, across the country?” As tempting as it may be to blame a single technology or vulnerability for the incident, those of us in the industrial security community know cyber risk management is a complex issue. We can still keep a sense of urgency on taking Pinellas County’s case to heart and fortifying our operator community while adding important nuances and context. Here are a few practical insights to consider.
Remote-access is a necessary risk but must be managed in “remote-everything” times
Security pros defending companies’ databases, bank accounts and other systems rightfully view remote access tools with wariness. After all, for office-led businesses keeping employees productive remotely with mobile, cloud and collaboration tools during the pandemic, there is little legitimate reason for anyone to remotely access a particular desktop PC back at headquarters.
But the utility world is different. Unlike sales, finance and other departments already living in SaaS applications, who simply fail-over to those same interfaces from home, critical infrastructure staff cannot simply tuck their control consoles in briefcases. When a physical safety obstacle like a pandemic reduces available healthy staff or forces a smaller crew of operators to work in shifts, remote access preserves integrity and availability by default, by literally keeping the lights on.
However, risk management is also about adapting to the day-to-day, and greater reliance on remote access tools puts a higher premium on hygiene and oversight of these previously physically secured systems. There are many forms of remote access – whether individual administration tools like RDP or TeamViewer or outsourcing management of a facility to a remote contractor. Make sure you have all remote access points accounted for – across operations staff, vendors, SOCs, NOCs, monitoring and maintenance. This situational awareness helps prevent blind spots or dangerous unchecked practices, like sharing passwords among employees – or neglecting to activate built-in encryption and other security features.
Add smart friction to counterbalance automation and other technology risks
While it sounds counterintuitive, sometimes effective risk management requires introducing new limits to check the sheer speed and precision of new technologies, like robotics and automation. Adding “smart friction” makes sense in industrial settings, where a malicious event – or an error in a data field or processing – could have catastrophic consequences when machines respond faster than human overseers can react. Rate-limiting how much the parts per million (PPM) ratio of a chemical can be changed within a few minutes’ time – or whether a certain high percentage change can be made at all – is a great example of smart friction serving as a safeguard that allows other alarms and monitors to work and is unlikely to hinder operations otherwise.
Know your “blast radius” and double-down on segmentation
Securing more connected utilities demands ongoing review of segmentation, or to what degree all connected assets are classified and separated according to owner, role and criticality. This reduces the risk of an intruder or outage hitting one segment and causing spiraling problems laterally across the organization.
Like a battleship with watertight compartments, strongly segmented networks can survive a “hit” or two by quickly sealing off the damage. Unfortunately, the Internet of Things (IoT) era makes many office networks seem more like open-concept floor plans, where myriad tenant, building and employee devices discover-and-connect by default. This yields flatter, more disruption-susceptible networks where sensitive business assets become comingled with things like guest Wi-Fi or shared conference room equipment. Compromise one device, and that blast puts you within reach of everything else.
Utilities set the standard for segmentation principles well before the IoT, owing to the risk of interference and unreliability when a connected device meets formerly isolated factory, field or facility sub-systems. Yet even these operators are increasingly challenged to maintain their OT/IT segmented security high ground in the face of connected everything. For example, my team regularly works with utilities implementing equipment modernization who realize hardware like turbines or other machinery now has to connect to the Internet in order for non-negotiable contract terms like manufacturers’ SLAs, preventative maintenance tracking and troubleshooting to be realized.
As more wireless connections and those quiet (but multiplying) third-party and supply chain pieces shift at water, electric power, gas, oil and other operators, what was once a straightforward segmented network can look very different over time. Our community has to deliver services first, squaring risk management as we go. In practice, this means fighting outdated asset inventories and doubling-down on segmentation’s advantages by continually looking for those new built-in antennas, ports and services that draw dotted lines over last year’s tidy network map.
As alarmed as we can feel about cyber risks to critical infrastructure, it is important to not lose perspective. Our infrastructure is more reliable than many outside our community may suspect, and it is staffed by dedicated personnel keeping everything running during a pandemic, on top of routine mechanical and engineering challenges – whether that means responding in a blizzard, pulling long shifts or taking a second look at an odd command or gauge reading. This is the culture we need to recognize, resource and build on to drive resiliency in the face of cyber risks introduced by the innovations we depend on to improve utility delivery.
