The hacker of a Florida city’s water treatment plant who attempted to remotely contaminate the supply with a caustic chemical could have been a disgruntled employee or a nation-state, experts said, but outdated software and remote access controls underscored the need for security investments in critical infrastructure.
The Pinellas County Sheriff’s Office said it was notified on Feb. 5 of computer software intrusions at 8 a.m. and 1:30 p.m. at the City of Oldsmar’s water treatment plant. The system “allows for remote access by authorized users to troubleshoot any system problems from other locations,” the sheriff’s office said.
The first intrusion of the day “was brief and not cause for concern due to supervisors regularly accessing the system remotely to monitor the system,” the sheriff’s office said. At 1:30 p.m., a plant operator “witnessed a second remote access user opening various functions in the system that control the amount of sodium hydroxide in the water.”
“The operator noted the remote access user raised the levels of sodium hydroxide in the water. The operator immediately reduced the levels to their appropriate amount,” the sheriff’s office said. “The initial investigation revealed that the hacker remotely accessed the treatment plant’s computer for approximately 3 to 5 minutes.”
“At no time was there a significant effect on the water being treated, and more importantly the public was never in danger,” Sheriff Bob Gualtieri said.
A Massachusetts Department of Environmental Protection advisory to public water suppliers said access to the supervisory control and data acquisition (SCADA) system was accomplished via remote access software TeamViewer. “All computers used by water plant personnel were connected to the SCADA system and used the 32-bit version of the Windows 7 operating system,” the advisory said. “Further, all computers shared the same password for remote access and appeared to be connected directly to the Internet without any type of firewall protection installed.”
The sheriff said TeamViewer had not been used in about six months but had not been removed from the system. And Microsoft stopped offering support for Windows 7, a 2009 release, a year ago.
At a Wednesday House Homeland Security Committee hearing on cyber threats, Chris Krebs, former director of the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency, said in cases such as this “there is the potential for insider threat.”
“I think it’s possible that this was an insider or a disgruntled employee. It’s also possible that it was a foreign actor. This is why we do investigations,” Krebs said. “But we should not immediately jump to a conclusion that it is a sophisticated foreign adversary. The nature of the technology deployment in Florida is frankly not — certainly not — where anybody, I think, any information security or operational technology security professional would like for that security posture to be.”
“I will also say that Oldsmar is probably the rule rather than the exception and that is not their fault, that is absolutely not their fault: These are municipal utilities that do not have sufficient resources to have robust security programs,” Krebs continued. “That is just the way it goes. They don’t have the ability to collect revenue at a rate enough to secure their deployments… we need to have more security controls in place.”
Cyber Threat Alliance president Michael Daniel stressed at the hearing that “we very much need to keep an open mind until the investigation gets further down the road as to who the perpetrators behind this might be.”
“It could be a nation-state. Iran has shown itself very interested in water systems in other countries like Israel and even in the United States in former situations,” he said. “It could be a lone actor, it could be somebody — it could be a disgruntled employee. There’s just a wide array of possibilities at this point and we really need to keep an open mind until the investigation concludes.”
Asked how many times a day bad actors attempt to breach U.S. critical infrastructure networks, Krebs replied that “it’s actually really hard to make any sort of meaningful quantification.”
“There are both automated tools that run on a regular basis looking for vulnerable systems connected to the Internet” along with “human-powered initiatives,” he said. “I mean, we’re talking just massive numbers of scanning attempts on a regular basis. And that’s just the noise of the Internet. The more sophisticated, capable efforts are going to be fewer in number, going after the bigger fish to catch.”
Krebs recommended three key steps for critical infrastructure networks. “First is we need to have more federal funding available to get these tens of thousands of water facilities and other municipal operational technology systems up to speed with better security, more updated systems — Windows 7, if that is what they had, we should be on Windows 10. It is those sorts of things we have to do,” he said.
“The second is we need more training available and we have to bring the training to the systems where they are — so whether it is working with private sector or CISA working with the EPA we can’t expect these vendors to go to Idaho National Lab or travel. We have to bring the training to them,” Krebs continued. “And third …we have to have regional approaches to better IT technology. We have to have consortia that allow for upgrades and maintenance that are available with better price, with better cost efficiencies and economies of scale. You can pull that together at a state or regional level and I think that is going to have to be the future of IT, IT deployments for systems like this.”
Daniel warned that we need to be “very much hardening those systems and raising the level of cybersecurity across the ecosystem,” including “employing things like the NIST cybersecurity framework to do that risk management to those systems.”
“But then also going on the offense to find those adversaries and to disrupt them and to prevent them from doing what they are trying to do, and then also being able to know that sometimes both of those things will fail and know that we need to be ready to respond and recover,” Daniel said at the hearing, stressing that “we need to get better at responding rapidly, identifying the malicious activity, containing it and then removing it from those networks so that we can minimize the amount of damage that we take.”
That preparedness and response posture needs to be considered “from a national critical function perspective about what is important to our economy and to the functioning of this country as a whole,” he added. “And sometimes that will not be obvious from the outside and it requires thought and analysis to our arrive at some of those critical functions and where they are vulnerable.”
The Massachusetts alert to public water systems advised the utilities to restrict all remote connections to SCADA systems, install and turn on a firewall, keep software and devices up to date, use two-factor authentication with strong passwords, and consider using a VPN.
“Implement an update- and patch-management cycle,” the recommendations continued. “Patch all systems for critical vulnerabilities, prioritizing timely patching of Internet-connected systems for known vulnerabilities and software processing Internet data, such as Web browsers, browser plugins, and document readers.”