February’s cyber incident involving remote access and a Florida water utility with intent to cause harm is a serious reminder that cyberattacks on our country’s critical water infrastructure are real, on the rise, and can impact public safety. It also reminds us how unprepared some water utilities, particularly those in smaller communities, are to prevent even the most unsophisticated attacks.
3 Reasons why smaller (and larger) utilities struggle with cybersecurity
Smaller utilities generally do not have the staff or budget to adequately address cybersecurity, particularly cybersecurity of the control and SCADA systems (also known as OT systems) that operate water or wastewater treatment facilities. We’ve identified 3 common reasons that smaller water utilities struggle with cybersecurity.
Water treatment plant management may not be aware of how vulnerable their systems are. For example, it is not uncommon for management to believe that their OT systems are isolated (a.k.a. air-gapped) and thus are immune to cyber-attacks. Sometimes there is the belief that a system or facility is small to be a target.
Another common situation is that plant management is aware that their OT systems are connected but is overly confident that the municipality’s IT department or IT contractors have cybersecurity covered.
Perhaps the most common situation is that plant management is aware that their OT systems are vulnerable but, because of tight budgets, are intimidated by the cost and effort required to improve cybersecurity.
Regardless of the scenario, it is highly likely that their OT systems are vulnerable and that cyber risks are not being managed.
Unclogging the pipes
So how can smaller utilities address these issues given the shortage of budget and staff?
First, we always recommend starting with an assessment to understand the vulnerabilities that present the highest operational risk to the organization and then follow that by preparing a mitigation plan that is prioritized by risk.
In fact, the America’s Water Infrastructure Act (AWIA) of 2018 mandates that same strategy. Specifically, AWIA requires water systems that serve greater than 3,300 people to conduct a Risk and Resilience Assessment (RRA) and create an Emergency Response Plan (ERP) that includes cybersecurity risks and responses. Although not required for smaller systems, it is still recommended that assessments and response plans be developed and implemented. The recommendations extend to wastewater treatment systems and facilities too.
While this may sound expensive, it doesn’t have to be. While it’s a good idea to engage a consultant with expertise in OT cybersecurity and the water sector to help with the assessment and mitigation plan, you don’t necessarily need to engage a large, national, or global cybersecurity firm. In many cases, specialty OT cybersecurity consultants or automation system integrators can offer better service at more cost-effective prices.
Additionally, there are free tools and guidance available that help a utility to conduct an RRA, such as the AWWA Water Sector Cybersecurity Tool. These tools can definitely help gather and organize the information to conduct an assessment. However, these tools have some drawbacks, such as not linking vulnerabilities to operational risks. In most cases, a deeper dive into the system by an expert is required to determine the true consequences and risks of a security breach. For example, not all vulnerabilities present high-risk to operations. The risk depends on the exposure of the vulnerability and the consequence of compromise. Conversely, what may appear to be a minor vulnerability may present high-risk to operations because the consequence of compromise is very high.
Best practices for conducting Risk and Resilience Assessments (RRA) include software tools and questionnaires to gather data, followed by expert analysis of the data, and a HAZOP-style cyber risk assessment workshop to relate vulnerabilities to operational risk. There are key topics and contexts that must be discussed in these workshops, like remote access, to determine if there is a significant risk that can result in an incident like what happened at Oldsmar.
How an average Joe made his water treatment plant more secure, but Fred failed
Let’s look at fictional average Joe. Joe is the plant manager for a water treatment facility. He has been uncomfortable for years about the cybersecurity of his plant but wasn’t sure what to do. Joe is proactive, so he attended an AWWA webinar about the AWIA cybersecurity requirements. His facility, due to its size, is not required to comply. However, he thinks it’s the right thing to do. So, Joe downloads some free assessment tools and starts to complete them himself but quickly gets frustrated. The tools show areas he should be concerned with, but he finds it difficult to make sense of what he needs to do about them. Joe has limited funds available. Where should he focus his spending? Does he need outside help? Joe seeks and finds a good OT cybersecurity consultant to work with. The consultant helps Joe leverage the work he has already done, performs some additional data gathering, prepares a summary vulnerability and gap assessment, and assists in prioritizing the most cost effective actions that can be taken to reduce risk and maximize his available resources.
A couple of counties over, Joe’s buddy Fred manages a water district with a couple of mid-sized treatment plants. Fred gets a call one evening with bad news. Both plants have been infected with ransomware. The control systems are shut down, and the plants are limping along on manual control. Fred is wondering how this could happen. How did this get past the safeguards that were set in place by the IT team? The district has a great IT team, so he is confused but not worried. A few calls should get things back up and running. During the first call, Fred quickly realizes that the IT team doesn’t know much about his control systems. They know a lot about the network and the business computers, the IT side, but not a lot about restoring an HMI and associated systems, the OT side. Fred still has no worries as the system does automatic backups. No worries, that is until he finds out his backups are stored on a machine that is also affected by the ransomware. Suddenly the nice recovery plan that was put in place doesn’t seem so nice anymore.
Keep the tap flowing
While it may seem overwhelming, especially if you don’t have a background in OT systems, networks, and cybersecurity, the challenge of addressing cybersecurity of your water treatment facilities is not insurmountable and does not have to be overly expensive. With the right tools and assistance, you can identify your highest risks as well as develop and implement a practical plan to address them. Learn where to allocate resources to maximize funding and personnel.
Take a look at the aeCyberSolutions’ webinar, Cybersecurity Challenges in the Water and Wastewater Industry, at Cybersecurity Challenges in the Water and Wastewater Industry (aecyber.podia.com)
Contact aeCyberSolutions at: firstname.lastname@example.org
About the Authors
Ted Justice, Principal Specialist at aeSolutions
Ted has 35+ years of experience in the industrial controls field. He has worked with Instrumentation, Industrial Control Systems, Electrical Systems, Industrial networks, and OT cybersecurity during his career. 15 years of that experience was gained during work at a large municipal water treatment facility.
Dave Gunter, Senior Principal Specialist & Business Development Manager at aeSolutions
Dave has over 20 years in the consulting engineering field working in Process Safety, Alarm Management, and Cybersecurity. Dave is co-author of the book “Implementing IEC 62443 – A Pragmatic Approach to Cybersecurity, 1st Edition”.