Comment From the top... (Score 1) 227
Before you begin, revisit the mission statement for the utility. Decide now what kind of disruptions you're willing to tolerate and how much expense you're willing to fund before anyone brings you any kind of technology. Make sure the costs of disruptions and impact to the community is considered in making that decision. Check policy and local/state/provincial laws for potential impacts from outages, and include those impacts in your analysis.
When the analysis is complete, check with public relations to see what they think the response will be. you may need to revisit that decision. When you have a safe way to proceed, loudly and repeatedly publish your decision on how you intend to proceed and what your goals and tolerances are. Make sure the utility people know it and can repeat it in their sleep. Make sure the vendors know. Leave no stone unturned.
When you're about to start buying things and collecting bids, be very, very careful. If you expect the solutions you buy to do something in particular, make sure you word it that way in your requests for proposals. Don't say something like "must be able to log to our log servers" when you mean to say "must log to our log servers." Vendors will seize upon the former and, when you see they're not doing what you want they tend to respond, "well, it's *able* to do that but it'll cost an additional 27 bazillion dollars for a pro services engagement. It doesn't do that out of the box." Small utilities have a hard enough time with finding money without having to put up with that kind of silliness.
When things begin, you will hear from two types of people: Control Systems people will repeat whatever the vendor tells them about what good security is (if they mention it at all), and the vendor will want to connect a lot of stuff to the Internet. They may also want to remotely access your control systems for support and maintenance. There are sometimes reasons for this, but it's almost always an extraordinarily bad idea. Vendor back doors and B2B connections into control systems usually lead to unauthorized changes and unexpected behavior. The next thing you know, you're dependent on the vendor for everything and they can start setting the contract renewal terms. This is bad and becomes costly.
Vendors and Control Systems folks also tend to focus entirely on encryption (and sometimes access controls) as the extent of security, totally ignoring data integrity and often hand-waving availability. Software vulnerabilities that lead to denials of service are an availability issue. Poor authentication and unencrypted protocols are a potential data integrity issue. Don't let the vendors or the Control Systems people slide on those- if they have integrity, encryption, and authentication controls, make the vendor implement them.
Make the vendor come on site to do maintenance and put your foot down with the Control Systems people who want remote access- if it's important enough for need their attention in the middle of the night or on Thanksgiving, it's important enough for them to drive to the site where the equipment is and deal with an issue. Utilities can usually pay for maintenance and overtime, but would rather not pay fines for an outage or a safety issue.
IT Security people will want to do all kinds of intrusive and disruptive scans on the systems, and they like to keep things patched. This is important, but sometimes patching can break the systems that the Control Systems people make. The reasonableness threshold for patching is usually tied to the most recent version of the software and patches that are available from the SCADA vendors. Absolutely keep the vendor's software and firmware patched to the most recent levels, and don't patch the OS beyond what's supported by the vendor. That sounds counter-intuitive to a lot of IT Security folks, but that's the way it is (hence the tedium about isolation.)
As for the collection of Control Systems folks and IT people, they'll need some way to get data into and out of the environment. If you have a small environment, resist the urge to connect the control systems network to the corporate network (even through a firewall). Depending on your mission, it may be a lot safer to transfer the data via pluggable hard disks or USB drives than to do it across the network. Make the IT people come into the control systems environment to do the patching. They shouldn't be reaching in from the outside, and they shouldn't be reaching out to the Internet to get patches. They should download the patches from the support sites and manually transport them in for installation.
Careful communication is key in this kind of an environment. Make sure that the IT people (security and otherwise) are speaking the language of the Control Systems people- make sure it's not the other way around (IT is there to support the business, not to make the business adapt to IT). IT people need to know what the HMI is, and they need to know about PLCs, RTUs, and Engineering Workstations.
Finally, remember that you are the customer. If you want security features for your systems, and if you want a secure design, It doesn't matter that every other municipal water utility on the planet doesn't care about security. Vendors sometimes say that kind of thing. You're paying for the product, you get to say how it gets installed and operated in your enterprise.
When the analysis is complete, check with public relations to see what they think the response will be. you may need to revisit that decision. When you have a safe way to proceed, loudly and repeatedly publish your decision on how you intend to proceed and what your goals and tolerances are. Make sure the utility people know it and can repeat it in their sleep. Make sure the vendors know. Leave no stone unturned.
When you're about to start buying things and collecting bids, be very, very careful. If you expect the solutions you buy to do something in particular, make sure you word it that way in your requests for proposals. Don't say something like "must be able to log to our log servers" when you mean to say "must log to our log servers." Vendors will seize upon the former and, when you see they're not doing what you want they tend to respond, "well, it's *able* to do that but it'll cost an additional 27 bazillion dollars for a pro services engagement. It doesn't do that out of the box." Small utilities have a hard enough time with finding money without having to put up with that kind of silliness.
When things begin, you will hear from two types of people: Control Systems people will repeat whatever the vendor tells them about what good security is (if they mention it at all), and the vendor will want to connect a lot of stuff to the Internet. They may also want to remotely access your control systems for support and maintenance. There are sometimes reasons for this, but it's almost always an extraordinarily bad idea. Vendor back doors and B2B connections into control systems usually lead to unauthorized changes and unexpected behavior. The next thing you know, you're dependent on the vendor for everything and they can start setting the contract renewal terms. This is bad and becomes costly.
Vendors and Control Systems folks also tend to focus entirely on encryption (and sometimes access controls) as the extent of security, totally ignoring data integrity and often hand-waving availability. Software vulnerabilities that lead to denials of service are an availability issue. Poor authentication and unencrypted protocols are a potential data integrity issue. Don't let the vendors or the Control Systems people slide on those- if they have integrity, encryption, and authentication controls, make the vendor implement them.
Make the vendor come on site to do maintenance and put your foot down with the Control Systems people who want remote access- if it's important enough for need their attention in the middle of the night or on Thanksgiving, it's important enough for them to drive to the site where the equipment is and deal with an issue. Utilities can usually pay for maintenance and overtime, but would rather not pay fines for an outage or a safety issue.
IT Security people will want to do all kinds of intrusive and disruptive scans on the systems, and they like to keep things patched. This is important, but sometimes patching can break the systems that the Control Systems people make. The reasonableness threshold for patching is usually tied to the most recent version of the software and patches that are available from the SCADA vendors. Absolutely keep the vendor's software and firmware patched to the most recent levels, and don't patch the OS beyond what's supported by the vendor. That sounds counter-intuitive to a lot of IT Security folks, but that's the way it is (hence the tedium about isolation.)
As for the collection of Control Systems folks and IT people, they'll need some way to get data into and out of the environment. If you have a small environment, resist the urge to connect the control systems network to the corporate network (even through a firewall). Depending on your mission, it may be a lot safer to transfer the data via pluggable hard disks or USB drives than to do it across the network. Make the IT people come into the control systems environment to do the patching. They shouldn't be reaching in from the outside, and they shouldn't be reaching out to the Internet to get patches. They should download the patches from the support sites and manually transport them in for installation.
Careful communication is key in this kind of an environment. Make sure that the IT people (security and otherwise) are speaking the language of the Control Systems people- make sure it's not the other way around (IT is there to support the business, not to make the business adapt to IT). IT people need to know what the HMI is, and they need to know about PLCs, RTUs, and Engineering Workstations.
Finally, remember that you are the customer. If you want security features for your systems, and if you want a secure design, It doesn't matter that every other municipal water utility on the planet doesn't care about security. Vendors sometimes say that kind of thing. You're paying for the product, you get to say how it gets installed and operated in your enterprise.
