Quote, "I'm OK with delayed patch installation and extra security measures, but every patch needs to be tested by them and certified for installation. They have no mechanism for doing that at all."

They actually do have a mechanism for testing and releasing what patches are acceptable on their systems. This articles talks about Wonderware and Rockwell systems, both of which I use on a daily basis as an end user. Both make available a list of what patches they have tested and vetted out for their systems. It is a pain in the butt to shift through their databases on their websites and you have to have support contracts with them to get to the information, but both companies do exactly what you say they do not do.

They do recommend turning off windows update and keeping it away from the internet and your business network as part of the security model, but that is a gross simplification of industry best practices and what they recommend.

This is an interesting question. Air gapped is the best solution, but not always acceptable.

Some places need to allow remote access to their controls systems for troubleshooting purposes because they have few experts and it is impractical to fly your controls engineers all over the place.

Even more common is the need to get historical and inventory information up to the business network real time so people can make proper decisions. Security best practices talk little about air gapping because almost everyone wants the data available on the buisness network. Instead, they focus on a multi layers security approach that includes patching, demilatarized zone, Intrusion threat prevention and detection software, etc.

Good companies will have an approach and weight the risk/benefit and the put in the security. When Stuxnet came out, the CEO of my company sat up and then started asking questions. Before that, we did very little and were at significant risk, but no one was looking at controls systems. Stuxnet opened up the world's eyes so security through obscurity became a whole lot more risky for controls systems. I ended up with a couple of others presenting a plan to the CEO and IT Steering committee a security plan and needs and walked out with permissions to spend a lot of money to get up to industry best practices fast! Some places aren't so lucky.

The last 8 years has been a game changer for controls systems security. It used to not be discussed much. Now, it is forcing more controls engineers to learn about network security and more IT people to look at the controls network differently and work with controls to harden connections. We still have a long way to go, but things are improving generally in industry.

Yes, like you were agreeing to, to run an industrial controls systems you don't need much power, it is just the SCADA (operator interfaces and historian) where you start needing a lot more and it is very difficult to do without getting into the PC and server areas.

As to controls systems being on the internet, yeah, those people are idiots or dealing with stuff that is non proprietary, non life threatening.

At my place, there is not outside logging into the system. They need troubleshooting, I drive in. Other places don't have that luxury and have to make it so you can VPN into the corporate network and then have a path that gets to your controls systems. If done right, can be pretty secure, but where I work we have decided it isn't a risk worth taking.

There are some places where they put their controls systems on their business network or some with no separation between the internet and their SCADA systems. Those people are just asking for trouble.

Agreed. Even air gapped, a system still needs to have security patches applied.

It doesn't matter how great your protocols are to limit access, included in your policy has to be patching. There are many things you have to consider when securing a network with life critical things attached to it. When and how you apply those patches is just as important as any other parts of the policy to secure your network.

Are my systems at risk from Meltdown and Spectre because I have not patched? Yes. Due to other layers in the security, is it likely that those virus's will get to my system? No. Can we wait until everything had been vetted by the vendor so we can apply them without introducing risk into our system? I am not a diviner so can't answer 100%, but we are betting the answer is Yes.

Yes, absolutely, most industrial automation are run on Windows.

No, it isn't that scary. It can be, but if properly implemented with the right security in mind, you can keep the system up and running reliably.

As stated below, the windows machines are used for the operator interfaces and to record information. The things that actually controls the process are different and unaffected by this and the screwy things with windows.

I am a controls engineer, i.e. program, spec, maintain, industrial controls systems for a living. I work with 4 others and we have a combine over 120 years experience doing this and between the 5 of us have seen hundred of manufacturing facilities. Yes, Windows do occasionally make us want to throw PCs out the window, but properly implemented the Windows box going down rarely is a big cause for concern as long as you can get it up quickly.

With that said, where I work we have been pushing to go over to servers and thin client implementations, but still running Windows servers as opposed to Win CE, 95, 98, XP, Vista, 7, 10 (we skipped 8)....yes, I have set up installed and troubleshoot industrial software on all of those.

A little off on your description. SCADA is Supervisory Controls and Data Acquisition. There are several parts and pieces to that and in most systems, that included the operator interface which is usually run on Windows machines.

Yes, the equipment that interfaces with the field equipment is fine, but the operators can't see what that equipment is doing.

It would be like saying, your car is fine and the engine is running, but your brakes, gas pedal, steering wheel all stopped responding and your windshield is covered in dirt so you can't see. Engine is fine though!

I work in a facility as a controls engineer that has Wonderware and Rockwell software and I use on a daily basis the software affected by these patches. We didn't path because we don't patch until the vendors vet out patches and say it is ok and we also received the notice that said don't apply the patch.

I know of other facilities that went down because they applied these patches. Yes, the PLCs and controllers were still working, but you can't run blind. Even if you could, the historians have the data you need for EPA compliance or to certify your product for customers so when that goes down, you stop running.

I am a controls engineer and use the software mentioned in this post.

First, controls guys who know anything and don't get IT telling them, you must do this now, will never install a patch until vetted by the manufacturer. I actually got a notice from the vendor saying, don't install this patch 2 days after the patch was available.

As to being more complex then they should be or simple...

The actual controllers that run the process are extremely simple, extremely hardened and designed to run 24/7/365. PLC processors cost $4000-$15,000 depending on type and memory and they get into the hundred of meg of memory.

Where it gets difficult is when you start using PCs to run your operator interface. There are tons of graphics, reports, trends, etc and you use software that is designed to run on Windows, which most of your operator interfaces are designed to do.

When a patch like this hits, the operator interface or historian has issues, but the PLC running the process keeps doing it's job, you just can't see into the PLC.

So yes and no. There are things that are more complex and that could be simplified/run separate from windows, but those start getting prohibitively expensive and the tiny bit of extra reliability is not needed. Those kinds of systems cost 2-5 times as much and the development of those systems is more expensive because there are even fewer people with experience with it. If I had experience with those systems, I would be making 70% more then I am now and I am making enough that I don't need to complain.

Yeah, not so much.

Controls systems not connected to the internet still need to be patched and maintained because there are vectors of attack that can still get across an air gap.

Yes, patching isn't as important, but you still have to patch for security and just to be able to stay compliant with software revisions of the software you are using.

FYI, I am a controls engineer, that means I do this for a living. I use the software mentioned that this crashes, but it didn't hit me because I would never apply a patch until it had been tested and approved by the vendor. This patch was not. As a matter of fact, they sent a notice to all of their customers to tell them not to apply this patch because it takes their software down!

Yes, those things exist and are used, but more often they are not used.

Even if you use those kinds of mechanical limits, there are more scenarios then I can count where those are not practical or even possible and you can fire open 2 valves if you have access to the code and can blow stuff up, or vent something to atmosphere or overwhelm a Waste water treatment plant.

When it comes down to it, most things in life are protected by the code of the systems, either process controls systems or safety instrumented systems. There are many ways you can secure systems, like the mechanical limits you mention, but it is all a matter of the risk analysis done and most times, it is in the code. If you have access to the code, all bets are off and you can do just about anything you want to the equipment.

Short version: Equipment which can "explode" because of ridiculous "superhackers" only happens in Hollywood or when you have a completely incompetent engineer, and I seriously doubt you're going to entrust a multi-thousand dollar rig to an incompetent engineer.

I replied to another of your posts, but let me say again here:

I am a controls engineer, do this for a living, know industry standards.

Yes, you have layers of protection to prevent things from happening, but the electrical with a mechanical back up you seem to think is required is not correct. Having one system that does not affect another system is correct, but quite often both systems are electrical and both systems tie into the same controls network and if you can get to one and reprogram, you can get to the other.

Quite often the mechanical things for protection are put in place for when the control system completely looses power and then the system has a back up safe state that requires no power, but if the controls system is in place and working, those mechanical limits don't matter.

Think about your car. It can go from 0-120 mph, but isn't save beyond 80 mph so they put in a software governor so the gas cuts out when you hit 80 mph. They could put in a mechanical limit as well, but it is more expensive, not required and you can't get to the software normally so they don't need to.

I hack your car and remove that, you can now go 120. I hack your car and remove control of the steering and gas/brake and put the pedal down until 115 is reached and then cut the wheel. Even if there was a mechanical stop so I could not get passed 80 mph, you want the car to be able to go 80 so I can still take control of the car and crash you at 80 mph.

Controls systems are generally safe, have many layers of protection, but most of the things you think exist to stop the controls system from being able to make things go boom don't exist most of the time in most industries. Normally, it is the programmable systems that protect you.

I am a Controls Engineer, i.e. I maintain, code, spec, etc. systems like this. Not a programmer for the vendors who make the software, but end user at a plant using controls software and hardware to make things happen.

The smartphone is not controlling anything, it is the window to look into the controls system to see what is happening.

All of the major companies are designing applications that can do the same thing the operator interfaces do from a smart phone that is connected to the same network as the machines. Valve manufacturers are building applications into their valves that a valve can be controlled by a smart phone!

Some facilities are perfectly fine doing things like this. The place I work, I tell the vendors no, do not want, will never want and if they build those into their equipment where I can't order without those options, I will disqualify them as a vendor. They have all said, yeah, so and so across the street said the same thing. I won't even let them have the options in and turned off by software because it could accidentally be turned on or hacked if they didn't secure it.

I also have known of and work in plants where they don't care because they are making food stuff out of food raw ingredients and nothing will blow up. They worry about microbial contamination. Lots of other examples.

Why would you want to be able to do things like this? I know some companies who have people like me that can remotely log into the controls system from anywhere in the world and make changes to their systems or run the system if need be. It is because they have a few experts on the system, the knowledge is not easily transferable so they want them to be able to have the ability to do those kinds of things in a moments notice because a line down costs $10,000 an hour and it is worth the risk.

So the vendors make the applications that are Operator Interface on the go and some people buy those and use those. I don't have any examples to show you exactly what they are because in my role at my company, part of my job is to say no we will not do that. I have seen them and they are really nice and convenient, but you have to ask is it worth the risk.

They are striking on iPhone launch day because that is when it will hurt their employer the most. Any other day of the week, it would be a blip on the radar. On the day that there will be people camped out in front of the store relying on the striking employees to get them their precious iPhone 8s, that is the day the retailers really need all hands on deck.

Getting Slashdot or others to take more of a notice is a side benefit because it is Apple's launch day, not the main benefit.

Interesting your take on the fact they didn't say control network to mean they didn't breach to the controls layer. They said:

"We're talking about activity we're seeing on actual operational networks that control the actual power grid"

"The extent of the campaign as well as the question of whether the attackers had breached operational IT networks, rather than merely administrative ones, was unclear at the time."

I read actual operational network and operational IT networks as they were saying the controls networks had been breached. There are quite a few vague things about the article but I thought they were meaning controls network without knowing the terms. They could have meant a DMZ between the two or something else as well. I read the same thing and took it to mean something other then you are asserting it meant.

You also said they would have spelled it out if there were breaches on the controls network. The vagueness of the article and the fact it came from fortune and not a controls publication make me think they have little clue what they are talking about or very few details. You are also talking about the power grid and there are homeland security implications to discussing breaches on those networks, as in, if there is a breach, it can be classified as secret and if you provide details one of the government agencies shows up to talk to you, which could have lead to limited details being shared.

And FYI, I know the difference. I am a controls engineer sitting at a desk with 2 laptops, one on the business network and one on the controls network and while I am not the sole person responsible for network security between the 2, I am one of the major played in it where I work.

I live near Chicago. Illinois is also an at will state.

One of my morning radio talk shows has a practicing Chicago lawyer come on regularly to discuss the legal aspects of current things in the news. They also at times do a call in and ask questions or text them in from listeners. This question comes up all of the time.

She says, unless the employee can prove discrimination due to being a protects class, then the employer can fire at anytime for any reason, no explanation or even wrong explanation and the employer is fine.

She always says the same example. Your boss could come in, look at your blue shoes and say, I don't like your green shoes so I am firing you. Despite the employer being 100% wrong, and you might be the best worker they have, you have no recourse. You are fired, law suit will not work, do not pass go, do not collect $200.

I am not a lawyer, just providing an explanation of what a lawyer has said multiple times on the subject. If what this lawyer says is correct, then this guy has no recourse.

Something else that comes into play is the free speech angle. I haven't heard this lawyer talk about it, but have seen many articles and cases about the fact free speech doesn't protect you from your employer. The Constitution says the government shall pass no law... The constitution doesn't stop employers from saying, you can't speak or talk about certain things. NDA agreements stop that all of the time. Look at Colin Kapernick, he spoke up and he is done over his free speech. NFL doesn't need to allow him to turn their games into his personal platform.

Unfortunately some industrial automation vendors and end users still do have the security mindset of the average IoT device. We are getting better as an industry, but some are still really scary!

One of my co-workers about 5 months ago found a site where someone wrote the script to crawl around the web and look for PLCs and DCS systems and the like that were on the web with no restrictions. Some of them were probably honeypots set to trap people, but as little as 6 months ago, there were still thousands of system that were still connected to the internet!

We didn't dig around to see what they were, but I saw in a tech journal about 2 years ago a controls guy saying he installed the Allen Bradley Logix software on his home PC and found their municipal waste water treatment Logix 5000 PLC right there. He called the people who ran the facility and told them and they blew him off so he logged into the PLC and added tags names, I_Llogged_into_your_PLC, I_did_this_Remotely, Your_systems_Can_be_hacked, etc. He then called them back and said he was already in their system and described what he saw and the tags. The blew him off again but he noticed about 10 minutes later, the PLC was no longer visible on the internet!

It is scary how little some people take security in the controls world, but we are learning! Stuxnet scared a lot of controls people!