by Paul Rostick
Once, back in my consulting days, I did a quick IT inventory of a newly-installed industrial automation system I was working on: It was an EtherNet/IP-based network consisting of 65 multi-vendor switches, within which ran a Microsoft Domain containing 42 VM-hosted servers running 145 core pieces of multi-vendor software, arranged in two separately-located fully redundant ‘mini data centers’ configured in real-time failover mode.
It’s no secret that modern industrial automation systems look increasingly like highly complex IT systems. However, I’ve found that modern Operations departments don’t act like they are fully aware of this resemblance, which is odd, considering that a failure in the Operations IT components will cause a failure in the Operation and such failures can cost millions. Operations departments traditionally obsess over managing and maintaining the reliability of their physical assets, but, from a lost-revenue perspective, does it really matter whether a shut-down is caused by a failing pump or a failing hard drive? And we haven’t even mentioned the prospect of cyber incidents.
In some parts of the industry there’s a begrudging sense that someone must look after the “IT” inside Operations, and so some poor Operations tech gets to run system backups or patches when he finds the time, or in other cases they’ll just call in an engineering firm or the vendor when something breaks. (And believe me, IT stuff breaks.)
But that’s not how our contemporaries in Enterprise IT behave when it comes to managing their IT. They’re fully staffed and tooled. They have sophisticated care-and-feeding regimens, specialty-knowledge disciplines, and deep security practices they’ve honed over decades of managing all that complexity. Why haven’t Operations departments accepted and embraced their new-found role as the owner and master of a vast complex of Operations IT assets?
It comes down to culture. Operations and Engineering cultures were fully formed long before the invasion of IT. These guys really know their stuff. Meanwhile the IT invasion into ICS is relatively new and is still evolving and expanding. To this relative unfamiliarity you can add that the vast majority of IT complexity is unseen – it’s in the network stacks and the software layers and in the system configurations and the myriads of services and protocols that comprise an IT platform. If you haven’t professionally grown up managing and securing IT, you are likely unaware of its deep and intricate complexity – and the significant effort it takes to keep it running reliably and securely.
Personally, I’m on a mission to convince Operations departments to embrace, rather than begrudge, their destiny as IT Shops, and to convince them of the mandate to commit to being as good at managing and securing their Operational IT assets as they are at managing the reliability, safety and security of the Operation’s physical assets.
What about you? What do you find in your experience with ‘Operations IT’?