by Rick Geiger, Cisco Systems Inc.
rid modernization, with its proliferation of smart meters, grid sensors, advanced distribution automation, wide-area measurements and the like, is driving rapid growth in the size and complexity of Internet Protocol (IP) networks and data centers responsible for grid operations—the area usually known as operations technology (OT).
Process maturity and best practices to design, secure and manage such large, complex systems, however, are outside the experience of most OT departments.
Many utility information technology (IT) organizations have the experience and process maturity these grid operations need.
To make this work, though, IT must recognize the requirements of critical control systems and the knowledgebase, expertise and perspective of OT.
And OT must understand the strengths and value of IT expertise in networking, computing and security.
Successful companies are finding ways to establish common ground and to combine the proficiency and knowledge of IT and OT.
Collaboration is essential to expose IT to OT challenges and requirements and to bring OT to understand IT capabilities.
Embracing IT-OT Convergence
Utility IT organizations are expected to have the development and process maturity for handling complex IT infrastructures and networks.
To communicate these capabilities, IT may help OT gain insight through a dialogue around questions that should be familiar from process maturity assessments:
- What is IT’s process model (such as ITIL)?
- Have all IT personnel been trained in this process model? Were they trained once or is there a cadence of continuing education and professional development?
- Is there a defined mentoring and education program for new hires?
- How is business process management maturity or progress measured?
- What process is in place to make corrections and apply lessons learned to continuously evolve and improve the model?
Business process models (BPMs) give structure and repeatability to the means used to achieve a business or operational result.
Repeatability is key; once a successful way is developed, it can be taught to others and used again without mistakes or false starts.
Structure is also important. It allows corrections and adjustments to improve or meet changing requirements.
Every IT and OT department will recognize these needs and qualities in its daily operations.
Process development and maturity for utilities also must consider the many mandates for regulatory compliance.
In OT departments, particularly, compliance planning, tracking, reviewing, submittal and auditing can dominate process needs and requirements.
Break Down Those Siloes
Few IT departments have had to understand the requirements and priorities of real-time control systems.
As a result, nearly every utility engineering department has horror stories about an IT technician who runs a port scan, installs patches or performs some other normal IT maintenance process that crashes a control system or otherwise interferes with operations.
Many control systems cannot be taken down for maintenance or any other reason until provision has been made for continuing safe operation.
There’s an inclination for IT to be critical of such circumstances without understanding how they came to be that way.
Operational control systems often originally were designed and installed as stand-alone systems and were not interconnected or available for remote access.
There was no requirement for periodic applications of patches or other maintenance processes once correct operation was established.
Likewise, there wasn’t any need to respond to new malware or keep updates current.
But the development of threat vectors aimed at stand-alone systems and the increasing connectivity required for grid modernization are changing all that.
OT system users are different from IT users.
Typical OT system users range from field-workers with different environments that require a different approach to the human-machine interface to users that operate 24/7 and require a different approach to maintenance, backups and patch management.
And the users’ tolerance for errors and failures is much smaller than the typical IT user’s.
Mirroring the difference in users, vendors of OT systems traditionally have been more limited and specialized.
Although the industry has begun to insist that OT vendors meet the same standards as IT vendors, this has been an evolutionary process.
North American Electric Reliability Corp. (NERC) critical infrastructure protection (CIP) has encouraged this transition, but the industry is still evolving.
In nearly every utility, IT and OT engage in their own separate strategies, planning and budgeting.
Executive management might not converge until the chief operating officer (COO) or even CEO level.
This autonomy and focus has served utilities well, but now it can result in a lack of dialogue, few or no relationships, and little opportunity to establish trust or the shared context for relevant collaboration.
This is familiar to every utility executive: separate departments, different missions, focused execution and few opportunities to develop or experience a culture of collaboration.
Collaboration is essential to expose IT to OT challenges and requirements and bring OT to understand IT capabilities.
The goal is development and implementation of best practices, governance and continuing education.
This includes ongoing assessment of security, quality and effectiveness.
Creating collaborative IT-OT teams is the best way to address the growing complexity of networking and computing brought about by grid modernization.
Building Common Ground
The start of common ground is shared purpose and objective.
Defining this is a critical role of executive management.
Utilities with the most successful convergence initiatives have executive management with a common vision, commitment and collaboration and can convey that commitment to all levels of the organization.
Without executive leadership, the respective organizations’ path of least resistance is to continue to focus on short-term goals without achieving long-term change.
IT and OT have been separate since the company was founded. What is the utility trying to achieve?
Why do IT and OT need to come together now? The vision and commitment must be clear.
IT and OT must gain understanding of the value, capabilities and needs of the other organization.
There’s no automatic recipe that works everywhere, but many examples of success exist.
Collaboration begins with building a common context: a basic understanding of the goals, requirements and processes of IT and OT. This starts with fact sharing.
In the U.S., IEEE Power & Energy Society (PES) regularly runs regional Plain Talk courses for nonengineering industry professionals.
Here IT professionals or anyone can learn the basics of how electric power systems work and the utility’s need for security, communication, telemetry and control.
In addition, most major OT suppliers have open professional development courses for utility personnel to learn more about the basics and specifics of their technology.
The goal is not to turn IT professionals into power system or control engineers; rather, it is to provide them with information, resources and a foundation from which they can understand OT needs and objectives.
Control system and power engineers benefit from IT for non-IT professionals.
Executive education courses are available at local colleges and universities and focus on the management of information and communication technology.
The aim is to raise awareness of best practices and process maturity that are IT’s stock in trade along with capabilities required to address the enterprise environment, security and operations management.
Courses also are available from vendors of business process models, such as the Information Technology Infrastructure Library (ITIL), a widely accepted approach to IT service management.
In addition to cross-training, common ground requires personal relationships.
A lot of business conflict centers around turf and trust issues. An underlying cause of such issues is insecurity because of a perceived threat to someone’s job, department or organization.
Relationships are required for trust to develop, turning “they” and “them” into “we” and “us.”
People don’t develop personal relationships without opportunity. If relationships are essential, then so is providing the opportunity for them to develop.
A good relationship builder is working on a cross-functional team to accomplish a short-term deliverable that requires the skills of both departments.
Careful attention needs to be given to team dynamics to make sure there’s no room for finger-pointing or one side’s blaming the other for failure to meet goals or schedules.
The team must be small enough that no one can hide or say he or she doesn’t agree with the goal or how it is met.
The first cross-functional team should be built using volunteers and thought leaders from the respective departments to help establish a new best practice.
One of the best ways to find common ground is to temporarily assign personnel from IT to OT and vice versa.
This assignment needs to be full-time and long enough to build the relationships that will be essential to future collaboration.
Six months is a good start; 12 months is even better.
Both organizations will take a resource and productivity hit with such a swap, as the skill set mix is not completely interchangeable.
Expectations should be set aggressively to bring people out of their comfort zones, but not so aggressively to result in failure or hopelessness.
Nothing works better than immersion, and both organizations benefit.
Temporary reassignment also should include a mentoring program. Assign a volunteer to train the person from the other department.
Working with the respective department directors, managers and supervisors is key so they are invested in its success.
Managing the Risk
How should an executive at a regulated, investor-owned utility manage or minimize the risks that have prevented his or her organization from bringing IT and OT closer together? Is it risk from past conflict or poor results? Or is it the perception of risk?
Often, perceived risks come down to the very same turf and trust issues discussed.
This is why common vision and commitment at the senior executive level is important.
Risk management often is achieved by repeating what works.
Repeating the past might work well in a relatively static environment where the business model, regulatory model, technology and expectations remain the same or change slowly.
The utility industry is transforming thanks to regulatory mandates and public policy.
Utilities are being driven faster and further than before. In this case, managing risk by repeating the past might increase risk in all areas.
Risk management has its own process formalizations that can be found in ISO 31000, Project Management Institute, and other standards.
As with all process formalization, these do not provide guaranteed answers but offer a framework for a methodical approach that can be applied with assessment of results and application of lessons learned.
As the industry changes and past practices no longer address changing needs, ISO 31000-based risk management becomes an active process of clearly identifying business objectives, assessing where uncertainty arises, and applying resources to manage and reduce uncertainty where possible and actively monitor and reassess as progress is made.
What are the IT-OT convergence risks and unknowns? Past attempts might have had poor results, such as the unfortunate IT technician mentioned.
Unsuccessful experience might be used as a reason for inaction when it should be examined for lessons learned.
It’s critical to manage risk based on the suggestions given: establish common context, provide opportunity for developing relationships and build common ground.
The other key to managing risk is establishing ownership and vested interest in the successful results by the IT and OT professionals responsible for daily operations. Industry consultants may provide essential information and guidance, but internal teams should never lose ownership of goals, process and results.
To achieve commitment, internal teams must engage in a collaborative process to develop requirements, architecture and a design for the future state; assess the current state; identify the gaps; and develop a plan to move from the current state to the desired future.
Building the Business Case
Factors in developing a cost-benefit analysis that shows the business case for IT-OT convergence include:
- Ongoing process development and optimization;
- Total system life cycle costs;
- Costs of synchronization and de-duplication; and
- Cost factors for system life cycle management on IT technology timescales.
It is logical to start by looking at the process evolution in IT and follow a parallel path of process development, maturity and optimization for OT. The business case analysis needs to address whether the organization can afford the time and resources required and whether the processes and systems developed to carry OT forward will be the same as those in IT.
The cost model can be complex because it must include system costs and implementation, maintenance and other life cycle costs that typically constitute 70 to 90 percent of total cost.
Given the much shorter technology time cycles of IT technology, replacement or expansion systems and components might not be available over the normal 20-year time frames expected of power delivery infrastructure. System life cycle planning including new system acquisition, migration planning and decommissioning is prudently considered up front.
Duplicate systems carry the risk and expense of making sure that any overlaps stay in synch. The alternative to duplicate process development siloed within OT is to expand IT to provide those process services in collaboration with OT.
Although not free of cost, IT departments have been evolving in this direction for some time, building capabilities such as Infrastructure as a Service (IaaS) and Applications as a Service (AaaS).
Implications for Security, Reliability and Compliance
Security and reliability are the strongest drivers’ bringing about the collaboration of IT and OT. The industry knows the magnitude and costs of NERC CIP compliance as it has evolved.
More changes remain with pending NERC CIP V5, NISTIR 7628, Common Criteria, the February 2013 Executive Order on Cybersecurity, and the potential unaddressed alignment with FIPS.
Integration of renewable and distributed energy resources significantly challenges legacy reliability practices.
Demonstration projects funded by the Department of Energy (DOE), research from the national labs, the Electric Power Research Institute (EPRI), universities and others point to near real-time telemetry and active control as essential for reliability of the envisioned ecosystem of distributed energy resources.
The technology solutions that are developing to address security and reliability in the face of these unprecedented requirements involve enormous expansion of network communications for telemetry and control and equally enormous expansion of data centers for the computing power to store, analyze and manage the vast quantities of data.
Summing it All Up
Who is better equipped to handle the significant challenges during the decades to come? IT or OT?
OT has the critical knowledge of power engineering, real-time control systems and the operational requirements of reliability and power engineering.
IT has the critical knowledge of advanced IP networking, architecture, security and the process maturity to address large, complex network, computing and storage systems.
Grid modernization has vital needs that can be addressed only through collaboration, executive leadership and alignment to common vision and goals.
IT and OT organizations must find common ground to which they can bring their capabilities to address the needs of the utility control systems that constitute critical infrastructure.
Rick Geiger is the executive director of Cisco’s smart grid and utilities business transformation team. He serves on the Gridwise Alliance board of directors and is an IEEE Senior Member and member of the Power and Energy Society.
PowerGrid International Articles Archives
View Power Generation Articles on PennEnergy.com