Project PowerUp – Helping to keep the lights on in Ukraine in the face of electronic warfare

Monday, December 4, 2023 08:01

• As Russia’s invasion of Ukraine entered its first winter in late 2022, nearly half of Ukraine’s energy infrastructure had been destroyed, leaving millions without power. The resulting energy deficit has exacerbated something that hasn’t had much media attention: The effects of electronic GPS jammers affecting vital electrical equipment.
• Ukraine’s high-voltage electricity substations rely on GPS for time synchronization. So, when the GPS is jammed, the stations can’t accurately report to power dispatchers on the state of the grid.
• This complicates efforts to balance loads between different parts of the system, which is necessary to avoid outages and failure — especially during peak demand and surge times. Until recently, there was no solution to this problem.
• Cisco Talos worked alongside several other teams at Cisco, along with government partners in the U.S and Ukraine, to find a technological solution.

Since the start of the Russian invasion of Ukraine, Talos has been unwavering in our commitment to protect Ukrainian critical infrastructure from cyberattacks.

In this blog post, you won’t find any mention of malware, DDoS, or espionage campaigns. In fact, it’s not about cybersecurity at all. This is a story about electronic warfare and GPS. It’s about how one chance conversation over dinner led me on a path through Cisco to find a solution to some very tough questions, and difficult answers.

So, who am I? My name is Joe Marshall. I work at Cisco Talos as a cyber threat researcher and security strategist. My expertise is in industrial control systems and electric grids. My colleagues and friends at Talos are on the front lines of keeping the internet safe — and from more than just cyber threats, as you’ll read.

Project PowerUp is the story of how Cisco Talos worked with a multi-national, multi-company coalition of volunteers and experts to inject a measure of stability in Ukraine’s power transmission grid.

Our ultimate goal was to “keep the lights on” in Ukraine, and help make the lives of Ukrainians who are living in an active war zone, just that little bit easier.

Chapter 1: The energy deficit

As Russia’s invasion of Ukraine entered its first winter in late 2022, Russia stepped up attacks on Ukraine’s energy sector to deprive citizens of electricity and heat during the coldest part of the year. Nearly half of Ukraine’s energy infrastructure had been destroyed, leaving millions without power. The resultant energy deficit was exacerbated by another wartime challenge that, for some reason, hasn’t had much media attention: the effects of deliberate GPS disruptions affecting vital electrical equipment.

For the past year, there have been numerous reports of Russia interfering with GPS signals, especially near and within its own borders. Use of electronic jamming devices has been linked to attempts to disrupt GPS guided munitions, protect troops, and advance the tactical and strategic goals of armed conflict.

While electronic interference can affect the battlefield, it is also having a secondary, unintended effect on Ukraine’s energy sector. Many of Ukraine’s high-voltage electrical substations — which play a vital role in the country’s domestic transmission of power — make extensive use of the availability of precise GPS timing information to help operators anticipate, react and diagnose a complex high-voltage electric grid. This is a complicated task during normal times, much less during a war.

When GPS signals are widely disrupted, substations cannot synchronize their time reporting accurately because they cannot assign accurate timestamps. Without good synchronized data, efforts to manage loads between different parts of the system can be affected, and this management avoids outages and failure, especially during peak demand and surge times. This disruption can be widespread, causing wide areas to lose GPS service for long periods of time.

Until now, Ukraine has not had a viable solution to this issue for electric power systems.

Chapter 2: A chance meeting

I first learned about this situation when I was in delivering a cyber security presentation in February of 2023. The audience just so happened to include a delegation from Ukrenergo, the electricity transmission system operator in Ukraine and is solely responsible for operating the country’s high-voltage electrical lines. Talos has been working with Ukrenergo for many years.

The night before the presentation, colleagues from Ukrenergo invited me to dinner. When we sat down, I couldn’t help but persist with a barrage of questions: “How are you? Are you safe? What’s going on?”

They started to tell me the true extent of what had been happening. This was one year since the start of the invasion. It was still deeply cold in Ukraine, and Russia had continually bombarded critical infrastructure for the entire winter. By March, there would be word that Russia’s campaign was beginning to tail off, but we didn’t know that at the time.

Ukrenergo started to list problem after problem, specifically with regards to the power grids. The obvious problems we all knew of course – kinetic strikes against substations were knocking out the power. Energy transformers were being destroyed, and replacements were scarce. One problem mentioned was rather casual, sandwiched in-between others, “We can’t get reliable timing due to electronic GPS jamming.”

As I mentioned earlier, Ukraine’s high-voltage electricity substations rely on GPS for time synchronization. So, when the GPS is deliberately disrupted, the stations can’t accurately report to power dispatchers on the state of the grid.

My ill-informed, rather bombastic response to this was, “Just buy some atomic clocks! You know…the type used by NASA.” Only after the words came out of my mouth did I remember that atomic clocks might not be a financially feasible option for this war-torn country. In fact, one member of the Ukrenergo delegation wryly retorted (I’m paraphrasing here), “Sure. Show me the aisle of the grocery store where atomic clocks can be found cheaply.”

For the rest of the night, we talked about the GPS issues, the war, and Ukraine’s response to being attacked. Despite the sober undertones, the dinner company was superb, and the fellowship top-notch. The GPS timing issue, however, wouldn’t leave my head. I tried to look at it from all different angles.

When we said goodbye that night, I silently vowed I was going to do everything in my power to help. But at the time, I had no answers.

💡

High-voltage substations are critical components in the power system where power can be pooled from generating resources, transformed to different voltage levels, and delivered to the load points. Substations are interconnected with each other, creating a network that increases the reliability of the power supply system by providing alternate paths for power flow. This ensures that power delivery is maintained at all times and there are no outages.

Substation in Ukraine damaged by Russian airstrikes

Chapter 3: The time paradox

While thinking about viable solutions, I was guided by an important principle: Whatever we do, speed is key. As I was wracking my brain, Ukraine was at war and suffering. However, I soon began to learn that it wasn’t as simple as that, due to the sheer complexities of what the country was up against.

To truly understand the layers of solving this issue, I need to talk about why GPS clock timing is so important to electric grids. Most people are familiar with GPS because we rely on it for navigation, but it has also become the dominant system for the distribution of time and frequency signals globally. The U.S. controls and operates at GPS satellites that orbit the earth twice a day which broadcast signals anyone in the world can use.

These satellites send very precise time data to GPS receivers on the ground that receive and decode the signals, effectively synchronizing each receiver to the same clock. This enables users to determine the time within 100 nanoseconds without the cost of owning and operating expensive and complex equipment, such as atomic clocks.

Because GPS time is so accurate, GPS-disciplined clocks are commonly used in industrial systems, like Ukraine’s power grid, that require extremely precise time across a vast geographic area.

Most devices that rely on time to calculate measurements have frequency references. The frequency reference is provided by an internal crystal oscillator within the device, and that crystal tells the device how fast time is going. However, these times are never perfectly accurate due to manufacturing variations and other variables in the crystal oscillators, causing time to advance at slightly different rates across various devices. This is why the clock on your laptop might be a few seconds or minutes ahead or behind the clock in your car.

GPS solves this challenge. Devices can use the GPS satellites’ time signal to determine how accurate its local time reference is and then adjust the time accordingly, thereby enabling all devices running GPS-enabled clocks to be aligned to the exact same time.

These GPS time signals are crucial for making a key piece of power equipment called a phasor measurement unit (PMU) run effectively. PMUs are used in power systems around the world to augment operators’ visibility into what is happening throughout a vast power grids network. A PMU measures a quantity called a phasor, which is the magnitude and phase angle of a voltage or current at a specific location on a power line.

PMUs are essential to providing a detailed and accurate view of power quality across a wide geographic grid. Data from PMUs allows operators to predict and detect stress and stability on the grid, identify inefficiencies, and provide information for event analysis after a disturbance occurs.

PMU data is time-stamped — to the precision of a microsecond — using the timing signal from GPS satellites. Therefore, measurements taken by PMUs in different locations are accurately synchronized with each other and time-aligned using the same global time reference marker. This allows all PMU data to be combined to provide precise and comprehensive information about an entire power grid.

When GPS clocks are unavailable and the corresponding time signal has an error, that error can cause false calculations of phase angle and mis-alignment of grid conditions relative to other PMUs. Without the ability to analyze the precise timing of an electrical anomaly as it propagates through a grid, grid operators have difficulty diagnosing the exact issue that requires correction. Relatedly, if GPS timing is down, grid operators will have increased difficulty balancing power during the adverse events that occur during wartime.

Chapter 4: “You don’t need atomic clocks”

After that fateful dinner with Ukrenergo, I spent the next few nights in deep thought. My brain wouldn’t let go of this timing issue. I consulted with colleagues and experts from other organizations who specialize in electric grid security, and ironically, they all suggested the same thing – atomic clocks.

I knocked on Talos Vice President Matt Watchinksi’s door. I explained the situation to him, and ended by saying, “So can Cisco make an atomic clock?” I’d got it into my head that the only possible solution was to create a version of an atomic clock, as their holdover is measured in nanoseconds of accuracy. More than enough accuracy for a power grid.

Matt responded by saying he had no idea, but he would make some phone calls. That led me to a meeting with our Cisco Internet of Things (IoT) division. I asked them the same question I asked Matt: “Can Cisco make an atomic clock to counteract the GPS jamming, like what is being reported in Ukraine?

After some research and identifying all manners of issues with locating an atomic clock, the team said, “Actually. We don’t think you need one. We think we have an existing solution within our IoT networking equipment. We can use that to build something unique for this specific situation.”

As is the case with most things in life, you should put your faith in the experts. And I’m so glad I listened to the IoT team. Because that was how we turned the ship, and Project PowerUp was a go.

Together with Cisco’s IOT networking team, we were going to design, create and deliver custom devices to Ukraine to keep substations running and delivering power to the entire country.

“Throughout this war, I’ve seen and heard how resilient Ukrainians are. It’s very true. Citizens are dealing with one awful situation after the other, to the extent that this mentality of everyday trauma has become normalized. However, ‘getting used’ to power outages and not being able to keep warm in the Winter shouldn’t be normal. That’s what this whole project is about.” Eric Wenger, senior director of technology policy for Cisco Government Affairs

Chapter 5: Is it good enough?

I mentioned earlier that this initiative was guided by the principle that speed was key. Delays meant potentially disastrous consequences. But I soon came to add another principle: Perfection is the enemy of good enough.

The IoT team’s suggestion was that a Cisco Industrial Ethernet switch would be the best starting point in identifying a potential solution to the issues caused by Ukraine’s GPS outages. Industrial Ethernet switches do not have atomic clocks for holdover accuracy – but they have a good enough clock, able to measure time accurately in microseconds, to sustain an accurate time sync. This is important – Ukraine’s electric grids operate on a 50hz frequency and have timing needs in microseconds.

An Industrial Ethernet switch is part of Cisco’s hardened suite of switches, routers and other products designed specifically for rugged deployment, and Ukraine’s warzone undoubtedly fits into that category. These devices are built to withstand harsh industrial environments and extreme temperature ranges (-40° to 75°C).

Hardened switches also have various internal resiliency features, including a source for its internal clock. Most network hardware devices use an internal crystal oscillator to generate their clock time, but these crystals’ frequencies can oscillate widely based on local conditions. However, an Industrial Ethernet switch can avoid this problem, as its crystal is a superior and resilient design, providing better frequency stability for precise synchronization of features such as GPS reception.

Despite an Industrial Ethernet switch’s advantages, we needed to make some software modifications that would enable the device to address the specific set of challenges facing Ukraine’s power grid.

There were two core issues we had to address with the Industrial Ethernet switch that required us to make enhancements to the device. First, we had to ensure interoperability between an Industrial Ethernet switch and the PMUs, and second, an Industrial Ethernet switch needed to provide the necessary holdover during GPS outages for the PMUs to continue working. Holdover is the time period to keep the clocks in sync until timing signals can be restored.

During Ukraine’s GPS outages, which can last several hours, the PMUs effectively declare that something is wrong and stop sending data to the broader power management infrastructure — which causes significant upstream effects. Our first goal was to find a way to keep the PMU transmitting data. By modifying the metadata that an Industrial Ethernet switch sends to the PMUs, the PMUs will continue operating and sending data even without that signal.

Next, we had to enable the Industrial Ethernet switch to provide an accurate time to the PMUs when time was unavailable (aka, the “holdover” period). We modified the Industrial Ethernet switch’s code to provide trusted time.

With an Industrial Ethernet switch deployed to Ukraine’s substations, it measures the difference between the PMU’s local time reference used by the PMU and GPS time while GPS is still active. Then, when GPS signal is lost, the PMU can revert to using the local time reference, which is now highly accurate from the earlier error measurements, thereby allowing the PMU to continue operating.

To ensure that an Industrial Ethernet switch fully understands what the GPS signal is telling it before the signal shuts down, Cisco created new, enhanced clock recovery algorithms. We also applied some additional filtering to the device’s software to allow it to recognize that the signal is down and to provide a “best guess” of what the time was when GPS was lost.

We now had a device that was ready for production, but the job wasn’t done until testing was completed. After successful testing, Cisco immediately prioritized production of these devices. Hardware and software engineers from across the company pooled their collective expertise and created a production line capable of supporting the unique needs of Ukraine.

Our switches in Ukraine!

From the very start of Project PowerUp all I kept thinking about was the big picture of what we were trying to achieve. I’m proud to say that Cisco did this in an incredibly fast timeline. It is no easy feat to re-prioritize production efforts, especially in a technology company as vast as Cisco. But we had that guiding principle of speed and urgency – the longer this took for us to get these devices into Ukraine, the more days Ukrainians would be threatened with grid instability.

A special shoutout to our Cisco Critical Accounts team. This team has been relentless in helping get key deliveries to Ukraine since the start of the invasion, and they were able to help drive the urgency for Project PowerUp too.

Chapter 6: Closing thoughts

As I write this, our Industrial Ethernet switches are in Ukraine, and helping keep the lights on. This reminds me of what we do at Talos every day. We fight the good fight every day to protect others.

It is a lamentable fact that in cybersecurity and in critical infrastructure protection, we’re often confronted with the fact that our work, while valuable, may never be realized in our lifetimes as professionals. It is the legacy we leave with others we help protect and is built upon a large community who believe in fighting that good fight for generations to come.

Project PowerUp is a little different. We know, beyond a doubt, that our work there will help save lives and will help keep the lights on. The effects are incredibly difficult to calculate, but we know it’s going to make life better. It’s helping others stay out of harm’s way. It’s helping a hospital that may not have reliable backup power. It’s giving a child just five more minutes of their childhood watching cartoons.

If anything can be taken away from this, it’s that acting and leading with empathy is core to our mission at Talos. This year we took a chance to make a tangible difference in the lives of others and help them have a better life. Fighting the good fight isn’t just about cybersecurity – it’s about doing the right thing and helping others in the face of adversity.

What started as a chance presentation this year turned into a multi-national, multi-company global team of power grid security practitioners who had never worked together before. As a team, we overcame numerous challenges to make Project PowerUp work. We could not have been successful without the support of numerous experts in Cisco who helped innovate this novel solution. And, of course, we must thank our partners in Ukraine, the U.S. government, and ICS vendors and experts who lent us their time, empathy, and expertise. We are humble and grateful for their help.

Slava Ukraini!