#Dark NL

Climate change and business continuity planning

Cars covered in Newfoundland, Canada, after 40cm snow, high winds and extreme cold lead to widespread and extended power failures across Newfoundland in January 2014.  Photo: Joanna McDonald. — Cars covered in Newfoundland, Canada, after 40cm snow, high winds and extreme cold lead to widespread and extended power failures across Newfoundland in January 2014.
Photo: Joanna McDonald.

Climate change, urbanisation and population growth are beginning to push the boundaries of human endurance. Today natural disasters involving flooding, bush fires, exceptional snowfalls, water shortages, power outages, to name a few also involve widespread collapse of transport and communications systems.

The connectivity that benefits us daily, is fragile and can collapse with triggers far outside the planning controls of any, single organisation. The ability to co-ordinate any response, or evacuate people, can be severely hampered, if not rendered impossible.

One emerging aspect to business continuity and crisis management, is the rapidity with which crisis events spread. We are reliant on complex systems to communicate, and to move; people, money, information, news and goods. We have achieved a complexity where a tiny, unseen, unmonitored outlier can become the catalyst that rapidly causes widespread systemic failure.

In addition to the physical impact and consequence of 21st Century crises, the shift to 24 hour news and social media has amplified the scrutiny of organisations and particularly, their leaders, and prolonged the post-crisis period. Adequate organisational response to crisis events may falter in the analysis of public opinion, politics and social media momentum.

With this context in mind it is worth studying ‘near miss’ events like #DarkNL for lessons in business continuity, crisis management and communications that can be carried into our own field of operations.

#DarkNL: Newfoundland & Labrador Hydro’s #majorfail and Twitter’s revenge

On Thursday 2 January, 2014, Christmas and New Year festivities abruptly ended in Newfoundland, Canada. As power went down, temperatures plummeted to -35C and 40cm of snow blasted in.

Between 2 – 8 January, 75%, or 190,000 of Newfoundland Power’s customers lost electrical service for an average of 12 hours over this time span, though for many it was 36 hours.

The situation developed its own hashtag #DarkNL which to this day appears when the still fragile power capacity of Newfoundland is discussed.

Radio and Twitter became essential forms of communication as people saved phone and tablet battery life to stay informed and report their needs. The twitter feed is an invaluable insight into the minutia of impacts on households and organisations. It also exposes the importance of thoughtful crisis communications and leadership.

#DarkNL pushed the province into a full-blown political storm that played into a premier’s resignation, and exposed Hydro’s failure to operate as a High Reliability Organisation.

The impact of #DarkNL was a mini-crisis. Whilst catastrophe was avoided, it delivered however a very severe ‘warning’ of just how catastrophic another winter power failure would be, how vulnerable the residents of Newfoundland are, and illuminates errors that can and should be prevented by robust business continuity planning.

How Newfoundland & Labrador Hydro’s ‘No fail’ regime failed

What happened?

The crisis in Newfoundland was a result of extreme weather and poor planning.

Thursday, 2 January, 2014 – extreme cold

Extreme cold caused a shortage of generating capacity at Newfoundland and Labrador Hydro (Hydro). Hydro and its supplier Newfoundland Power (NP) asked Corner Brook Pulp and Paper, one of its major power customers, to cut back on usage. They also issued a public appeal to homeowners to reduce power before finally implementing rolling blackouts. Shortage was most acute on the Avalon Peninsula, where the majority of Newfoundland residents live, and most people have electric heat.

NP originally planned to run outages, each lasting around one hour, for four hours until 8pm, but sustained cold temperatures saw the outages continue until midnight. Even with outages, usage on Thursday 2 January night set a record load on the system.

Friday 3 January – Severe winter + unprepared Hydro

Temperatures with wind chill were given at -35C.

The Avalon was again the worst affected region, with nearly 40cm of snow falling and heavy winds. Metrobus services and all flights and ferries were cancelled.

As power was restored to areas on rolling outage, the system was unable to cope with load surges. Businesses were asked to turn off the lights and any non-essential equipment.

Customers were asked to conserve power, turn down thermostats and hot water tanks, and not run washing machines, dishwashers, or tumble dryers.

In a press conference that day, Rob Henderson, VP Hydro, referring to the grid capacity said “we saw this week coming…we didn’t have everything available on 1 December” – the date of completion for all system maintenance and repairs ahead of winter peak demand.

Saturday 4 January – Severe winter + unprepared Hydro + Storms + equipment failure and fire

Room with a view? This is what 40cm of snow in St John’s, capital of Newfoundland, Canada, with severely restricted heating and electricity looks like.  Photo: Joanna McDonald. — Room with a view? This is what 40cm of snow in St John’s, capital of Newfoundland, Canada, with severely restricted heating and electricity looks like.
Photo: Joanna McDonald.

The winter storm continued and at 9am, 190,000 customers lost power.

Temperatures in St John’s, the capital, were -15C, or -35C with windchill.

Warming centres were established for vulnerable residents, however ongoing blizzard conditions made it difficult to keep roads clear enough to reach them, and many shelters did not have generators to run their electric heating systems.

Equipment failure at the Sunnyside substation caused a transformer malfunction and resulted in a fire. While the fire was quickly brought under control, grid power for Holyrood Substation was lost, forcing its shutdown. By 11pm, 87,000 customers still had no power.

Frozen pipes were a major household issue. In rural communities, most households have wood stoves as their primary heat source, but failure of electric pumps meant they had no water. Production at the Come-by-Chance oil refinery was also impacted.

Sunday 5 January – Severe winter + unprepared Hydro + Storms + equipment failure and fire + evacuations + continued power loss + Premier’s disastrous press conference.

“This is not a Crisis,” said Newfoundland Premier Kathy Dunderdale.

Even as more equipment failed, care homes were evacuated, back up failed to arrive, and 35,000 people remained without power.

Rolling outages were stopped by 9.30pm but a fault in the Holyrood switchyard caused 100,000 customers to lose power again. 65 households in a St John’s subdivision also lost water due to a power outage at the pumping station.

Additional resources were mobilised from Prince Edward Island to help reconnect power but were unable to land until Monday when the weather improved. Regional hospitals had problems with power outages and flooding, including in one operating theatre.

Dunderdale was roundly criticised for her absence during the unfolding situation, with rumours that she was in Florida, or comfortable in a house with a generator while people endured freezing, dark houses, burst pipes, and listening on battery radios or watching Twitter feeds for any information to help them.

Said Dunderdale: “This is not a crisis. Usually most of these utilities will have a plan that is configured to meet one difficulty or two difficulties happening at the same time. What we had here is a series of events, all coming together, including a fire which has been a major piece at a transformer at one of our stations.”

*Residents helped from Cambridge Estates personal care home in St John’s, Newfoundland, Canada, in 2014. (Photo: CBC).*

All St John’s schools, and its college and university closed for Monday and Tuesday to preserve power. Households were asked again to conserve power. Elderly residents were rescued from care facilities.

Monday 6 January – Severe winter + unprepared Hydro + Storms + equipment failure and fire + evacuations + continued power loss + Premier’s disastrous press conference + help arrives + more power failures + a fatality.

The weather abated by Monday and the extra power crews were able to land in Newfoundland and start helping reconnect households. Eastern Health confirmed a fatality from carbon monoxide poisoning and treatment for nine others in emergency departments.

*Power crews restore damaged lines in Newfoundland, Canada, 2014. (Photo: Globe & Mail).*

A power storm that went on and on and on…

Between 2 – 8 January, 75%, or 190,000 of Newfoundland Power’s customers lost electrical service for some time, on average for 12 hours over this time span, though for many it was 36 hours. Power outages continued throughout the rest of January.

The Inquiry Findings – widespread failure in business continuity planning

Liberty Consulting Group was appointed to conduct an inquiry into events. They found that the outage stemmed from two differing causes:

(a) the insufficiency of generating resources to meet customer demands, and

(b) issues with the operation of key transmission system equipment.

Underpinning this was a widespread failure in organisational decision making and response planning.

Liberty Group found that even with new capacity arrangements, the generation reserves were very low, and the risk of outages remained high for the 2015-2017 winter seasons.

By 2018, a new power source was due to come online from a massive hydro-electric project in Muskrat Falls, Labrador. In fact, the first power was only synchronised with the grid in September 2020.

In summary, Liberty Group found Hydro:

· Used a supply planning weather forecast that was too optimistic and counter to industry norms and did not adequately account for winter peak demand in colder periods.

· Had multiple equipment failures: “Transformer failure, protective relay design, circuit breaker malfunction, and operator knowledge issues all contributed to the January 2014 outages… Not only did equipment fail, but failures had consequence beyond what one would ordinarily expect to occur”.

· Did not complete recommended maintenance on “the failed equipment, and that protective relay design issues and insufficient operator knowledge of the protective relay schemes existed”.

· Reduced transmission reliability performance from 2009 to 2013, below that of Canadian comparators.

· Operated a manual, paper-based outage management process that did not conform to best practice.

· Did not have a customer service strategy in place to “guide day-to-day service response or customer service response during outages”.

· Did not have a single executive with responsibility for all principal functions associated with delivering utility service.

· Did not complete essential maintenance and repairs were not completed by 1 December ahead of peak winter loads.

· Maintenance standards were not robust enough for the 40-year-old system. For example, technologically dated air blast circuit breakers were still in use, but Hydro did not test these devices. Three of these devices failed to operate in the January 2014 events. Testing was only begun afterwards.

· Used a cost-driven asset management program which Liberty Group found “gives more visibility to cost effectiveness than to preventing the kinds of equipment failures”. Examples include deferral beyond established time cycles for maintenance” including equipment that failed. In general, maintenance backlogs were significant, and had grown since 2011.

The lesson for business continuity planning

Rarely will a business suffer a crisis in isolation and be able to assume every other organisation it leans on to support it is operating as “business as usual”. Crisis planning is no longer a function of middle administration completing risk assessments and devising risk protocol, but is, or should be, a Leadership Team led function, embedded within organisational culture. Imagine the very worst that could happen, including multiple simultaneous crises, and ensure that you can still operate.

It could be argued that Premier Dunderdale in her “this is not a crisis” quote to the media perfectly outlined the circumstances, and definition, of a crisis. Crises are often miscategorised and planned for as major incidents.

Modern crises step beyond past lessons and expose the vulnerabilities of our interconnected world in how rapidly the crisis escalates and ripples, and how hindered an effective practical intervention becomes. Decision making can be slowed by critical lines of communication failing, and by there being multiple centres of crisis incident and command. Planned operations can be rendered useless as critical infrastructure dominoes in its failure.

High Reliability Organisations should watch for near misses, patterns among chaos and increasing incidents of smaller events which provide the best steer on which potential crisis scenarios are most likely, if they aren’t addressed. Vigilance akin to a high functional safety-led culture is required. Time spent in rehearsal and training is never wasted.

Someone at Leadership Team level must have the moral courage to speak up when things are patently wrong. Ignoring essential maintenance and repairs, dropping reliability standards, failing to organise for a crisis, and using bottom-line profit as the driver of all activity – all faults that Liberty Group found with Hydro, are sure signs that the Leadership team is under-performing.

Leaders and brand reputation will be held accountable and cross-examined in the court of public opinion even during the crisis event, and for a sustained period afterward. Intensive media and political scrutiny can be assumed. Returning to business as usual is highly unlikely. Indeed, in modern crisis management, the minimum expected is that an organisation demonstrates how it has learned from any perceived failings – whether actual or not – and so change is to be anticipated as part of the renewal strategy, even if it is impossible to know how the existence of the organisation will change.

Related Information:

If you are interested in other blogs written by Joanna check out her Crisis Communications piece here. If you have your own opinion on what Joanna has written, please feel free to comment.