Apr 09, 2024 | Posted by Abdul-Rahman Oladimeji
Content Delivery Network (CDN) company Cloudflare suffered a second data center power failure at its core facility in Portland, Oregon, within six months.
As a result of the first incident which happened in November, Cloudflare introduced ‘Code Orange’; a new process where the company shifts most or all engineering resources to addressing the issue at hand when there’s a significant event or crisis.
“Over the past five months, we shifted all non-critical engineering functions to focusing on ensuring high reliability of our control plane,” the company said. “We had spent months preparing, updating countless systems, and activating huge amounts of network and server capacity, culminating with a test to prove the work was having the intended effect, which in this case was an automatic failover to the redundant facilities.”
In March, the company suffered another power issue again at the same facility. However, thanks to several months of failover work, the impact on its services was minimal compared to the November event. “On March 26, 2024, at 14:58 UTC, PDX01 experienced a total loss of power to Cloudflare’s physical infrastructure following a reportedly simultaneous failure of four Flexential-owned and operated switchboards serving all of Cloudflare’s cages,” Cloudflare said of the outage. “This meant both primary and redundant power paths were deactivated across the entire environment.”
