Extreme California heat knocks key Twitter data center in Sacramento offline
Extreme heat in California has left Twitter without one of its key data centers, and a company executive warned in an internal memo obtained by CNN that another outage elsewhere could result in the service going dark for some of its users.
Twitter, like all major social media platforms, relies on data centers, which are essentially huge warehouses full of computers, including servers and storage systems. Controlling the temperature in those centers is critical to ensuring the computers don't overheat and malfunction. To save on cooling costs, some tech companies have increasingly looked to place their data centers in colder climates; Google, for example, opened a data center in Finland in 2011, and Meta has had one center in northern Sweden since 2013.
"On September 5th, Twitter experienced the loss of its Sacramento (SMF) datacenter region due to extreme weather. The unprecedented event resulted in the total shutdown of physical equipment in SMF," Carrie Fernandez, the company's vice president of engineering, said in an internal message to Twitter engineers on Friday.
Major tech companies usually have multiple data centers, in part to ensure their service can stay online if one center fails; this is known as redundancy.
As a result of the outage in Sacramento, Twitter is in a "non-redundant state," according to Fernandez's Friday memo. She explained that Twitter's data centers in Atlanta and Portland are still operational but warned, "If we lose one of those remaining datacenters, we may not be able to serve traffic to all Twitter's users."
News of the outage comes a day before Peiter "Mudge" Zatko, Twitter's former head of security who turned whistleblower, is due to testify before the Senate Judiciary Committee.
In his whistleblower disclosure to multiple US government agencies, first reported last month by CNN and The Washington Post, Zatko warned that Twitter had "insufficient data center redundancy" raising the risk of a brief service outage or even Twitter going offline for good.
Twitter has not disclosed the number or locations of its data centers, but Zatko's whistleblower disclosure cites a public news report that identifies the Twitter data center in Sacramento and another in Atlanta. In 2020, Amazon announced that Twitter had selected its cloud computing platform, Amazon Web Services, to serve some tweets from Amazon data centers.
During his time at Twitter, Zatko learned that "even a temporary but overlapping outage of a small number of datacenters would likely result in the service [Twitter] going offline for weeks, months, or permanently," according to his whistleblower disclosure.
In a statement about the Sacramento outage, a Twitter spokesperson told CNN, "There have been no disruptions impacting the ability for people to access and use Twitter at this time. Our teams remain equipped with the tools and resources they need to ship updates and will continue working to provide a seamless Twitter experience."
Data centers need "reliable water, power, humidity controls and refrigeration to live," said retired brigadier general Greg Touhill, who served as the US government's chief information security officer in 2016 and 2017.
"You want redundancy, not duplication, of your data locations to enhance your cyber resilience so you can take a punch from a natural disaster [or other event] that can take down a single piece of equipment or data center," Touhill, who now heads the CERT Division at Carnegie Mellon University's Software Engineering Institute, told CNN.