Zapped: Google Loses Data from Belgium Data Center
Google has reported that one of its Belgium data centers was repeatedly struck by lightning during a severe weather event causing the cloud giant to lose a “tiny” portion of its data.
Lightning is said to have struck the data center 4 times in a row, causing engineers inside to scramble when the battery backups failed. Google reports that the data loss experienced inside of the Belgian facility equates to 0.000001% of the total disk space held within.
The data center in St. Ghislain, Belgium hosts Google Compute Engine nodes. When battery backups began failing, users of Google Compute Engine saw their workloads suddenly disappear. This happened in part due to the power outage, which caused some servers to fall offline thus killing the GCE instances without explanation.
“In almost all cases the data was successfully committed to stable storage, although manual intervention was required in order to restore the systems to their normal serving state,” Google mentioned in a release.
“However, in a very few cases, recent writes were unrecoverable, leading to permanent data loss on the Persistent Disk. This outage is wholly Google’s responsibility,” Google added.
Engineers were reported to have begun implementing backups as soon as power became available within the facility. In addition to Google’s apology note, Google has also mentioned that it would take steps to improve the equipment within its facilities so that the data center isn’t prone to this sort of outage again.
Google didn’t provide many more details than that. Although lightning falls under a “Natural disasters” category in terms of disaster recovery, lightning strikes impact data more often than you’d think. Events such as this illustrate that a facility doesn’t need to be struck in the same place for failures to occur. Lightning strikes are a real possibility that can lead to data loss as evidenced by Google’s data center blip.
