Montreal-based web hosting provider Web Hosting Canada (WHC) has revealed the cause of this weekend’s major outage.
In a blog post, founder and chief executive officer Emil Falcon said that the issue was caused by unauthorized activity by a third-party service provider.
“Based on our investigation to date, the morning of August 28 at approximately 6 a.m., an individual with a third-party service provider used their privileged account access to connect to one of our datacenter’s management portals and without authorization, initiated server reimaging on some of our backup servers, then on some of our production servers,” he blogged.
“Within only hours our incident response team had identified the issue and disabled access to the source account, preventing any further damage. The environment was secured, the individual fully locked out, and our disaster recovery plan immediately kicked into action, but damage was already done.”
He said that both production servers and backup servers were affected, resulting in data loss that may be permanent for some. A “large number” of web hosting and reseller hosting accounts were affected; Falcon said that his team was able to recover, or is in the process of recovering, more than half of the lost accounts.
“We can confirm that Cloud, Dedicated, Weebly and Managed WordPress accounts were largely unaffected,” he added.
However, he said, several production servers and their backup servers are still unrecoverable by his team, and the data recovery specialists he has enlisted believe that the potential for their recovery is low. While their efforts will continue, he has shifted focus to setting up new accounts for affected customers.
For customers with their own local backups, he advised contacting WHC’s support team, who will assist in getting sites up and running. Those without local backups have to start from an empty account. For them, WHC has activated new “LifeBoat” hosting accounts, accessible from the Client Area, which will remain free until at least January 1, 2022.
Update: by contributing reporter Howard Solomon
The head of a Quebec-based company that uses WHC and who asked not to be identified said his firm has lost three to four months of data as well as copies of websites he manages. His firm’s most recent local backup is three months old.
It will take time to recover. “It may take two or three days, we’re going to put a little work into it, get back on our feet and have our websites back and running.” But, he added, those who don’t have local backups will be in trouble.
“I’m more disappointed than nervous,” he said. Until now he was happy with the service he got from WHC. “They were on the ball,” he said.
It was on Sunday, when he realized he wasn’t getting any new email, that he suspected there was a problem. He went to WHC’s support chat page and saw there were several dozen queries already lined up. Only when he dialed into the support phone line and heard a recorded message did he learn of the seriousness of the incident.
The destruction of data, he said, is as bad as a ransomware attack.
