If you want to hear cyberattack horror stories, ask a penetration tester.
But you might have trouble beating the one pen tester Terry Cutler of Montreal’s Cyology Labs told Monday during the MapleSec online conference hosted by IT World Canada.
It started with a plea for help he received on a Sunday night from an unnamed Canadian laboratory with offices across the country. It had been hit two days earlier by a ransomware attack.
Most firms pay the ransom, clean the computers and get back online. But this incident turned into a six-day nightmare. It’s “one of my most stressful stories,” Cutler recalled. Here’s why:
1. The ransomware was distributed by a malicious attachment. Unfortunately, employees across the country kept clicking on the attachment, increasing the number of infected computers (and, separately, increasing the ransom).
2. Before any software could be installed, Cutler’s firm had to collect forensic evidence on each computer for insurance purposes. But by then all the machines were offline, so data collection had to be done manually on each of the 200 computers. That involved bringing in another IT support firm.
3. A clean parallel network was built so the company could get back online and the PCs were scoured. For extra protection, a new endpoint detection and response (EDR) solution was installed on each. But when the PCs were connected to the new network they got infected again. The network and PCs were rebuilt, and all were infected again when connected to the new network. After it happened a third time they discovered that the laptop of the technician doing the rebuilding had been hacked with a remote access software called Off The Record, which the ransomware gang was using to keep uploading malware.
4. By Day 5 the ransomware demand for decryption keys had hit $800,000. Deciding not to pay, the company turned to its backup tapes, “but the technicians didn’t know where all the tapes were,” said Cutler. “They were not in order, they were all over the place. You have to have them in a specific order to retrieve the data. And there were terabytes of data. When we finally got all the tapes, we couldn’t mount it (the recovery software) because the driver for the tape drive was on the old Windows Server 2003. It wouldn’t run on (the environment’s new) Server 2019. So we had to wait for the evidence collection to run on the old server before we could even start to restore process.” That took 17 hours.
5. The tape library database had been destroyed by the ransomware, so had all the backup tapes had to be re-indexed. A data recovery firm had to hired to do that.
Ultimately the ransom was negotiated down to $175,000. But the firm lost a week’s work, during which the bulk of the employees were still being paid.
“I was so sure they were going to shut their doors because nothing was going right for us until they had to pay the ransom,” said Cutler.
There was no shortage of lessons from this attack:
- Every firm must inventory all the software it has on every computer and server. This is particularly vital for old software. Do you have all the software installation keys for each application in case it needs to be re-installed? Remember, some data may need to run on the software it was captured on and not the latest version.
- In addition to making sure data is backed up, test your backup and restore procedures to make sure staff understand what has to be done.
- Have offsite backups that aren’t connected in real-time to the network so they can’t be infected.
“IT guys are always asking for money,” said Cutler. “Upper management never believes the organization will be hacked. I can promise you once ransomware occurs, the budget magically shows up because they don’t want to go through this again.”
The MapleSec conference continues Tuesday and Wednesday. Registration is free.
