Cause and Effect: When an Managed Service Provider (MSP) Falls Victim to Ransomware

The Problem with RATs

When a Managed Service Provider (MSP) is a victim of cybercrime, the scope of damage can be vast. Most MSPs are using Remote Management and Monitoring (RMM) tools to allow them to provide quick and effective service remotely, instead of needing to get an employee on-site for every little fix. These tools, and their function as Remote Access Tools (RATs) in particular, means that if a cyber criminal gains full access to the MSP’s RMM tool, the threat actor can have full access to every system for every client. Additionally, it’s very common that an MSP uses the same RMM tool to manage their own systems, meaning that the damage caused by the cyber criminal can ransom both the client’s site, the MSP managing the client, and remote copies of backups, leaving the MSP in a very difficult position.

Tetra Defense has a wealth of experience in the world of Incident Response (IR), ranging from assisting clients with just one single machine that runs their organization to MSPs that have all available systems fully encrypted. Last year, we were contacted by one of our cyber insurance carrier-partners requesting we assist with an MSP that was the victim of cybercrime.

In this particular case, the ransomware known as ‘Gandcrab’ was deployed to all managed systems through one of the MSP’s RMMs, ‘Kaseya’. The cyber criminals were targeting MSPs with a specific plugin for Kaseya that had a vulnerability allowing root access via SQL injection.  Unfortunately, all managed systems included every client production machine, every client backup server, all machines owned and operated by the MSP for their internal use, and their servers hosting remote copies of backups.

All Hands on Deck

Tetra began the engagement with a scoping call that included the MSP, letting them describe the situation in their own words. There were 36 affected clients that needed help, and every single one was down. The MSP simply didn’t have enough manpower to assist every client and was so over-worked that they had to turn off their main phone line. The MSP was doing their best but didn’t have the experience or the capacity to make their clients whole.

After reviewing a list of clients affected by the ransomware, Tetra made the decision that we needed to engage each client individually to most efficiently remediate the problem. This allowed Tetra to get a deep understanding of each client’s setup, needs, and most critical systems. It also allowed Tetra to break the clients up into groups and assign both its employees and the MSP’s employees as the primary leads, letting the strengths of each team member work best with the unique situations of each client.  Additionally, scoping each client individually allowed for rapid assessment of how many machines truly needed decryption, which gave Tetra substantial leverage in negotiating the final ransom payment.

For some clients, the solution was simple. A single server runs the organization and fresh installs of Windows could be deployed to all impacted workstations.  For more complicated clients, however, it can take hours to manually identify each key encrypting data on a critical system. For those clients, Tetra turned to its development team; in less than an hour, Tetra’s developers wrote custom software to automatically identify all encryption keys on all critical systems. This dramatically reduced both DFIR team member billable hours (which would have been spent on menial tasks) and the amount of time it took come to a conclusion on how many keys need to be negotiated for.  While the cyber criminal may identify thousands of keys, by programmatically determining only the critical keys needed, Tetra was able to keep ransom payment costs to a minimum.

Bugs in the System

Soon after receipt of the decryptors, it was discovered that there were bugs embedded with them. While the data decrypted typically was restored correctly, large amounts of files were being skipped over by the decryptor, resulting in unstable systems. The Tetra development team again was brought into action, and by dissecting the decryptors, the bugs were discovered, and programmatic workarounds were developed, allowing the MSP and Tetra to decrypt systems without needing to remotely access the hundreds of servers.

As the clients with simpler environments were getting back up to full operation, the Tetra team continued to focus on the more complex cases. We quickly identified and addressed issues with domains and active directories, SQL databases, vendor-specific software, networking, and others by leveraging the experience and undivided attention of multiple DFIR agents.

As restoration efforts were conducted, a final forensic examination for data access or exfiltration was performed, and Tetra again brought in its development team, this time to programmatically collect evidence from all impacted sources, ranging from workstations to servers to the compromised Kaseya server itself. By aggregating all of the evidence from different source types into groups, Tetra was able to quickly determine that there was no data exfiltration or access, and thus, no need for notification to affected individuals.

Check out some related content on our blog: