Forensic Case Files: Virtual Machine Matryoshka (Virtual Machine Forensics)

“Keep thinking. You can hear our brains rattling around inside us, like the littler Russian dolls.”
– M. T. Anderson

In this case study, we got a forensic investigation client who needed some help with their server. They were conducting an investigation into some possible impropriety committed by an ex-employee or two. These two former workers had left to work for a direct competitor. Our client was concerned they may have taken some things with them.

And they weren’t talking about pilfering pens or other office supplies. Rather, our client was concerned that these ex-employees had made off with something much more valuable: Data. More specifically, our client hoped we could help them determine, through computer forensics, whether or not these employees had either:

  1. Taken data on our client’s own clients and brought them to their competitors.
  2. Taken proprietary documents from a client’s virtual machine and edited them to use for their new employers.

Our clients ordered a ten-hour virtual machine forensics examination to see if we could shed some light on how these two ex-employees had been using (or abusing) the virtual desktops they’d used during their time at the client’s business.

Virtual Machine Forensics: A Computer Forensics Matryoshka Doll

Virtual machine forensics is like opening up a Matryoshka doll--There's always more inside.

Virtual machine forensics is like opening up a Matryoshka doll: There’s always more inside.

Virtualization and servers mix as well as peanut butter and jelly. Organizations of any size can use their servers to host “virtual machines” (often abbreviated “VM”). A virtual machine is a file that essentially “pretends” to be an entire hard disk drive, complete with an operating system. They are computers within computers, like dolls within dolls, wheels within wheels, and birds within birds. Virtual machines can act as “remote desktops,” allowing employees to access their computers from afar; but all roads lead to Rome and everything feeds back to the host machine.

Most servers take several hard disk drives—from two to twenty, or even more for organizations with very large storage needs—and link them together into single storage volumes. The way each hard drive connects to the others is called “RAID,” for “redundant array of independent disks”. There are many RAID levels, all of varying degrees of complexity, as they use various methods to combine the disks.

VM forensics often ends up providing us with a situation not unlike Russian nesting dolls, also known as Matryoshka dolls… or an onion (or a turducken). A case involving a virtual machine typically has layers upon layers. We must image the individual hard disks, figure out the RAID jigsaw puzzle and piece the drives together, explore the full storage volume (and however many iSCSI targets they carved it into), and finally go through the virtual machines themselves. There are wheels within wheels in just about every digital forensics investigation to some degree. But nothing takes nesting quite so literally as virtual machine forensics.

Opening the Nesting Doll: Making a Forensic Image of Virtual Machines

First, we had to take each individual hard drive in the host computer and make forensically-sound disk images. Next, we combined each forensic image file according to the appropriate RAID protocols (in this case, RAID-5, which breaks the drives up into “stripes” and creates blocks of “parity” data to maintain data integrity in case of one drive failing). Then, we explored the file system in the forensic image to find the Hyper-V virtual machines used by the two employees in question. After finding the virtual machines, we could finally open up each virtual machine and explore its contents. From that point on, the virtual machine forensics case proceeded like any other computer forensics operation.

That’s So Meta

Much of digital forensics involves taking a careful look at metadata. The prefix “meta-” refers to something that is about itself. For example, metadata is “data about data”. When you create any kind of data, such as a Word document, or an MP3, you also create data about that data. This often includes when the data was created, last modified, and last accessed, along with who created it. To say metadata helps out our forensic investigators is the understatement to end all understatements. Metadata tells the story of data. It is absolutely invaluable to in forensic investigations. In this case study, taking a look at the metadata keyed us in quite a bit on these ex-employees’ behaviors.

Our digital forensics examination of the first user’s remote desktop on the virtual machine turned up the location of a mapped network drive, as well as several documents that appeared to belong to a rival business (which both users in question had recently left our client to work for).

We could not turn up the documents shared on the network drive (since it was not a part of the virtual machine server). But we could find the next best thing:  a list of the documents that had been copied over to it.

The documents still living on the virtual machine proved easier to find. By examining the virtual machine’s registry files, we could figure out which files and documents the user had most recently created and edited.

We examined the second user’s virtual desktop. We found plenty of suspicious documents labeled with the name of the same business as before. By examining registry keys and other metadata, our engineers found some interesting contradictions in these documents. The strange inconsistencies suggested that the user had created at least one of the documents by copying an analogous document belonging to our client, editing it, and saving it in new locations. By following a trail of metadata breadcrumbs, we could even find the original document they had used as a “template.”

Virtual Machine Forensic Assessment Results

We found no concrete evidence that either user had transferred any of our client’s documents containing especially sensitive data about their customers, as our clients had feared. That said, we provided them with our list of the documents the ex-employees had transferred to the mapped network drive so they could review it to see if any file names stood out to them.

We did, however, find much more concrete evidence through digital forensics that both users had been appropriating at least one of our clients’ proprietary documents (such as templates for forms and invoices) from the virtual machine and editing them, possibly for use by a rival business.

We fully documented every step of our ten-hour forensic assessment and relayed all of our findings to our client to assist in their further internal investigations. The task of unraveling this situation now lay in their (now much more well-equipped, thanks to us) hands.

Check out some related content on our blog: