Live Linux forensics in a KVM based environment (part 1 memory)

Live Linux forensics in a KVM based environment (part 1)

Most of this blog will be based on a image that I created that I will be walking through. You can obtain this image https://docs.google.com/file/d/0B4pePbirlzGvMjdIblpiV1FjanM/edit?usp=sharing .  You will need to image this to a usb drive preferably a 8gb drive like I used in the talk  To write the image you just need to issue a dd if=./4n6.img of=/dev/your_drive.

Scenerio:
Network team has mentioned they are seeing abnormal traffic to 172.20.20.114 please check out srv03 at 192.168.122.226.

Host system:
OS= ubuntu 12.04 server
user = admin-user
pass = master

Compromised guest:
OS= centos 6.4 64bit
HDD config = 3 disk RAID5 luks encrypted
luks passphrase = mi4n6mi4n6
root pass = master

I will try and write this in a way that will parallel using the techniques on a live virtual instance.

Second, note that the domain that I will be using is srv03. You can replace this value with whatever running domain that you have in your instance.

Memory

One important piece of the incident response puzzle is a memory dump. We know this and obtaining it from a windows machine is fairly trivial. Getting it from a Linux machine also not that hard you just need to have root level access. What about if you have access to a KVM based virtual machine that has several servers running on it? How do you get the memory from the running instances of these systems without having to touch each one of them? Well lets dive in and see. 

First things first, once you have booted the image you will need to run the "srv03_restore.sh" in order to restore the state of the running VM. Once this has completed you will have a running instance of the VM. If you open up xterm you can run the "virt-manager" command and have a visible console on the VM instance.

The quick and dirty

If you have a suspect system and you want to look into what is going on you can just do a dump. This will essentially dump the used memory of the running server. One caveat is that it will temporarily pause the running OS.While it dumps the memory. The nice thing is that it will only dump the used memory, but that can be bad as well since we could miss something, but alas this is quick and dirty. So lets get to it.

You can dump using "virsh dump svr03 ./mem.dump"

now that we have this we can start our investigation. The fast way is just using strings and grep. Since we know were to start traffic to 172.20.20.114 we can grep for that.  "sudo strings mem.dump | grep 172 | less"

with a little searching in the less we come across this:

This is interesting I would say. Something has opened a reverse shell to the suspected ip address. Since I want to be thorough I keep looking and then I find this later in the memory:

Hmm what is this matahari.py and what is it doing the uploads directory? A quick search lends to the knowledge that it is a simple command and control system. This command is showing that that is checking in with the server at 172.20.20.114 . Ok, I think at this point we can say the system is compromised. I am not going to dive into this just yet since I want to show another way that we can access the memory of the system since we have the nice feature of being virtual.

Memory for Volatility

Since we talked the quick and dirty lets get a little more elegant. Let's say that we want full memory in a format that we can use such awesome tools as volatility. How can we do this? With libvmi of course. http://code.google.com/p/vmitools/ . This tool ends up being perfect for what we are looking for.

You will first need to install the tool which has a perfectly good readme that will tell you how to install. For those that cannot wait its just as simple as

autogen.sh
./configure
make
sudo make install

The tool that we are going to talk about is pyvmifs.py . In order to use this tool you will also need to install it. It will be located in the libvmi/tools/pyvmi directory. The installation is also strait forward with "python setup.py build" and "python setup.py install" . This will get you up and running. As a side note there are package dependencies they are met with the build that you downloaded.

Now that we have it installed we can actually mount the memory in a format that we can read with volatility. You can mount it like this. "sudo python ./pyvmifs.py -o domain=srv03 /path/youwant/to/mount/"  and you should see a file names "mem" the size of the memory that is allocated for that virtual instance.

Now we have the mounted memory we can do what we want with it. Strings and grep or use volatility. Let's go the volatility route since we already went the strings grep way. To do this you will need to have a profile for the running linux instance that include the dwarf file and the symbols from boot. Here is the profile for a default centos 6.4 64bit.

http://dl.dropbox.com/u/12565646/cent64.zip

This profile I placed in the volatility directory on the image. If you downloaded the image and are using it there is no need to download again.

One interesting command that requires a little extra information that I want to run first is the "linux_bash" command. This command requires the use of a value that you can obtain by disassembling the /bin/bash binary history_list. You can obtain the value from a previous post but I will also include it here.  Let's run the command and see what we can get from it.

"sudo python vol.py --profile=Linuxcent64x64 -f /media/g/mem linux_bash -H 0x6e0970 -P"

You can see that this gives some interesting information but doesn't show the host that we want. You can see that some one has used "nc" to 192.168.122.129. This is interesting but not what we are looking for. Let's try another command such as netstat.

Great this gives us what we are looking for further confirmation. Let's try another command just for fun linux_psaux.

"sudo python vol.py --profile=Linuxcent67x64 -f /media/g/mem linux_psaux"

Ok, now we have more information.  We can now proceed and get the forensic disk image and see what this matahari.py is.

In conclusion we now have 2 ways to obtain memory from a virtual instance. I also talked about a few ways to deal with this memory. How you choose to go forward with this will be your choice. Hopefully this gives you the chance to experiment and play with these ways on a running system. 

Part 2 will entail how to use the images and mount the luks encrypted raid 5 guest OS .