Live Linux forensics in a KVM based environment (part 1 memory)

Live Linux forensics in a KVM based environment (part 1)

Most of this blog will be based on a image that I created that I will be walking through. You can obtain this image https://docs.google.com/file/d/0B4pePbirlzGvMjdIblpiV1FjanM/edit?usp=sharing .  You will need to image this to a usb drive preferably a 8gb drive like I used in the talk  To write the image you just need to issue a dd if=./4n6.img of=/dev/your_drive.

Scenerio:
Network team has mentioned they are seeing abnormal traffic to 172.20.20.114 please check out srv03 at 192.168.122.226.

Host system:
OS= ubuntu 12.04 server
user = admin-user
pass = master

Compromised guest:
OS= centos 6.4 64bit
HDD config = 3 disk RAID5 luks encrypted
luks passphrase = mi4n6mi4n6
root pass = master

I will try and write this in a way that will parallel using the techniques on a live virtual instance.

Second, note that the domain that I will be using is srv03. You can replace this value with whatever running domain that you have in your instance.

Memory

One important piece of the incident response puzzle is a memory dump. We know this and obtaining it from a windows machine is fairly trivial. Getting it from a Linux machine also not that hard you just need to have root level access. What about if you have access to a KVM based virtual machine that has several servers running on it? How do you get the memory from the running instances of these systems without having to touch each one of them? Well lets dive in and see. 

First things first, once you have booted the image you will need to run the "srv03_restore.sh" in order to restore the state of the running VM. Once this has completed you will have a running instance of the VM. If you open up xterm you can run the "virt-manager" command and have a visible console on the VM instance.

The quick and dirty

If you have a suspect system and you want to look into what is going on you can just do a dump. This will essentially dump the used memory of the running server. One caveat is that it will temporarily pause the running OS.While it dumps the memory. The nice thing is that it will only dump the used memory, but that can be bad as well since we could miss something, but alas this is quick and dirty. So lets get to it.

You can dump using "virsh dump svr03 ./mem.dump"

now that we have this we can start our investigation. The fast way is just using strings and grep. Since we know were to start traffic to 172.20.20.114 we can grep for that.  "sudo strings mem.dump | grep 172 | less"

with a little searching in the less we come across this:

This is interesting I would say. Something has opened a reverse shell to the suspected ip address. Since I want to be thorough I keep looking and then I find this later in the memory:

Hmm what is this matahari.py and what is it doing the uploads directory? A quick search lends to the knowledge that it is a simple command and control system. This command is showing that that is checking in with the server at 172.20.20.114 . Ok, I think at this point we can say the system is compromised. I am not going to dive into this just yet since I want to show another way that we can access the memory of the system since we have the nice feature of being virtual.

Memory for Volatility

Since we talked the quick and dirty lets get a little more elegant. Let's say that we want full memory in a format that we can use such awesome tools as volatility. How can we do this? With libvmi of course. http://code.google.com/p/vmitools/ . This tool ends up being perfect for what we are looking for.

You will first need to install the tool which has a perfectly good readme that will tell you how to install. For those that cannot wait its just as simple as

autogen.sh
./configure
make
sudo make install

The tool that we are going to talk about is pyvmifs.py . In order to use this tool you will also need to install it. It will be located in the libvmi/tools/pyvmi directory. The installation is also strait forward with "python setup.py build" and "python setup.py install" . This will get you up and running. As a side note there are package dependencies they are met with the build that you downloaded.

Now that we have it installed we can actually mount the memory in a format that we can read with volatility. You can mount it like this. "sudo python ./pyvmifs.py -o domain=srv03 /path/youwant/to/mount/"  and you should see a file names "mem" the size of the memory that is allocated for that virtual instance.

Now we have the mounted memory we can do what we want with it. Strings and grep or use volatility. Let's go the volatility route since we already went the strings grep way. To do this you will need to have a profile for the running linux instance that include the dwarf file and the symbols from boot. Here is the profile for a default centos 6.4 64bit.

http://dl.dropbox.com/u/12565646/cent64.zip

This profile I placed in the volatility directory on the image. If you downloaded the image and are using it there is no need to download again.

One interesting command that requires a little extra information that I want to run first is the "linux_bash" command. This command requires the use of a value that you can obtain by disassembling the /bin/bash binary history_list. You can obtain the value from a previous post but I will also include it here.  Let's run the command and see what we can get from it.

"sudo python vol.py --profile=Linuxcent64x64 -f /media/g/mem linux_bash -H 0x6e0970 -P"

You can see that this gives some interesting information but doesn't show the host that we want. You can see that some one has used "nc" to 192.168.122.129. This is interesting but not what we are looking for. Let's try another command such as netstat.

Great this gives us what we are looking for further confirmation. Let's try another command just for fun linux_psaux.

"sudo python vol.py --profile=Linuxcent67x64 -f /media/g/mem linux_psaux"

Ok, now we have more information.  We can now proceed and get the forensic disk image and see what this matahari.py is.

In conclusion we now have 2 ways to obtain memory from a virtual instance. I also talked about a few ways to deal with this memory. How you choose to go forward with this will be your choice. Hopefully this gives you the chance to experiment and play with these ways on a running system. 

Part 2 will entail how to use the images and mount the luks encrypted raid 5 guest OS .

Linux bash history_list info for volatility

Starting Linux analysis with volatility

When working in the security field you will eventually come across a time that you will need to do some memory analysis. This analysis will entail working with volatility for memory forensics. Most of the time you will end up working with windows systems that may have been compromised. When you are working against a Linux memory dump you will need a few extra things to make this possible.

linux_bash

The linux_bash option within volatility requires you to to know the history_list location so that you can scrape the bash history out of memory. The way that you would do this would be to use gdb and disassemble the history_list and in the comments you will note the information that you will need. I will include a few of them on this page. I don't want to include too many since they are trying to create a way to determine the value on the fly but I cannot confirm the status of that. Here are a few values that I quickly gdb grabbed out that might help others as well. I will include more if people find it to be beneficial to have a single location. A well documented way to obtain the values are located on the volatility site http://code.google.com/p/volatility/wiki/LinuxCommandReference23#linux_bash


Centos
6.4  - 0x6e0970 - bash-4.1.2-14.el6.x86_64.rpm
6.3 -  0x6e0950 - bash-4.1.2-9.el6_2.x86_64.rpm
6.2 -  0x6e0910 - bash-4.1.2-8.el6.centos.x86_64.rpm
6.1 -  0x6e0910 - bash-4.1.2-8.el6.centos.x86_64.rpm
6.0 -  0x6e1af0 -  bash-4.1.2-3.el6.x86_64.rpm
5.9 -  0x6bf970 -  bash-3.2-32.el5.x86_64.rpm
5.8 -  0x6bf970 -  bash-3.2-32.el5.x86_64.rpm
5.7 -  0x6bf970 -  bash-3.2-32.el5.x86_64.rpm
5.6 -  0x6bf970 -  bash-3.2-24.el5.x86_64.rpm

Ubuntu
11.04 - 0x6ed3a8


This probably gives an idea of what I will talk about next ...linux profiles.  I will create a profile for all of the above systems and provide them on the next post. This post will also be updated until the disassemble piece in 2.3 happens.

References:

http://code.google.com/p/volatility/

Tale of the misconfigured script

This attack attempt made me laugh a bit. I see the following event from my Sguil instance running on Security Onion that is monitoring my honeypot:

I pull the transcript and find the following:

I thought that was a very odd password to attempt. I wanted to a transcript of all the attempts the attacker tried, so I used tcpdump to carve out the session between my honeypot and the attacker with the following commands:

 cd /nsm/sensor_data/jbc-eth0/dailylogs/2013-02-24/
 tcpdump -r snort.log.1361664061 -w ~/ftpbruteforce-pcap-20130224.pcap ip and host 61.129.71.42 and host 192.168.1.20 and port 21 and proto 6

 Securityonion will log full pcap's to /nsm/sensor_data/<sensor-name>/dailylogs/<date>. The tcpdump command is pretty much the same command that sguil issues to the sensor to generate transcripts. The -r is to read in a pcap file, -w is to write the results to a pcap file. After that is the BPF (Berkley Packet Filter) filter which defines what traffic we want to carve out.

I opened ftpbruteforce-pcap-20130224.pcap in wireshark and started looking at the sessions and found this one:

Turns out, there are multiple attempts of using %username% and some other strings. It seems our attacker forgot to configure this field. Logically, I assume it's supposed to try different variations of Administrator and a string such as:

Administrator1
Administrator12
Administrator123
Administrator1234

If anyone knows the FTP brute forcing tool that was likely used, please let us know. My Google-fu is failing as Google drops punctuations from searches.

Brief NSM Analysis of FTP Dictionary Attack

On the 15th I saw an event in Sguil I had been waiting for: "ET POLICY FTP Login Successful". The credentials are Administrator/password. I was surprised how long it took.

I ran an event query on the destination IP (this traffic is flipped from the true src/dst):

You'll see PADS registering a new asset as the attacker's IP is first observed, an alert that the attacker is trying at least 5 unsuccessful attempts to log in as Administrator, then finally a successful login:

Since FTP is in cleartext, we can easily inspect the traffic by right clicking the Successful Login event and selecting "Transcript". I'm pasting the traffic instead of using a screen shot:

DST: 220-FileZilla Server version 0.9.41 beta
DST:
DST: 220-FileZilla Server version 0.9.41 beta
DST:
DST: 220-written by Tim Kosse (Tim.Kosse@gmx.de)
DST:
DST: 220-written by Tim Kosse (Tim.Kosse@gmx.de)
DST:
DST: 220 Please visit http://sourceforge.net/projects/filezilla/
DST:
DST: 220 Please visit http://sourceforge.net/projects/filezilla/
DST:
SRC: USER Administrator
SRC:
DST: 331 Password required for administrator
DST:
DST: 331 Password required for administrator
DST:
SRC: USER Administrator
SRC:
SRC: USER Administrator
SRC: USER Administrator
SRC:
DST: 331 Password required for administrator
DST:
DST: 331 Password required for administrator
DST:
DST: 331 Password required for administrator
DST:
DST: 331 Password required for administrator
DST:
DST: 331 Password required for administrator
DST: 331 Password required for administrator
DST: 331 Password required for administrator
DST:
DST: 331 Password required for administrator
DST: 331 Password required for administrator
DST: 331 Password required for administrator
DST:
SRC: PASS
SRC:
DST: 530 Login or password incorrect!
DST:
DST: 530 Login or password incorrect!
DST:
SRC: USER Administrator
SRC:
DST: 331 Password required for administrator
DST:
DST: 331 Password required for administrator
DST:
SRC: PASS abc123
SRC:
DST: 530 Login or password incorrect!
DST:
DST: 530 Login or password incorrect!
DST:
SRC: USER Administrator
SRC:
DST: 331 Password required for administrator
DST:
DST: 331 Password required for administrator
DST:
SRC: PASS password
SRC:
DST: 230 Logged on
DST:
DST: 230 Logged on
DST:
SRC: RMD sarcaxxo
SRC:
DST: 550 Permission denied
DST:
DST: 550 Permission denied
DST:
SRC: QUIT
SRC:
DST: 221 Goodbye
DST:
DST: 221 Goodbye
DST:

We see the attacker used Administrator as the username in all the attempts and iterated through a couple of guesses for the password: abc123, password. What I was curious about was the "RMD sarcaxxo" command the attacker issued after logging in which is attempting to remove the directory (RMD) named "sarcaxxo", which did not exist on my honeypot. After searching Google, it seems this is a command issued by a tool "Multi-thread FTP scanner v0.2.5" by Inode. If someone wanted to create an alert for the use of this tool, they could use something like the following Snort rule (not tested):

alert tcp any any -> $HOME_NET 21 (msg:"Multi-thread FTP scanner v0.2.5 by Inode - Successful Login and Attempted Directory Removal"; flow:from_client,established; content:"RMD sarcaxxo"; classtype:misc-activity; sid:5001990; rev:1;)

While there were no more alerts, it does not mean the attacker did nothing else. I right click on the Dst IP part of the FTP Successful login event again and select Quick Query -> Query Sancp Table -> Query DstIP/1 Hour.

The results show us that he attacker connected on port 3389 but unlike his connection on port 21, there is no byte count. You can confirm the lack of interesting traffic by pulling the transcript. Since the end times for the traffic on 3389 were before the end time of the FTP traffic, we can guess that the attacker did not have the credentials yet and therefore is likely to be part of a port scan.

Content Security Policy

I promised a post on Content Security Policy (CSP), so here it is. I recently gave a five minute talk about CSP and found it to be really well received. It is an exploit mitigation technology, created by the Mozilla Foundation, designed to minimize the surface area available for Cross-Site Scripting (XSS) attacks. I first learned about it at ArbSec, a monthly local security meeting that has beer. Jon Oberheide gave the presentation, referring to it as the "No NX Bit of the Web". NX bit (never execute) was created to mark pages in memory that should not contain executable code. So if an attacker was able to get code into a memory page that should only contain data, the NX bit will state to the CPU that it should not be executed. For some further reading: http://en.wikipedia.org/wiki/NX_bit

In the basic setup, Content Security Policy does the same in that a conforming website will remove inline JavaScript. Instead, that JavaScript will reside in a separate file and will be allowed to be executed in the policy. The server gives web browsers the policy through HTTP Headers:

Firefox and IE 10:
X-Content-Security-Policy

Google Chrome and Safari:
X-WebKit-CS

Official W3C Header (supported by Chrome version 25):
Content-Security-Policy

Once one of those headers is present, the web browser will disable inline JavaScript in these forms:
   - <script> code </script>
   - onclick
   - eval
   - setTimeout
   - setInterval

Source: http://en.wikipedia.org/wiki/Content_Security_Policy

When visiting http://ipsec.pl/node/1064 you can see the following sent back from the server:

HTTP/1.1 200 OK
Date: Mon, 31 Dec 2012 16:40:33 GMT
Server: Apache/2.2.16 (Debian)
Set-Cookie: SESS67bfceea1c6380ba6e0ac811371dccf6=vYrEGqQOJQnsIiNUBFC5djzTy-5; path=/; HttpOnly
Last-Modified: Mon, 31 Dec 2012 16:40:32 GMT
ETag: "29dc312167f2b7f026f157d5942f0e9d"
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: must-revalidate
Content-Encoding: gzip
X-Frame-Options: sameorigin
X-Content-Security-Policy: allow 'self'; script-src www.google.com www.readability.com; options inline-script; img-src *.creativecommons.org
X-XSS-Protection: 1; mode=block
Content-Type: text/html; charset=utf-8
Keep-Alive: 60
Content-Length: 7842

The key here is: X-Content-Security-Policy: allow 'self'; script-src www.google.com www.readability.com; options inline-script; img-src *.creativecommons.org

It is the policy pushed to your browser saying that JavaScript should only be loaded from google.com or readbility.com, no other places. To successfully perform a XSS attack in this scenario, the attacker would have to find a way to load the malicious JavaScript to google.com or readbility.com, greatly increasing the cost and effort required by the attacker.

For the full W3C CSP document: http://www.w3.org/TR/CSP

To utilize this technology, it is as simple as moving the JavaScript in your website to another file and configuring your web server to send the policy via the HTTP Headers. For Apache, allow the module:

LoadModule headers_module modules/mod_headers.so

And add this very simplistic policy to .htaccess (source: http://community.developer.authorize.net/t5/The-Authorize-Net-Developer-Blog/HTTP-Headers-to-Help-Secure-Your-Website/ba-p/8604):

# Don't allow any pages to be framed by my site or any others
# Defends against CSRF. Current only support by Firefox 4 but the
# other browser will add support soon.
Header set X-Frame-Options DENY

# Only allow JavaScript from the same domain to be run.
# Also, don't allow inline JavaScript to run.
Header set X-Content-Security-Policy "allow 'self';"

# Turns on IE 8 XSS prevention tools
Header set X-XSS-Protection "1; mode=block"

# Don't send out the Server header. This way no one knows what
# version of Apache and PHP I am using and try to exploit it.
Header unset Server