How to Investigate a compromised Linux Server

Author: nathacof
Published: Tuesday 14th of October 2008

This article will assist you with a preliminary investigation of a server compromise. If the server appears to have been compromised at the root level, the server is to be considered compromised until it is rebuilt. This is not to say that you have to rebuild just because an intruder gained access to an un-privileged account. You must identify how the server was compromised, so you can patch those areas.

Note: Don't get distracted with what you find, focus on gathering as much information as possible before disturbing the environment

Identify Who is on the Server

Look for suspicious logins. If the customer always logs in from a DSL line in California and then suddenly logs in from Japan, you may want to make note of that.

w && echo "netstat listing" && netstat -nalp |grep ":22 "

last -a

zgrep ssh /var/log/secure* |grep Accept

zgrep ftp /var/log/secure* |grep Accept

Identify current network activity

netstat -nalp

View IP Connection Count

The following command will tell you how many connections are being made to the webserver on port 80.

Replacing :80 , with the port of your application will allow you to see the number of connections associated with any service. If you are using IPv6, replace cut -f1 -d: with cut -f4 -d:

netstat -plant | awk '$4 ~ /:80$/ {print $5}' | \
cut -f1 -d: | \
sort | uniq -c | sort -n
      1 0.0.0.0
      1 127.0.0.1
      1 149.254.192.205
      1 151.65.171.19
      1 165.155.200.87
      1 173.66.139.70
      1 195.93.21.97
      1 60.48.171.251
      1 60.53.227.174
      1 72.30.142.83
      1 75.101.147.30
      1 79.7.248.51
      1 82.206.136.38
      1 83.229.112.20
      1 96.231.93.237
      2 202.133.102.242
      2 41.210.38.158
      2 86.16.94.89
      3 208.54.94.9
      5 41.210.17.188
      5 41.210.35.165
      5 66.150.96.121
      5 83.87.69.25
      9 68.191.207.0
     11 65.49.2.92

What is the state of the current connections?

netstat -plant | \
awk '/^tcp/ {print $6}' | sort | uniq -c | sort -n
     13 FIN_WAIT2
     53 LISTEN
    129 TIME_WAIT
    316 ESTABLISHED
    754 CLOSE_WAIT

Type, and process name:

netstat -plant | \
awk ' /^tcp/ {split($7, a, "/"); print $6, a[2]}' | \
sort | uniq -c | sort -n| tail
      1 LISTEN xinetd
      2 LISTEN memcached
      2 LISTEN slapd
      2 LISTEN smbd
      2 TIME_WAIT 
      3 LISTEN httpd
      3 SYN_SENT firefox
      9 ESTABLISHED httpd
     11 ESTABLISHED firefox
     46 ESTABLISHED slapd

List open files

In Linux everything is a file, including network connections:

lsof -i -n

To view the numeral port number, as opposed to the service name

lsof -nPi

What Processes are Running?

ps -elf

ls /proc/*/exe -la

Unhide

Sometimes process will hide them selves well enough that our shell scripts aren't gonna pick up the process. In these instances I use unhide:

http://www.security-projects.com/?Unhide

Compile Unhide

$ wget http://www.security-projects.com/unhide20080519.tgz
$ tar xzf unhide20080519.tgz
$ cd unhide-20080519/
$ cc unhide-tcp.c -o unhide-tcp
$ chmod o+x unhide-tcp
$ cc unhide-linux26.c -o unhide
$ chmod o+x unhide
$ mv unhide* /usr/sbin

Using Unhide

$ unhide-tcp
Unhide 20080519
yjesus@security-projects.com


Starting TCP checking

Starting UDP checking

$  unhide proc
Unhide 20080519
yjesus@security-projects.com


[*]Searching for Hidden processes through /proc scanning

Found HIDDEN PID: 740
Command:

Found HIDDEN PID: 775
Command:

Found HIDDEN PID: 1004
Command:

Found HIDDEN PID: 2996
Command:

Found HIDDEN PID: 26921
Command: ./123qwelb

Found HIDDEN PID: 27109
Command: ./123qwelb

Found HIDDEN PID: 27213
Command: ./123qwelb

Found HIDDEN PID: 27216
Command: ./123qwelb

Found HIDDEN PID: 27284
Command: top

Check Binary Files

Often times malicious users will replace system binaries with modified copies which will leave back-doors for the attacker to use in the event that the original vector of attack is corrected.

You can use the command strings to view the text data in a binary file. As such you can use this as a way to determine if a binary has been modified in any way.

Compare the output of the following command with that of a known good server:

strings /usr/bin/top

Investigate Process Activity

Wanna see what a process is doing? Run the following command replacing $PID with the actual process id:

strace -p $PID

DESCRIPTION

In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.

The -p flag allows you to attach strace to an already running process.

Suspicious Files

Are suspicious files located in the world writeable directories?

The next thing you want to look at are the directories that are world writeable. More often than not, the intruder is not a hacker at all, but a worm that is spreading through the internet. Many attacks will store a binary or will leave behind other temporary files. The three most common directories to search in are /tmp, /var/tmp, and /dev/shm.

ls /tmp -lab

ls /var/tmp -lab

ls /dev/shm -lab

Many times you will find that the worm/intruder will try to hide subdirectories in ways that make it hard to find how to enter the directory. Using the tab key for the auto-complete often helps. Here are some examples of what to look for:

root:~# ls -la
total 2
drwxr-xr-x 2 nobody nobody 48 2005-11-25 18:32 
drwxr-xr-x 5 nobody nobody 120 2005-11-25 18:32 .
drwxr-xr-x 33 nobody nobody 2320 2005-11-25 18:31 ..
drwxr-xr-x 2 nobody nobody 48 2005-11-25 18:32 ..
drwxr-xr-x 2 nobody nobody 48 2005-11-25 18:31 ...

Point(s) of Entry

Simply cleaning a server will not prevent a future compromise. We need to help the customer identify the point of entry to protect the customer, and our network.

Many times vulnerable web scripts (php, perl, etc) are exploited and commands are then executed on the server as the web user. We are going to want to use grep to search the apache logs for some common commands that are often used by intruders.

You will want to use different commands depending on what control panel software the server is running

No control panel

for i in `locate access_log` ; 
do
 echo $i ; egrep -i '(chr\(|system\()|(wget|curl|perl|gcc|chmod)%20' $i ;
done

You may have to look in the customer's VirtualHost container to ascertain the real name of the log file.

cPanel

The following code will check if any system functions were called using the webserver:

egrep -i '(chr\(|system\()|(wget|curl|perl|gcc|chmod)%20' /usr/local/apache/logs/*

The next command searches for XSS vulnerabilities (with the added benefit of searching for positive HTTP status codes):

awk '$7 ~ /http/  {print}' /usr/local/apache/domlogs/*/access_log | awk '$9 ~ /[2-3]/ {print}'

Ensim

egrep -i '(chr\(|system\()|(wget|curl|perl|gcc|chmod)%20'/home/virtual/site*/fst/var/log/httpd/*

Plesk

egrep -i '(chr\(|system\()|(wget|curl|perl|gcc|chmod)%20' /var/www/vhosts/*/statistics/logs/*

On servers with a large number of sites, running the previous command will give you an argument list too long error. Try this instead:

for i in `ls /var/www/vhosts`; do 
  egrep -i '(chr\(|system\()|(wget|curl|perl|gcc|chmod)%20' /var/www/vhosts/$i/statistics/logs/access_*log 2/dev/null;
done;

egrep -i '(chr\(|system\()|(wget|curl|perl|gcc|chmod)%20' /var/log/httpd/*

To locate XSS vulnerabilities try:

awk '$7 ~ /http/  {print}' /var/www/vhosts/*/statistics/logs/access_*log | awk '$9 ~ /[2-3]/ {print}' 

This command searches the URI string for the text http. URIs with a protocol identifier in them often times indicate a XSS attack. However some applications such as WordPress, among others, can result in false positives. Additionally this command will only return results for requests with a positive reply code, indicating a successful request to the web server.

Reminders

Keep in mind that not all results mean the server has been compromised, it takes some interpreting. You want to look for obvious things such as calls to wget to download a file, or a call to perl that looks out of place. You may come up with some false positives so using grep to cut 404's and 400's out may be a good idea. You can do this by tacking a "| grep -v 404" on to the end of any of those commands.

Document all of your findings!

Wrap Up

Root Compromise

If you determine that an attacker has gained root access you will need to contact your sales representative, to have a replacement server built. There is no way for you to guarantee that a server will be 100% safe after a root compromise.

Ideally you should upload your sites to the new server from a local backup, however we can attempt to clean up the sites as best you can if local backups are not available.

User Compromise

If your investigation determines that the server was not compromised at the root level, then it should be safe to remove the compromised files, if any, and inform the customer of your findings, along with recommendations to prevent this issue from recurring.