Detection
At September 27th, 13:37 Eastern, I received two bounced messages from my internal mail server. It was unable to send mail from my fake internal domain through Bellsouth’s SMTP server, which I use as a smarthost. Reviewing the bounce, it was immediately apparent it had a large amount of internal information about the machine that is my border router and firewall.
This message was created automatically by mail delivery software.
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
stufftempo@yahoo.com
SMTP error from remote mailer after MAIL FROM:[jasonb @internal.foo] SIZE=7821:
host mail.bellsouth.net [205.152.59.16]: 553 internal.foo does not exist
Response
I quickly logged into the box and ran a w. The attacker was still on and was running something.
rebecca:~# w 14:05:38 up 265 days, 14:47, 4 users, load average: 0.06, 0.17, 0.31 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT jasonb pts/0 rachael.internal 19Sep05 34:54 0.92s 0.89s ssh host.example.com root pts/1 acb434f0.ipt.aol 13:28 27:41 7.21s 0.09s /bin/sh ./string jasonb pts/2 faith.internal.f Fri10 0.00s 0.21s 0.06s sshd: jasonb [priv] jasonb pts/3 faith.internal.f Fri10 3days 3:28m 0.06s sshd: jasonb [priv]
I unplugged the network interface.
Forensic Analysis
Thereafter, I poked around briefly to see what was happening. The box seemed utterly owned. df segfaults and netstat no longer understands the ‘-lpn’ arguments.
I ran lsof which provided some seemingly valuable information. The box had at least one trojan running, listening on two different ports. Later examination indicated it was an SSH daemon. The occurance of ps running seemed odd. It’s likely a tool such as wget to download exploits. The target host doesn’t appear to be live now.
rebecca:~# lsof -niTCP COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME sshd 1172 root 3u IPv4 3816 TCP *:ssh (LISTEN) sshd 17930 root 4u IPv4 27312494 TCP 1.2.3.4:ssh->172.180.52.240:4030 (ESTABLISHED) smbd 18747 root 3u IPv4 27313684 TCP *:65114 (LISTEN) smbd 18910 root 3u IPv4 27314662 TCP *:exec (LISTEN) ps 19864 root 7u IPv4 27319653 TCP 1.2.3.4:4908->207.66.155.21:www (SYN_SENT)
I logged into my backup server, which did not show any obvious signs of intrusion or login by anyone other than myself, and proceeded to take a snapshot of the compromised system for forensic analysis. Ideally I’d take the compromised host offline and image the disk, but I chose to keep the host online for now.
Before rebuilding the compromised host, I have obtained some information from its current running state. Most interesting, I looked at /var/log/auth.log and /var/log/syslog. From the former, I deduced the attack vector, but more on that later. From the latter it was clear at least one trojan backdoor had been installed. It’s interesting to note that the wiping of the logs was incomplete, so a wealth of information was available, partially as a result of the incomplete installation of the rootkit.
By comparing the system’s current state with that of its last known good snapshot, I was able to determine precisely which files had been modifed and added, which provides for interesting reading. Below is the output, pruned for files contained within the rootkits themselves and later unpacked to replace some existing system binaries. (i.e. only new, evil files.)
sarah:/snapshot/owned/usr/share/.a# rsync -ne 'ssh -c arcfour' \ --delete -avH /snapshot/rebecca/rebecca-usr/latest/tree/ rebecca:/usr building file list ... done deleting bin/x.pid deleting bin/snfs deleting bin/smbd -D deleting bin/sl2 deleting bin/sense deleting bin/logclear deleting bin/initrd deleting bin/crontabs deleting bin/(swapd) deleting include/linseed.h deleting include/linpid.h deleting include/linkey.h deleting include/linconf.h deleting include/iceseed.h deleting include/icepid.h deleting include/icekey.h deleting include/iceconf.h deleting lib/libshtift/top deleting lib/libshtift/ps deleting lib/libshtift/netstat deleting lib/libshtift/ls deleting lib/libshtift/ifconfig deleting lib/libshtift/ deleting lib/libsss deleting lib/libsnf.log deleting share/.a/rk/wget.tgz deleting share/.a/rk/sysinfo deleting share/.a/rk/rk.tgz deleting share/.a/rk/pico.tgz deleting share/.a/rk/mail deleting share/.a/rk/kde.c deleting share/.a/rk/install deleting share/.a/rk/informatii deleting share/.a/rk/chattr.tgz deleting share/.a/rk/ava deleting share/.a/rk/ deleting share/.a/diav/w.tgz deleting share/.a/diav/string deleting share/.a/diav/snf.c deleting share/.a/diav/setup deleting share/.a/diav/p.tgz deleting share/.a/diav/inf deleting share/.a/diav/desc deleting share/.a/diav/ctl deleting share/.a/diav/c.tgz deleting share/.a/diav/ deleting share/.a/x deleting share/.a/w00t deleting share/.a/test deleting share/.a/sparge deleting share/.a/prt deleting share/.a/mrk deleting share/.a/mig deleting share/.a/mem deleting share/.a/inst deleting share/.a/brk deleting share/.a/b00m deleting share/.a/ sarah:/snapshot/owned/usr/share/.a# rsync -xe 'ssh -c arcfour' \ --delete -avH /snapshot/rebecca/rebecca-root/latest/tree/ rebecca:/ building file list ... done deleting bin/pico deleting dev/hpd/ deleting dev/ttyop deleting dev/ttyof deleting dev/ttyoa deleting dev/hdx2 deleting dev/hdx1 deleting etc/psdevtab
Given the output, I ran a copy of chkrootkit to see what was found.
sarah:# chkrootkit -r /snapshot/owned/ -q /bin/sh /bin/sh Checking `date'... INFECTED Checking `echo'... INFECTED Checking `ifconfig'... INFECTED Checking `ls'... INFECTED Checking `netstat'... INFECTED Checking `ps'... INFECTED Checking `top'... INFECTED /snapshot/owned/dev/ttyoa /snapshot/owned/dev/ttyop Possible Madalin rootkit installed
Some Google searching for various filename keywords and some additional reading indicates that none of the above is anything amazing or zero day. In fact, the various rootkits installed have already been analzyed at length. Brian Carrier’s analysis ofthe Honeynet Project’s Scan of the Month #29 covers many of the files listed above. Most notably, /usr/bin/top, /bin/ls, /bin/netstat, /bin/ps, and /sbin/ifconfig. (It’s unfortunate for the attacker that his version of netstat and mine sported differing arguments results in unexpected syntax errors, an obvious sign that something bad has happened.)
The string catalinum@yahoo.com exists in rk/install, so my investigation of the files in /usr/share/.a begins there. Unsurprisingly, it’s been seen before. Christophe Grenier performed an analysis for, you guessed it, Scan of the Month #29. In addition to the email, the script performs a variety of tasks including unpacking some included binaries, removing some log files, and attempting to start a SSH backdoor, which can be seen in action below.
rebecca:~# tail /var/log/syslog Sep 27 13:37:07 rebecca smbd -D[18747]: log: Server listening on port 65114. Sep 27 13:37:07 rebecca smbd -D[18747]: log: Generating 768 bit RSA key. Sep 27 13:37:07 rebecca smbd -D[18742]: log: Server listening on port 65114. Sep 27 13:37:07 rebecca smbd -D[18742]: fatal: listen: Address already in use Sep 27 13:37:10 rebecca smbd -D[18747]: log: RSA key generation complete. Sep 27 13:38:01 rebecca /USR/SBIN/CRON[18842]: (mail) CMD ( if [ -x /usr/lib/exim/exim3 -a -f /etc/exim/exim.conf ]; then /usr/lib/exim/exim3 -q ; fi)
At that point syslogd was stopped. It was not restarted by the attacker before the attacker was expelled.
The attacker was in the process of running something called ./string when he lost connectivity. It does not appear to have finished running.
The attacker downloaded a variety of toys onto the system in various tarballs, including b00m, w00t, test, and sparge containing exploits and network DoS tools.
Attack Vector
Access was obtained via SSH brute force attack against the root account. Having made its way into a password list, the root password was inevitably guessed eventually. I’m surprised it took so long.
Sep 27 13:27:25 rebecca sshd[17902]: reverse mapping checking getaddrinfo for www.saturne.com.tw failed - POSSIBLE BREAKIN ATTEMPT! Sep 27 13:27:25 rebecca sshd[17902]: error: Could not get shadow information for root Sep 27 13:27:25 rebecca sshd[17902]: Failed password for root from 211.21.45.91 port 64911 ssh2 Sep 27 13:27:28 rebecca sshd[17904]: reverse mapping checking getaddrinfo for www.saturne.com.tw failed - POSSIBLE BREAKIN ATTEMPT! Sep 27 13:27:28 rebecca sshd[17904]: error: Could not get shadow information for root Sep 27 13:27:28 rebecca sshd[17904]: Accepted password for root from 211.21.45.91 port 65018 ssh2 Sep 27 13:27:29 rebecca sshd[17906]: (pam_unix) session opened for user root by root(uid=0) Sep 27 13:27:37 rebecca sshd[17912]: reverse mapping checking getaddrinfo for www.saturne.com.tw failed - POSSIBLE BREAKIN ATTEMPT! Sep 27 13:27:37 rebecca sshd[17912]: error: Could not get shadow information for root Sep 27 13:27:37 rebecca sshd[17912]: Failed password for root from 211.21.45.91 port 61345 ssh2 Sep 27 13:28:43 rebecca sshd[17930]: Accepted keyboard-interactive/pam for root from 172.180.52.240 port 4030 ssh2 Sep 27 13:28:44 rebecca sshd[17935]: (pam_unix) session opened for user root by root(uid=0)
Above, several failed login attempts are witnessed. Once a hit occurs, a single successful login occurs from a different box at which point the attacker begins playing with his toys.
Recovery
Having a last known good snapshot available from the previous night, I simply reverted the box to its previous state and disabled root access via SSH. I have not encountered any odd behavior since.