Remote Shell Trojan Infection on Virtuozzo VPS

I have recently noticed  a Remote Shell Trojan Infection on Virtuozzo VPS where executable binaries in /bin folder were getting infected by some kind of Trojan. This must have happened due to a server compromise that would end up in most of the commands in /bin folder which will show the following message while executing.

root@myserver:/# hostname -i
-bash: /bin/hostname: cannot execute binary file
root@myserver:/# ps
-bash: /bin/ps: cannot execute binary file
root@myserver:/# mv
-bash: /bin/mv: cannot execute binary file
root@myserver:/#

Generally this error message means Linux Kernel doesn’t recognize the file as a shell script or as an executable file.  At first, I thought that its just a file permission issue because in some cases, customer may mess up the permission with the system files for no good reason and update us that “hey i need  to get this thing fixed”  😉

So as a first work around, obviously I checked the permission of the binaries and I found there were no issues with them!

root@myserver:/#
-rwxr-xr-x 1 root root   12360 Feb 22  2012 hostname
-r-xr-xr-x 1 root root   81120 Jan  9 00:43 ps
-rwxr-xr-x 1 root root   78008 Mar 21  2012 mv
root@myserver:/#

This led me to a conclusion that these binaries have been corrupted for some reason. I was so keen to find the reason  and of course the fix for this.

The worst thing I found during my investigation is that, if we initiate a system reboot, the server won’t come back online because in Virtuozzo environment, during start up of each VPS, there are some scripts that will modify files like /etc/hosts /etc/resolv.conf on every boot; to make sure it always have working, resolving nameservers, hostname etc. Here in my scenario, it is not possible for root user in VPS to do as such, since the action uses execution of binaries inside /bin directory of corresponding VPS. So at the back end, a Virtuozzo Administrator can see the following entries in the  log when starting the container.

2014-12-11T12:11:43-0500 vzctl : Container 808080 : mv: cannot move `/etc/resolv.conf.14′ to `/etc/resolv.conf’: cannot execute binary file
2014-12-11T12:11:43-0500 vzctl : Container 808080 :  ERROR: Can’t change file /etc/resolv.conf
2014-12-11T12:11:43-0500 vzctl : Container 808080 : File resolv.conf was modified
2014-12-11T12:11:44-0500 vzctl : Container 808080 : Container is unmounted
2014-12-11T12:11:44-0500 vzctl : Container 808080 : Failed to start the Container

As we see, VPS start failed due to corrupted executable binary file /bin/mv

2014-12-11T12:11:43-0500 vzctl : Container 808080 : mv: cannot move `/etc/resolv.conf.14′ to `/etc/resolv.conf’: cannot execute binary file

But if we don’t touch, VPS will work as normal, unless we do a reboot.

The reason is that most of the executable binaries in /bin directory  have been infected by Remote Shell Trojan. A security incident regarding this behavior is posted here http://seclists.org/incidents/2002/Jan/73

What it really does is, it initiates a virus-like self replication process that infects the executable binaries in /bin directory. The infection process is like when an infected binary is executed, the code “Shell Trojan code” will be executed by issuing the HTTP GET request “GET/~telcom69/gov.php HTTP/1.0″ to port 80 inside the box which turns the infected machine to a network sniffer.

To confirm, we would use the “strings” command on the infected binary files.

root@myserver:/# hostname -i
-bash: /bin/hostname: cannot execute binary file
root@myserver:/# strings /bin/hostname | grep telcom69/gov.php
GET /~telcom69/gov.php HTTP/1.0
GET /~telcom69/gov.php HTTP/1.0
GET /~telcom69/gov.php HTTP/1.0
GET /~telcom69/gov.php HTTP/1.0
root@myserver:/# ps
-bash: /bin/ps: cannot execute binary file
root@myserver:/# strings /bin/ps | grep telcom69/gov.php
GET /~telcom69/gov.php HTTP/1.0
GET /~telcom69/gov.php HTTP/1.0
GET /~telcom69/gov.php HTTP/1.0
GET /~telcom69/gov.php HTTP/1.0

So the container is infected and  have confirmed the presence of Remote Shell Trojan. Now the important thing, how can we overcome the situation here ?

Well, the solution that I found is to copy the /bin directory from a working VPS. This is somehow a pretty easy task if you have access to Physical host (otherwise I recommend that you keep coming here, because I am keeping the secret for another article 😉 ).

So if you have Virtuozzo node access, you can do the following.

Solutions:

1) Copy directories from a working VPS:

cp -pR /vz/private/VEID1/fs/root/bin /vz/private/VEID/fs/root/
cp -pR /vz/private/VEID1/fs/root/usr/bin /vz/private/VEID/fs/root/usr/

where,
VEID1 is the working VPS
VEID is the VPS having problems.

2) Copy directories from the OS template. The OS templates are stored under /vz/template/cache directory.

a) Extract the corresponding template file the VPS is using

# cd /vz/template/cache/
# tar -zxf os-templatename.tar.gz

b) Copy the directories to the VPS private area

cp -pR /vz/template/cache/root/bin /vz/private/VEID/fs/root/
cp -pR /vz/template/cache/root/usr/bin /vz/private/VEID/fs/root/usr/

This fixes the issue with /bin directory and we will be able to run the binaries as usual but we should keep fingers crossed since we’re running it again at our own risk.  Besides, it is strongly recommended that anyone running Linux should run any rootkit/virus detector to see if the container is still infected. I’d rather reinstall the container once I have the backups to make sure that any presence of this trojan is not there on the server.

As I close in to the end of this article, I do not think I need to specifically remind anyone that you should implement a good security measurement all the time on our servers as its a mandatory thing that every System Admins must do to avoid getting into hot water.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts