The first step is to look at rkhunter and see if it reports anything that is bad. If rkhunter reports that the server has been rooted it is probably going to best to reimage the server once the method of intrusion has been determined.
Download and unzip rkhunter
tar -zxf rkhunter-1.2.1.tar.gz
Install and run it
If you are unsure of the output either look on google or you can post it here and I can try to help.
If that does not turn up anything and you are still having issues with the server sending out a DOS attack, hosting a botnet, or some other “bad” activity the next thing to check into how apache may have been used to exploit the server. This part is going to be hard for many that are not experienced with server administration because you have to be aware of what is normal and what is not. The directories that I am going to reference will have legitimate files but may also contain the scripts used to hacked or send out an attack.
Go ahead and look in /tmp first and delete the uneeded files that are normal
rm -rf sess*
rm -rf ssh-*
rm -rf dos-*
On a normal basis the files are fine, the files are only being deleted now to aid in determining what is a legitmate file. The files that are going to be of interest are anything that are bold or green because that means that they can be executed. Typically the hacked files are going to have names that use numbers in the names, end in .pl (perl files), or simply look different from everything else in /tmp. It is impossible to make a list of everything, this is going to be where experience counts as to determining if something is in fact a “bad” file. If youare unsure of the file head over to google and search the name. If it is infact some sort of a rootkit or script you will probably find references to it and what it does.
Another thing to look at is who the file is owned by. If a file is owned by the user and group nobody it was probably created by apache. These files are going to be of particular interest during an investigation.
Make sure to look at the very top for a directory named …, .tmp, or anything with a period at the begining as that means it is a hidden file, something that normally would not be in tmp. If you find nothing in /tmp also check /var/tmp and /dev/shm. These directories are where most scripts are stored but it does not mean it is the only place they may be! It might require some more investigation as to what is happening if you are unable to determine what happened from the above.
Once you find the file you can start the investigation as to what went wrong. The best bet is going to be to look in /usr/local/apache/logs and /usr/local/apache/domlogs and to use the egrep command to search the files. For instance if you have the scriptr r0nin in /tmp you will want to go ahead and check how exactly it got in.
egrep r0nin *
Chances are you are going to see something like this:
Request: XXXXXXXXX – - [30/Apr/2005:10:41:20 -0400
] “GET /index.php?t=5704&rush=%65%63%68%6F%20
68%20%5D%29.%2527′; HTTP/1.1″ 403 219
(I have XXX’ed out some information)
If you do in fact find something like the above, there are many variations so it will not look exactly like that, then you were probably hacked via a mysql injection. To further prove that this is the case you can look at the timestamp on the file and the timestamp in the logs.
One nice tool to use to look into what else might be hacked is the command lsof. This command will let you see all files are are currently being accessed by a given user. First stop apache then run lsof to see what files it is using, the reason it must first be stopped is that if you do not there will be a huge list of libs and other files listed.
service httpd stop
lsof -u nobody
Looking at the lsoof output you may be able to find some other directories that are being used. Again here is something where looking at the naming will help, if some directory looks out of place take a look at it and you may be able to find something.
If that fails you can also use the ps command to try and find the offending process and track it down. Use
To look at everything that is running. Most times it is going to be hard pretty hard to just look at everything if you are not very familiar with what is going on and spot the offending process. I would go ahead and start out looking for only perl scripts since they are popular right now:
ps -aux |grep perl
If you do find something go ahead and search the drive for the file then open it with pico, you should be able to look at the commenting in the code to find out what sort of things it is capable of. Luckily most of the script kiddies have a section at the top that describes the file and what it does.
find / |grep somesript.pl
then edit it via:
pico -w /location/of/script.pl