Recover deleted file using “lsof”

Whats your favorite command in linux ? 🙂 . I know its hard to say, so as I’m. But one of my favorite command is “lsof“. It can be used in many ways like listing application’s opened files, dumps info about every opened files ( like a snapshot of all process details ), pretty handy to locate the suspicious process as such. But among them, one of coolest thing I have noticed with lsof is to recover a deleted file. You may already know, if not, Yes! its possible to recover a deleted file using lsof.

Let me say a scenario, where someone was trying to grab the access log details of the apache, same time other person has just deleted it, probably a hacker or a dump techie 😉 , just kidding but yeah we do mistakes, this happens to everyone. I’ve done many times :-P. What can we do if log file was deleted, how can we get information that we were looking ? this is where lsof comes in to play in recovering the deleted file.

To know where to go, you need to get the PID of the process that has the file open, then FD handle ( file descriptor ) and locate it in /proc folder. Once you get the info from lsof, you can copy the file out of it and can recover.

I will demonstrate with an example,

For instance, Lets assume that you are running apache on your box, and its access log is stored in /var/log/httpd/access_log

[root@webhost01 ~]# ps aux |grep httpd
root 22115 0.0 3.5 308620 18840 ? Ss 03:53 0:00 /usr/sbin/httpd
apache 22117 0.0 1.2 242184 6588 ? S 03:53 0:00 /usr/sbin/httpd
root 22534 0.0 0.1 7208 804 pts/0 R+ 04:11 0:00 grep httpd
[root@webhost01 ~]# lsof -p 22117 | grep /var/log/httpd/access_log
httpd 22117 apache 18w REG 144,43 1350 155976448 /var/log/httpd/access_log
[root@webhost01 ~]#

We could see that apache process is running with PID 22117 and is accessing the file /var/log/httpd/access_log. So here, if process is running and someone has deleted the file /var/log/httpd/access_log accidentally, its not actually deleted from the system, linux kernel always keeps a copy of data in /proc

Lets say I have following info in my apache access_log

[root@webhost01 ~]# cat /var/log/httpd/access_log
130.255.190.221 – – [09/Jun/2014:04:19:21 -0400] “GET /phppath/php HTTP/1.0″ 404 266 “-” “-”
46.137.141.224 – – [09/Jun/2014:09:21:51 -0400] “GET / HTTP/1.1″ 200 62236 “-” “Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK – spider 4.1)”
198.20.69.74 – – [09/Jun/2014:12:48:18 -0400] “GET / HTTP/1.1″ 200 61598 “-” “-”
87.106.167.12 – – [09/Jun/2014:12:49:56 -0400] “GET //phppath/php HTTP/1.1″ 404 273 “-” “Mozilla/5.0 (Windows NT 6.1)”
85.17.59.114 – – [09/Jun/2014:13:45:54 -0400] “GET / HTTP/1.1″ 200 62011 “-” “Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK – spider 3.0)”
87.230.57.102 – – [10/Jun/2014:04:56:37 -0400] “GET /phppath/php HTTP/1.0″ 404 266 “-” “-”
54.216.121.76 – – [10/Jun/2014:07:31:19 -0400] “GET / HTTP/1.1″ 200 62816 “-” “Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK – spider 3.0)”
46.105.77.3 – – [10/Jun/2014:07:33:44 -0400] “GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1″ 400 283 “-” “-”
41.190.91.212 – – [10/Jun/2014:08:39:52 -0400] “HEAD / HTTP/1.0″ 200 – “-” “-”
205.207.165.210 – – [10/Jun/2014:09:24:06 -0400] “GET /phppath/php HTTP/1.0″ 404 266 “-” “-”
218.75.108.204 – – [11/Jun/2014:03:14:23 -0400] “HEAD /manager/status HTTP/1.1″ 404 – “-” “Java/1.6.0_02″
[root@webhost01 ~]#

and apache process with PID 22117 is accessing it.

root@webhost01 ~]# lsof -p 22117 | grep /var/log/httpd/access_log
httpd 22117 apache 18w REG 144,43 1350 155976448 /var/log/httpd/access_log
[root@webhost01 ~]#

Now just do the worst thing, yeah delete the file.

[root@webhost01 ~]# rm -f /var/log/httpd/access_log
[root@webhost01 ~]#
[root@webhost01 ~]# cat /var/log/httpd/access_log
cat: /var/log/httpd/access_log: No such file or directory
[root@webhost01 ~]#

Now we have no apache log available in our system but we know the process with PID 22117 is still using it. Do the lsof again to see it.

[root@webhost01 ~]# lsof -p 22117 | grep /var/log/httpd/access_log
httpd 22117 apache 18w REG 144,43 1350 155976448 (deleted) /var/log/httpd/access_log
[root@webhost01 ~]#

We see that it was deleted but process with PID 22117 is still using it.

Now recover the deleted file. There is always have a place to run to, always, the place is /proc ( got this word from actual quote ” There is always have a place to run to, always, the Place is God 🙂 )

As mentioned earlier, kernel keeps a copy of the file in /proc, we can get it from /proc/PID/fd/file descriptor. So here, our PID is 22117 and file descriptor is 18.

[root@webhost01 ~]# ls -l /proc/22117/fd/18
l-wx—— 1 root root 64 Jun 11 04:11 /proc/22117/fd/18 -> (deleted)/var/log/httpd/access_log
[root@webhost01 ~]#

Now copy the file out from /proc to some location in the disk, eg /root

[root@webhost01 ~]# cp /proc/22117/fd/18 access_log
[root@webhost01 ~]#
[root@webhost01 ~]# ls -l access_log
-rw-r–r– 1 root root 1350 Jun 11 06:26 access_log
[root@webhost01 ~]#

Now we have the log file back in our system. Now, lets verify it.

[root@webhost01 ~]# pwd
/root
[root@webhost01 ~]# cat access_log
130.255.190.221 – – [09/Jun/2014:04:19:21 -0400] “GET /phppath/php HTTP/1.0″ 404 266 “-” “-”
46.137.141.224 – – [09/Jun/2014:09:21:51 -0400] “GET / HTTP/1.1″ 200 62236 “-” “Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK – spider 4.1)”
198.20.69.74 – – [09/Jun/2014:12:48:18 -0400] “GET / HTTP/1.1″ 200 61598 “-” “-”
87.106.167.12 – – [09/Jun/2014:12:49:56 -0400] “GET //phppath/php HTTP/1.1″ 404 273 “-” “Mozilla/5.0 (Windows NT 6.1)”
85.17.59.114 – – [09/Jun/2014:13:45:54 -0400] “GET / HTTP/1.1″ 200 62011 “-” “Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK – spider 3.0)”
87.230.57.102 – – [10/Jun/2014:04:56:37 -0400] “GET /phppath/php HTTP/1.0″ 404 266 “-” “-”
54.216.121.76 – – [10/Jun/2014:07:31:19 -0400] “GET / HTTP/1.1″ 200 62816 “-” “Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK – spider 3.0)”
46.105.77.3 – – [10/Jun/2014:07:33:44 -0400] “GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1″ 400 283 “-” “-”
41.190.91.212 – – [10/Jun/2014:08:39:52 -0400] “HEAD / HTTP/1.0″ 200 – “-” “-”
205.207.165.210 – – [10/Jun/2014:09:24:06 -0400] “GET /phppath/php HTTP/1.0″ 404 266 “-” “-”
218.75.108.204 – – [11/Jun/2014:03:14:23 -0400] “HEAD /manager/status HTTP/1.1″ 404 – “-” “Java/1.6.0_02″
[root@webhost01 ~]#

Aha!.. we got our log file back. We can append or insert this entry to the exact access_log file later since this deleted log file is usually recreated during next apache restart. One important thing we should taken care of, even though the file was deleted but the process still accessing it, should not be killed at all ( do not do this while recovering ) otherwise the corresponding entry in /proc will be removed and process of recovering the deleted file will be troublesome.

I’m learning what other interesting stuffs can be done using lsof and yeah its one of my favorite command.

Happy File Recovery.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts