Web Application Testing – Parsing Directory and File Listing

During web application testing, it is useful to get the directory and file listing of the root of the web application that you are testing so as to ensure complete coverage of the application.

You can use the below command to get a files and directories listing of the web application root

ls –laR /var/www > cd–filelist.txt

I wrote a simple script to parse and convert the output so that I can pipe the URLs directly to Burpsuite.

 

The script can be found at https://github.com/milo2012/pentest_scripts/blob/master/web/parseFileList.py

 

Below is an example of how you can use the script.

python parseFileList.py -f cd-filelist.txt > filelist_out.txt

After running the command, you must modify the filelist_out.txt to search/replace each lines with the FQDN of the website.

E.g. replace /var/www/html/www.domain.com with https://www.domain.com

Next, start Burpsuite and point the proxy listener to 127.0.0.1 port 8080.

The next line will use send each URLs in teh filelist_out.txt to Burpsuite using Curl and Xargs.

cat filelist_out.txt | xargs curl –user-agent “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36” -k -x http://localhost:8080 >/dev/null 2>&1

 

Sit back and enjoy some coffee as this process could take some time.