I. AWStats
1. Log onto Cpanel
2. Under 'Stats', click AwStats Stats
3. From the menu located on the left side, click Full List (under Robots/Spiders visitors)
This will tell you what spiders have looked at your site. This is very generic information (i.e. no specific pages, no specifics of the actual spider, etc.). Also note, 'Hits' does not refer to the number of pages. One page generally involves many 'hits' (each graphic for instance counts as a hit). The amount of bandwidth gives you an idea of how much was 'taken' but again, be careful. What if the spider visited the same page multiple times (which is quite frequent). Again, I warn that this information is very generic.
II. Raw Logs - the 'Real Deal'
1. Log onto CPanel
2. Under FTP Section, click on Manage Accounts
At the bottom of the page, you will find how to download your logs to your PC.
Once you have your logs downloaded, you can view them using the built in WordPad utility that comes with Windows 98/ME/2000/XP and search for 'googlebot/2.1'. This will show you all the actual pages that Google visited. A sample would look like this:
64.68.82.32 - - [03/Jun/2003:21:15:53 -0400] "GET /dried_flowers-whl.htm HTTP/1.0" 301 342 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
Comments:
- The IP address (64.68.*) indicates this was a fresh crawl.
- You can see the date of the crawl (June 3)
- The file crawled (dried_flowers-whl.htm)
- Note: the 301 means that this particular page had a 301 redirect
Further comment:
Logs are reset at the end of each month, so the logs of today began on May 31.
