Using the Access Log to Check if Content Crawled by Google

Two weeks ago I tried to retrieve my access log for this very site, from my host https://wpmudev.com/. I don’t have a lot of experience with WPMU’s hosting, I have mostly worked with WP Engine the last 5 years or so. But I have had an account with WPMU for years, in order to use some of their plugins. I get a limited amount of hosting included with that account, so I have put my personal site on here.

I had some new content that wasn’t showing up in Google Search Console (GSC), but I have head a lot of people have had content indexed but not showing up in GSC. This is a new site that doesn’t have a lot of content or backlinks yet. Plus I am still working through some site structure stuff. It may be that Google just hasn’t gotten around to crawling the new content yet, they may not think it’s worth indexing (new site, still work in progress etc), or it may be that GSC is not telling the whole story. I figured this would be the perfect time to break out the Screaming Frog Log File Analyser and see if Google was crawling my site more recently then GSC would indicate.

WPMU Access Logs Not Working

Unfortunately, I found that I was getting an internal server error when I tried to download my access log. But it was a weekend, so I shelfed it and came back Monday to tried again. Still no luck. I then submitted a ticket with WPMU support and received the following message:

Knowing that the fix was WIP, I figured it would be back up in a day or two. Access logs are pretty standard right? Well it’s been two weeks and I was starting to think that I would have to look for other options for hosting if I wanted to have an access log.

The Workaround to Download WPMU Access Log File

Then I vaguely recalled that I had seen the logs while using SFTP on the site before. Sure enough, if you SFTP into the root of your WPMU hosted site, and go to site > logs you will find your access log, as well as the PHP Errors, PHP Slow, and Web Application Firewall (WAF) logs. That is how you can download the WPMU Access Log file.

Summary

I’m a little annoyed with myself that it took me this long to find the log (and why didn’t support mention I could find it there?!). But I usually spend more time in FileZilla (SFTP client) then I do in my WPMU dashboard anyway. I would just as soon download it this way then though the dashboard.

For the record, I did find some areas for improvement after reviewing my log file.