Changed permalink settings, now have url errors

[jmk909er]

I checked my Google webmaster and found many errors. I think that it happened because I changed my permalink settings from showing, to not showing the date based on some seo advice. I am getting a warning that "Some important page is blocked by robots.txt."

It lists 13 crawl errors, most of them with a date in them. Do you know how to fix this?

Thanks, Joe

 

[jcorkern]

Sometimes what GWT shows as "crawl errors" aren't actually "errors" at all. Sometimes all they're doing is letting you know that a page you blocked them from crawling was successfully blocked; the "error" doesn't always mean that something is wrong. Did you want these pages that they're listing blocked so Google can't crawl them? It's only an error if you want Google crawling those pages.

What pages are blocked? And did you block these pages intentionally? If so, you can more than likely ignore the crawl error. Give me some more details; maybe I can help. If not, I'm sure someone else here can.

 

[djbaxter]

With version 3.x, WordPress claims to redirect changed Permalinks automatically - see http://wordpress.org/support/topic/...llowing-change-to-permalinks-is-it-301-or-302

There are several plugins you can install if you don't trust this, e.g., http://wordpress.org/extend/plugins/redirection/

But this isn't about your permalinks:

I am getting a warning that "Some important page is blocked by robots.txt."

Your robots.txt file is the issue. Have a look and see if anything is being "Disallowed" unintentionally, or you can post it here if you like.

 

[jmk909er]

I went back to GWT this morning and it has miraculously all cleared up! There are no longer health error warnings. now when I check crawl errors it says "No errors detected in the last 90 days. Nice!"

Another thing that has showed up is now in GWT it is showing my website twice, one with www and one without. This is also something I changed based on seo advice. Should I delete the one I am not using out of GWT?

Just a shout out to you guys, since I am using wordpress I do a lot of posts in the wordpress forums. Many times I never get a reply but when I post here I almost always get a reply. I really appreciate you all thanks!

 

[djbaxter]

Another thing that has showed up is now in GWT it is showing my website twice, one with www and one without. This is also something I changed based on seo advice. Should I delete the one I am not using out of GWT?

Not necessarily if you're still getting stats for both.

Are you already using a 301 redirect to the version you prefer?

 

[jmk909er]

Not sure, per the seo recommendations I went into GWT and changed the preferred domain to be minus the "www" and I changed it in the general settings of wordpress also, so maybe GWT took care of this.

When I go to Google analytics it only shows the one website without the "www" as the default and not the other one at all. So I am only getting stats for the one.

When I search in google and click on one of my posts it still is coming back as page not found, but it puts me on my website with a good link to the post. Maybe I can't do anything about it?

 

[djbaxter]

I know you have a robots.txt file but do you have an .htaccess file in your blog root directory?

 

[jmk909er]

Yes I do, I don't know anything about this. I opened it up and here is what is in there"

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

 

[djbaxter]

Got to leave soon but I'll come back later and give you some things to try to redirect requests for the www version to the non www version.

 

[djbaxter]

Okay.

Edit your .htaccess file so that it reads as follows:

# redirect to non www version
Options +FollowSymLinks
RewriteEngine On 
RewriteBase /
RewriteCond %{HTTP_HOST} !^www\. 
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=301,L]

# BEGIN WordPress
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
# END WordPress

 

[jmk909er]

Hi minstrel I think the www issue is cleared up. I am getting a little confused, I no longer have any errors but I just noticed that in GWT under the new non-www site that it says no site map has been generated so it must still be looking at the site map for the www site which is still there and has the www in them. That is probably why google search is returning results for sites with www and dates still in the permalink and causing a "page not found"

I think I will delete the www site from GWT and the current site map file from the root directory and submit a new one. What do you think?

 

[djbaxter]

Hi minstrel I think the www issue is cleared up. I am getting a little confused, I no longer have any errors but I just noticed that in GWT under the new non-www site that it says no site map has been generated so it must still be looking at the site map for the www site which is still there and has the www in them. That is probably why google search is returning results for sites with www and dates still in the permalink and causing a "page not found"

That's why you should add those lines to .htaccess. If they are still in Google's results, searchers may still click on the www version of the links and you do NOT want them getting 404 errors. That is lost traffic, lost potential sales, and unnecessary.

You might also seriously consider adding a custom error page to your WordPress site, such as WordPress › 404 Redirected ? WordPress Plugins - this is a way to keep visitors on your site and help them find what they're looking for. (For a non-WordPress version, see Sorry! Page not found... | Dr. David J. Baxter - Ottawa Psychologist which has the essential elements of a good 404 page).

Then, add the following line to near the top of your .htaccess file to let Apache know how to find the error page:

ErrorDocument 404 /{name of error page}

While you're at it, add the following line to the very top pf .htaccess for security - this prevents anyone from viewing your directory and file structure:

Options All -Indexes

I think I will delete the www site from GWT and the current site map file from the root directory and submit a new one. What do you think?

Sounds good to me.

 

[jmk909er]

Hey minstrel I added the .htaccess but it did not work, when I tried to open the link from google search I got an error and then I tried to login to my wordpress dashboard and got the same error and it would not let me in. I attached a screenshot of the error.

I reverted back to the original file and now it is working to log in to wordpress. To the best of my knowledge I only have 1 or maybe 2 posts that have the original date in url issue since my site is so new.

At least right now when I get the error it will put you on my site and suggests the right link to the post. I deleted the old site and sitemap and uploaded a new sitemap for the current site. I am not receiving any errors in GWT, maybe I should just leave it alone and give it time to crawl the new site/sitemap??? Google is returning resultes for all my pages that are correct and loading.

What do you think?

 

[djbaxter]

OK. The leave it without the www -> non-www redirects but add in the custom 404 page I described above for human visitors.

 

[jmk909er]

OK minstrel, thanks so much for your help!