HTTP 410 Gone Code Stops Search Engines Re-Crawling Dead Urls

Here is how to use HTTP 410 GONE status code to tell search engines those 404 urls they keep re-crawling over and over again have been permanently removed and will never be back.

If your website or blog is quite old, chances are you have removed posts, added and removed categories, tags, and media attachments over the years.

Google crawls your whole website. This includes your JavaScript, CSS files, Plugins, on top of your posts, pages, tags, categories, etc, etc, etc. Jokers redirecting a url to your domain also redirects search and other bots searching for that domains resources on yours racking up more 404s.

One of my older blogs has been up since 2007 and has had numerous code updates, themes, plugins, etc. And also went from http to https encryption, and was a nightmare of old dead urls that Google and Bing kept re-crawling over and over again.

While as a general rule 404 urls will not hurt your search rank, all this crawling over and over again looking for material that was removed long ago never to return sucks up server resources. It’s best to redirect those 404s to 410 Gone http status code. This signals bots the url they were requesting is permanently gone and to remove it from their re-crawl list.

For WordPress sites this plugin works great: 410 for WordPress.

This plugin will issue a HTTP 410 response to requests for articles that no longer exist on your blog. When you delete a post or page, it records the URL for that page and issues a 410 response when that URL is requested. You can also manually manage the list of obsolete URLs.

The HTTP Specification defines the 410 response header for use when a resource has been permanently removed. It informs robots visiting your site that the requested content has gone, and that they should stop trying to access it.

This plugin will record any crawled 404 urls in a list that can be easily added to it’s 410 url database (see below.) I have found some JavaScript and CSS links are added to the WordPress 410 database as “IS 404” represented by a “1” in the far right row. Browsing your database and double clicking on 1 and changing that to a 0 sends the proper 410 gone http status code.

For static html websites this process can be accomplished by adding 410 Gone entries to your .htaccess file. An .htaccess file if configured incorrectly will crash your website hard! Be sure to download it before performing any mods. If you muck up and can’t figure out how, just re-upload the saved copy.

Assuming your website is running on an Apache http server, and mod_rewrite is installed, this works the best for me.

This entry will Wildcard 410 status anything in a tilde directory.
RedirectMatch 410 /~/(.*)

This entry will wildcard 410 status anything in this directory.
RedirectMatch 410 /foo/(.*)

These entries will 410 status an old unused feed url.
RedirectMatch 410 /feed/
RedirectMatch 410 /comments/feed/

Other advanced options can be found by searching Google.

Have questions? Comment below 😉