Usually, when someone requests a page that doesnâ€™t exist, a server will return aÂ 404 (not found) error. This HTTP response code clearly tells both browsers and search engines that the page doesnâ€™t exist. As a result, the content of the page (if any) wonâ€™t be crawled or indexed by search engines.
In addition to returning a 404 code in response to a request for a page that doesnâ€™t exist, the server will also display a 404 page. This may be a standard “File Not Found” message, or it could be a custom page designed to provide the user with additional information.Â The content of the page is entirely unrelated to the HTTP response returned by the server. Just because a page displays a 404 File Not Found message doesnâ€™t mean that itâ€™s a 404 page. It’s like a giraffe wearing a name tag that says “dog.” Just because it says it’s a dog, doesn’t mean it’s actually a dog. Similarly, just because a page says 404, doesn’t mean it’s returning a 404. You can useÂ Fetch as Googlebot (or other tools available on the web) to verify whether the URL is actually returning the correct code.
Returning a code other than 404 or 410 for a non-existent page (or redirecting users to another page, such as the homepage, instead of returning a 404) can be problematic. Firstly, it tells search engines that thereâ€™s a real page at that URL. As a result, that URL may be crawled and its content indexed. Because of the time Googlebot spends on non-existent pages, your unique URLs may not be discovered as quickly or visited as frequently and your siteâ€™s crawl coverage may be impacted (also, you probably donâ€™t want your site to rank well for the search query [File not found]).
We recommend that you always return a 404 (Not found) or a 410 (Gone) response code in response to a request for a non-existing page. You can improve the user experience by configuring your site to display aÂ custom 404 page when returning a 404 response code. For example, you could create a page containing a list of your most popular pages, or a link to your home page, or a feedback link. You can also use the Webmaster ToolsÂ Custom 404 widgetto add a search box and more site search options to your site. But itâ€™s important to remember that itâ€™s not enough to just create a page that displays a 404 message. You also need to return the correct 404 or 410 HTTP response code.