How to Identify and Fix 404 Errors

A 404 error, also known as ‘HTTP 404,’ ‘404 Not Found,’ or simply ‘Page Not Found,’ is an HTTP response code indicating the server’s inability to locate the requested resource. This can mean that the page or file the user is trying to access has been moved or deleted, or the URL has been entered incorrectly. This guide will explore ways of identifying and fixing 404 errors.

Why Addressing 404 Errors Is Important

While 404 errors are standard on the web, they can significantly impact user experience, SEO rankings, and performance, making it crucial to address them effectively. In WordPress, these errors often arise due to misconfigured permalinks, missing pages or broken links, but they can be fixed with the proper troubleshooting steps.

Improved Speed

Any request to the server will add load to your website, primarily when the request is delivered with an error.

Improved Scalability

Identifying and fixing 404 errors will contribute positively to the scalability of your website, as it will improve the end-user experience.

Reducing Dynamic Requests

Websites with a high number of 404 errors may experience an increased number of Dynamic Requests. One of the most common reasons for this is bots crawling the site (for example, Google, Blingbot, Twitterbot, etc.).

Ways of Identifying 404s

As mentioned, identifying 404 errors is essential for maintaining a healthy website. Various methods can be used to do so, depending on your setup and needs. Below, we’ll cover how to find 404s in your browser and by analysing the AccessLog.

In Your Browser

The Chrome Developers Tool is an excellent way to spot 404 errors. In your browser’s menu bar, navigate to View → Developer → Developer Tool to access the tool. After refreshing the page, you will see all requests in the ‘Name’ column, with their respective HTTP response codes in the ‘Status’ column.

In the AccessLog

The AccessLog will give you more information on which requests are throwing 404s. Here is a list of useful commands for checking your 404s in the terminal.

Continuously print 404s from the AccessLog to the console in real-time:
tail -f ~/logs/http/AccessLog | grep " 404 "

Print 404s from the current day’s AccessLog to the console:
cat ~/logs/http/AccessLog | grep “ 404 “

Count and print the total number of 404s from the last 6 days:
zcat ~/logs/http/AccessLog-* | grep " 404 " | wc -l

Dealing with 404 Errors

404 errors can frustrate users and negatively impact SEO, but luckily, there are multiple ways of addressing them. By doing so, you’re keeping your website functioning and user-friendly. Whether the issue you’re facing originates from broken links, outdated URLs or missing pages, these methods can be used to approach them.

Redirecting 404 Pages

There are several ways to handle redirects, namely plugins and manually editing your .htaccess file. You can find more information on the available redirect options in our Admin Panel here. 

Update WordPress Permalinks

If you’re experiencing issues related to broken or outdated rewrite rules, updating or re-saving your permalink settings. That can be done in your WordPress Dashboard by following these steps: 

1. Log in to WordPress Dashboard.
2. Navigate to Settings → Permalinks and select a different permalink structure than you currently use.
3. Save the changes.
4. Select the original permalink structure again and save the change once more. If you are using a custom permalink structure, make sure to change it back to the same structure.

This will make WordPress regenerate the rewrite rules, ensuring your URLs are appropriately structured. 

Creating Empty Files

In some cases, 404 errors can be caused by requests attempting to access missing or incorrectly referenced pages and files. A simple way to resolve these errors is to create empty files for the missing resources. This prevents WordPress from logging unnecessary 404 errors, improving site performance. The process can look something like this:

1. Identify and locate the files or resources causing 404s. For more information on viewing and examining log files, see this guide.
2. To access your environment’s file structure, use a file manager (such as FileZilla) or connect to it via SSH.
3. Navigate to the directory where the missing resources should be located, or ensure you are targeting the correct path. Create empty files with the same name as the missing ones.

Control Bot Access with robots.txt

If you see 404 errors related to bots crawling the website, you can consider setting up instructions to block specific paths in a robots.txt file and placing it in your site root directory. All Servebolt environments have a standard robots.txt file in place that is delivered automatically. Please note that it’s up to the individual bots and crawlers to obey these instructions.

Example

In this example of a robots.txt file that targets all web crawlers (Googlebot, Bingbot, etc.), instructing them to wait 1 second between requests. Furthermore, it blocks access to all URLs in the /wp-admin/-directory except for the file /wp-admin/admin.ajax.php.

User-agent: *
Crawl-delay: 1
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Remember to submit an updated version of your robots.txt to Google whenever it’s updated.

Continue the Improvements

In addition to identifying and fixing 404 errors, there are multiple steps to continue optimizing and improving your website(s). Whether you’re boosting performance or troubleshooting WordPress issues, our Help Center offers a variety of guides and resources to help you along the way. Below is a list of related resources on improving and optimizing your website(s).

• Optimization for Faster Performance and Increased Scalability
• How to Improve Front-end Performance With the Help of Dequeuing
• How to Lower Your Dynamic Requests Usage

---

As always, should you have any additional questions, please don’t hesitate to contact us through our support chat at servebolt.com!