robots.txt

A standard robots.txt file is automatically delivered in both production and dev/stage/test environments, if no robots.txt file exists in the public folder.

robots.txt in dev/stage and test environments

In testing environments a robots.txt file is delivered to prevent robots from indexing the testing domain. The robots.txt file urges all robots to not index anything from the domain.

robots.txt for production environments

In production environments, a standard robots.txt file is delivered if the file does not exist. 

This standard robots.txt includes a single policy that tells robots to not index faster than one page per second. 

Dynamic delivery of robots.txt

If our standard robots.txt is in the way for your application, you can work around this by adding this rule, after your standard rewrite to index.php.

RewriteCond %{REQUEST_URI}  ^/robots.txt$
RewriteRule ^ index.php [L]

An example of this in action for Drupal:

The same method works for WordPress, add the two rewrite lines for robots.txt after your standard WordPress .htaccess block.

Give us your feedback on this article