Chris Mendez in DevOps, For Developers, Wordpress, AWS, EC2, LightSail

AWS: Modifying Robots.txt within a Bitnami EC2 instance

Bitnami offers Wordpress for AWS Cloud which is great for developers whom are not interested in focusing on DevOps topics such as installing php, Apache and Wordpress.

The Bitnami package does it all but there are a few things you'll need to modify including robots.txt files.

In this blog entry, I will show you how to modify your robots.txt file for a Wordpress staging server. The goal will be to create a document that prevents search engines from showing your staging server on their search results.


Finding robots.txt

When you create a Bitnami Wordpress site, by default it comes with a /robots.txt file but it's not easy to find. If you're interested in understanding its path, run a find command with grep.

sudo find / | grep robots.txt  

Modifying robots.txt

Although the we were able to find a working copy of robots.txt above, this is not where we should modify the file. Instead, we should create a new robots.txt the "Bitnami Way".

Depending on how you intend to install Wordpress, there are two possible places to add a robots.txt.

Option 1 (recommended)

If you access your Wordpress website as www.yourdomain.com/robots.txt then place the file here:

/opt/bitnami/apps/wordpress/htdocs

Option 2

If you access your Wordpress website as www.yourdomain.com/wordpress/robots.txt then place the file here:

/opt/bitnami/apache2/htdocs

Optimal Solution

I absolutely love the solution presented by Henning Koch. It's smart without running the risk to forget deleting the file once we go live.

Step 1 - Modify Robots.txt file

Create a new file titled robots.exclude.txt.

nano /opt/bitnami/apps/wordpress/htdocs/robots.exclude.txt  

Within this new file, disallow everything.

# This file is returned for /robots.txt on staging servers
User-agent: *  
Disallow: /  

The allows us to create a file specifically for a staging server without touching the live version of robots.txt.

Step 2 - Modify Apache VHosts

The next step is to write a configuration that says, "If this website is staging or development, serve robots.exclude.txt instead of robots.txt.

This special rule is best written within the Apache config file. Specifically, the virtual hosts (VHosts) file.

Open the Apache VHosts config within the Bitnami package. To keep things simple, I will use nano editor.

nano /opt/bitnami/apache2/conf/bitnami/bitnami-apps-prefix.conf  

I'm choosing to paste this rule within bitnami-apps-prefix.conf.

# Bitnami applications installed in a prefix URL

# Custom Robots.txt data here
RewriteEngine On  
RewriteCond %{HTTP_HOST} ^subdomain.domain.com$  
RewriteRule ^/robots.txt$ /robots.exclude.txt

...

Step 3 - Restart Apache

sudo /opt/bitnami/ctlscript.sh restart apache  

Troubleshooting

Double checking your VHosts

If you want to double check the paths and better understand how things are connected, visit bitnami.conf.

nano /opt/bitnami/apache2/conf/bitnami/bitnami.conf  

Resources