Start Your Blog Technology

How to Create Dynamic Robots.txt File for Subdomains using .htaccess Redirect

Disclosure: When you purchase a service or a product through our links, we sometimes earn a commission.

What is a robots.txt file? Most websites have one of these very simple file called “robots.txt” on the main directory of their server. The robots.txt file has been around for almost two decades, and it is now a standardized way of communicating what pages search engine bots (or crawlers) should and should not visit.

dynamic-robots-txt-techblogcorner

What about if your robots.txt file adjusted itself automatically based on whether it’s the test site or the live site?

Below is the code for generating dynamic file for your main domain site along with sub-domains :-

RewriteEngine on
RewriteCond %{HTTP_HOST} ^subdomain.website.com$
RewriteRule ^robotx\.txt$ robots-subdomain.txt
Then add:
User-agent: *
Disallow: /
to the robots-subdomain.txt file

Save

Don't miss out!
Learn How to 10X Your Blog Traffic
Invalid email address
Give it a try. You can unsubscribe at any time.

About the author

Rahul Setia

Rahul Setia was born and raised in the Kaithal, Haryana. He worked at brands like Jabong, ProProfs etc. He was also in the List of Top 100 Social Media Influencer's 2019 by Status Brew. He lives in Delhi/NCR and is a Digital Gig & Founder of Websites i.e TechBlogCorner.com, ViralMasalla.com, DealorCoupons.com.
Follow me on: LinkedIn, @rahulsetia007 and Facebook.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.