Skip to content

Latest commit

 

History

History
42 lines (28 loc) · 1.07 KB

README.md

File metadata and controls

42 lines (28 loc) · 1.07 KB
Raw

LawsOfRobotsTxt

  1. A robot may not index staging servers
  2. A robot must obey the sitemap
  3. A robot may not injure SEO or, through inaction, cause SEO to come to harm.

Installation

Add this line to your application's Gemfile:

gem 'laws_of_robots_txt'

And then execute:

$ bundle

Be sure to remove public/robots.txt, if it exists

Usage

LawsOfRobotsTxt installs a rack middleware into your application which renders a different /robots.txt based on the request's domain.

It looks in config/robots/ for a file named <DOMAIN>.txt, for example, config/robots/www.example.com.txt. A server restart is required to pick up changes.

If no file exists for the requests domain, it renders a default

# Robots not allowed on this domain
User-Agent: *
Disallow: /

Contributing

  1. Fork it ( http://github.com/freerunningtech/laws_of_robots_txt/fork )
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request