BooRah is an automated search engine that uses software "crawlers" to index the web on a regular basis and find sites to add. The majority of sites and reviews listed in our results aren't manually submitted for inclusion, but found and added automatically as we crawl the web.

You can instruct us not to include content from your site in our index or to remove content from your site that is currently in our index in the following ways:

• Remove your entire website or part of your website using a robots.txt file.
• Remove individual pages of your website using a robots meta tag.

The Robot Exclusion Standard provides a way for web site administrators to restrict robot access to their web server by specifying crawler directives in a file called /robots.txt and with meta tags placed in the "HEAD" section of each html page.

Our crawler's HTTP user-agent string is "boorahbot"

Boorah attempts to access each web server no more than once every few seconds. This rate may occasionally increase due to network delays. It may also increase periodically as we test new crawler software while running our operational crawl at the same time.

For efficiency reasons, boorah caches a copy of the /robots.txt file locally, which it refreshes every 24 hours. It can therefore take up to 24 hours for changes in a /robots.txt file to get picked up by the crawler.

If the /robots.txt file is not in the proper location, it will not get picked up. Make sure you're following the Robot Exclusion Standard exactly.

If /robots.txt is not configured or inaccessible on your web server, the crawler won't be able to read it and will assume access to your entire site is allowed.