This question about Not sure...: Asked

Prevent Crawling Certain Webs

I have the recommended robots.txt file at the top of my wiki so that robots can only access view links.

Is there a best practice for preventing indexing/crawling of one or more webs?

Is that as simple as, e.g., adding

to the top level robots.txt file, or is there something else I need to do?

If it makes a difference, I've got short URLs enabled via Apache.

-- VickiBrown - 27 Feb 2014


QuestionForm edit

Subject Not sure...
Version Foswiki 1.1.9
Status Asked
Related Topics
Topic revision: r1 - 27 Feb 2014, VickiBrown - This page was cached on 16 Jan 2020 - 15:24.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy