cross
New Foswiki release 2.1.6 is available with important security fixes.
Sourceforge foswiki email lists being discontinued. Subscribe to the new Foswiki announce and discuss lists at MailingLists
This question about Not sure...: Asked

Prevent Crawling Certain Webs

I have the recommended robots.txt file at the top of my wiki so that robots can only access view links.

Is there a best practice for preventing indexing/crawling of one or more webs?

Is that as simple as, e.g., adding
/bin/view/System

to the top level robots.txt file, or is there something else I need to do?

If it makes a difference, I've got short URLs enabled via Apache.

-- VickiBrown - 27 Feb 2014

 

QuestionForm edit

Subject Not sure...
Extension
Version Foswiki 1.1.9
Status Asked
Related Topics
Topic revision: r1 - 27 Feb 2014, VickiBrown - This page was cached on 18 Jun 2018 - 08:23.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy