Coming soon. The Wiki is in active development, so features and these docs may still change. Early adopters: contact us for access.

Search engines and robots.txt

Choose what the Wiki's robots.txt tells Google, Bing and other crawlers

The Wiki serves /robots.txt, the file search engines read before crawling. Settings → Security → Search engines (settings:write) decides what it says, for the whole installation.

Settings → Security → Search engines: the three choices and what robots.txt serves now
Settings → Security → Search engines: the three choices and what robots.txt serves now

1. The choices

Choicerobots.txt
Index nothing (the default)Asks every crawler to stay away
Index public spacesAllows the space list and the published pages of public spaces, and nothing else. Page history stays blocked. Blocks everything while anonymous access is off
Use my own robots.txtServes the text you enter, as it is (up to 16 KiB, no control characters). It is kept when you pick another choice

What robots.txt serves now shows the saved result. After a change, each server takes up to a minute to serve it, and crawlers may keep the old file for up to five minutes more. If the server can't be reached, robots.txt answers 503 so crawlers try again later rather than reading a wrong answer.

2. Things to know

  • robots.txt is a request, not access control: what crawlers can read is decided by anonymous access and each space's visibility.
  • Index public spaces lists at most 1,000 public spaces; any beyond stay blocked.
  • There is no sitemap yet.
  • Changing the setting raises the robots.update event and is audited.