warc-specifications icon indicating copy to clipboard operation
warc-specifications copied to clipboard
verified publisher icon iipc

Common "robots" values are undocumented

Open notcancername opened this issue 7 months ago • 6 comments

warcinfo records may contain optional fields. One such field is robots. The only suggested value is classic. No other values are documented. This may lead to implementations choosing different values with the same meaning, making the field more difficult for consumers to interpret.

notcancername avatar May 23 '25 20:05 notcancername

I think this suggestion predates RFC 9309, which is the new hotness in the robots.txt space.

wumpus avatar May 23 '25 20:05 wumpus

@notcancername Could you give examples of other used or suggested values, and what they would mean?

@wumpus I don't see how RFC 9309 solves this problem, what did you mean?

petsva avatar May 26 '25 09:05 petsva

Heritrix, which is probably where the 'classic' value originally came from, currently has these policies:

  • classic (alias obey)
  • ignore
  • robotsTxtOnly (obeys robots.txt but ignores the robots meta tag)

Another common configuration that might be useful to record is following robots.txt at the page level but ignoring it for subresources.

ato avatar May 26 '25 09:05 ato

You can also use value "obey" and in org.archive.crawler.prefetch.PreconditionEnforcer specify calculateRobotsOnly=true.

Then robots.txt is NOT obeyed, but exlcuded URIs are annotated in the crawl.log. for use in presentation and usage decisions later.

petsva avatar May 26 '25 10:05 petsva

It's only since recently that the robots.txt was standardized by RFC 9309.

Before, practically every search engine crawler set its own standard modifying the original RFC proposal from 1994.

My interpretation of classic was that it refers to the 1994 version. Since summer 2018 the Common Crawl WARC files refer to the version of the robots.txt parser library used by the crawler at this point.

Given multiple robots.txt standards (or specs) and implementations, a value obey isn't really precise.

  • robotsTxtOnly (obeys robots.txt but ignores the robots meta tag)

Good point. Information about the meta tags should be also there. But again: interpretation of nofollow, noindex, noarchive, nosnippet etc. is not standardized - there's an expired draft.

sebastian-nagel avatar May 26 '25 13:05 sebastian-nagel

@sebastian-nagel is getting good at reading my mind! I was suggesting "rfc9309" as a value, since it's different from "classic".

wumpus avatar May 26 '25 20:05 wumpus