Extended docs for the Crawler in the back end

[Toflar]

Draft because the to.contao.org link is up for discussion: https://github.com/contao/to.contao.org/pull/80

[Toflar]

This is ready now.

[leofeyer]

The translation file should have a %s wildcard and the URL should be added using sprintf().

https://github.com/contao/contao/blob/d50e67901a36692f57f1a27680dced916ce6ddd6/core-bundle/contao/languages/en/default.xlf#L1205-L1207

https://github.com/contao/contao/blob/d50e67901a36692f57f1a27680dced916ce6ddd6/core-bundle/contao/controllers/BackendMain.php#L187

[Toflar]

Ah right, that's why it was blocked. Updated - while doing that I also migrated to the translator because we had to do this anyway at some point.

[leofeyer]

I would have preferred it if you had just changed the language string. 🙈

[leofeyer]

Moreover, make sure you analyze the debug log to see which URLs are crawled exactly. You can ask Contao to skip links from being crawled. Please refer to the Contao docs for more information.

What are you trying to tell the user? Is it the crawler might crawl fewer URLs than you expect, so check the debug log or is it you can exclude links from crawling? I don‘t quite understand the intention here.

[Toflar]

you can exclude links from crawling

This one. If the log contains URLs you don't want to have crawled (aka it takes too long), you can exclude them.