fess icon indicating copy to clipboard operation
fess copied to clipboard

Exclude certain mimetypes from crawling

Open defensivedepth opened this issue 7 years ago • 0 comments

For web crawlers, is there a way to exclude certain mimetypes from being crawled/indexed? For instance, mimetype:"application/rss+xml"

I can try to exclude it through the use of certain URL patterns under Excluded URLs (like RSS or MCRSS) but it is not comprehensive.

defensivedepth avatar Feb 16 '18 18:02 defensivedepth