Option not to block bots and Google Crawler

[spektrumede]

We block quite a bit of external content with Klaro on our site. The problem is that when Google crawls our site it sees lots of notices like "this content is blocked..." and doesn't load the full site, which is not ideal for SEO.

It would be good if you could choose to treat bots as having accepted all cookies so that the content loads properly.

[adewes]

Hi @spektrumede! Yes that's a good point and we're working on this, there will be an option to either automatically enable or disable all trackers for bots.

As a quickfix you could do the following now: Create a script that executes right after Klaro and contains the following code:

klaro.addEventListener('render', function(config){
    if (!/bot|googlebot|crawler|spider|robot|crawling/i.test(navigator.userAgent))
        return; // only consent automatically for bots
    let manager = klaro.getManager(config);
    manager.changeAll(false); // disable all trackers / third-party services for bots
    manager.saveAndApplyConsents();
})

This should disable all trackers and third-party services for bots. You could also choose to enable them of course.

[spektrumede]

Thanks @adewes for the quick reply. Looking forward to that feature!

[robinson2]

@adewes , an SEO pointed out to me that this technique could possibly be classified by Google as "cloaking" , which could result in the page being removed from the index.

Perhaps the long term goal could be to be included in Google's list of supported consent tools, which are listed at https://www.cookiechoices.org , instead of hiding the banner from search engine.

[adewes]

Thanks for the info @robinson2! What we noticed is that the Googlebot is actually able to click on the consent dialogue, and for our own websites and our customer's websites we could not detect any downsides when letting the consent notice enabled even for crawlers. So if we add this as a feature we'll make it opt-in since I'm not fully sure of the effect that will have.

[robinson2]

Thanks for the quick reply @adewes . That sounds good! Do you have any experience how the Googlebot reacts when using the more aggresive setting "noticeAsModal: true"?

[adewes]

We haven't tested that but I would assume that Google is able to either ignore the popup or click on "Accept". As long as you don't keep your main content from loading before the user accepts optional third-party services Google should be able to index your website just fine.

We use Klaro on all our own webpages and we haven't had a problem with indexing. This is e.g. how our new website shows up in Google search results (with Klaro enabled of course)