feat: do not load discussions if it's a bot

[mariojsnunes]

PR Checklist

• [x] - Commit messages are descriptive, it will be used in our Release Notes

PR Type

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] Developer experience (improves developer workflows for contributing to the project)

Description

Stops discussions from loading if the page is requested by a bot/crawler.

Git Issues

Closes #3468

Screenshots/Videos

If useful, provide screenshot or capture to highlight main changes

---

What happens next?

Thanks for the contribution! We try to make sure all PRs are reviewed ahead of our monthly maintainers call (first Monday of the month)

If the PR is working as intended it'll be merged and included in the next platform release, if not changes will be requested and re-reviewed once updated.

If you need more immediate feedback you can try reaching out on Discord in the Community Platform development channel.

[cypress[bot]]

1 failed and 2 flaky tests on run #5453 ↗︎

1

89

1

0

2

Details:

feat: do not load discussions if it's a bot
Project: onearmy-community-platform	Commit: 09a7ecb6f7
Status: Failed	Duration: 04:33 💡
Started: Apr 22, 2024 7:41 AM	Ended: Apr 22, 2024 7:45 AM

  src/integration/howto/seo.spec.ts • 1 failed test • ci-chrome

View Output Video

Test		Artifacts
[How To] > [SEO Metadata] > [Populates title and description tags]		Test Replay Screenshots Video

  research/write.spec.ts • 1 flaky test • ci-chrome

View Output Video

Test		Artifacts
[Research] > [Create research article] > [Warning on leaving page]		Test Replay Screenshots Video

  profile.spec.ts • 1 flaky test • ci-chrome

View Output Video

Test		Artifacts
[Profile] > [By User] > [Can contact profiles by default]		Test Replay Screenshots Video

Review all test suite changes for PR #3470 ↗︎

[thisislawatts]

question: I might be missing the intention of this change, but won't this limit the ability for comments to be indexed by automated crawlers? This seems counter intuitive when we are looking to create a shared repository of knowledge, sure the primary articles are Research/How-to posts, but each additional comment contributes to that original item.

[mariojsnunes]

question: I might be missing the intention of this change, but won't this limit the ability for comments to be indexed by automated crawlers? This seems counter intuitive when we are looking to create a shared repository of knowledge, sure the primary articles are Research/How-to posts, but each additional comment contributes to that original item.

Yes it would. A few points to think about:

• We could allow some search index crawlers like google, but most of them are not great at waiting/running javascript, or even need comment information... I'd guess most crawlers that access our pages are for link previews.
• All these extra reads are expensive... need to at least paginate them, and with pagination, the first page comments you index today could be different tomorrow.

[benfurber]

My comment is more about the issue than the code, so left a response there: https://github.com/ONEARMY/community-platform/issues/3468#issuecomment-2088162894

[mariojsnunes]

Dropped in favor of https://github.com/ONEARMY/community-platform/issues/3467