Dev: Add bot detection for magic links

[lonnieezell]

In one of my projects we ran into issues with Android mobile devices and SMS where the OS was trying to be too helpful and would visit the links to ensure they were safe, or prep for read aloud, etc, and it would invalidate the magic link since it had already been visited.

While this was specifically for SMS, I can see email clients attempting to be helpful breaking the magic link system, also. Additionally, there is potential use by developers using that link for SMS messages.

There are 2 solutions and we might want to consider both long-term.

1. Analyze the user-agents and/or IP address of known bots that might interfere and block their attempt at that URL.
2. A better long-term alternative is to make it something that requires user input, like sending 4-6 numbers in the email and require user input so we don't have to play cat and mouse with bots.

Here's the list of user agent strings that we've used and is working so far:

• Googlebot
• Bingbot
• Slurp
• DuckDuckBot
• Baiduspider
• Sogou
• Exabot
• ia_archiver
• Google-PageRenderer
• Google-Read-Aloud
• Google-Safety
• bot
• crawler
• spider
• Yandex
• BingPreview

[michalsn]

I believe someone mentioned a similar issue on Slack some time ago. I prefer option 2, as it appears to be the most widely adopted. We would still use the magic link, but require a 6-digit code for verification.

We could even automatically redirect users to the magic link and wait for them to enter the code - similar to how 2FA typically works. This approach would work for both email and SMS messages.

I'm not sure how many changes this would require, but it would likely "break" the current magic link implementation, since the logic would need to be updated and a new view added.

That said, option 1 seems less invasive, as we could provide a config to specify which user agents should be blocked.

[lonnieezell]

I agree with those sentiments. I think maybe option 1 is a good short term fix, and then we either create a new path for option 2. or deprecate the existing magic links and replace it in the next major version?

[michalsn]

Sounds good to me. Though I wonder - wouldn't it make sense to just update the UserAgents::robots array and use that to block access? That way, it wouldn't be configurable at the Shield level, but rather handled within CodeIgniter itself.

[lonnieezell]

Yes, that sounds perfect.

[datamweb]

2. A better long-term alternative is to make it something that requires user input, like sending 4-6 numbers in the email and require user input so we don't have to play cat and mouse with bots.

The solution I prefer is the one I’ve previously implemented for activation links using SMS in several projects.