AI based moderation for toxicity, spam and AI detection

[leedenison]

[nairiboo]

Adding to this; the current method of moderation has a large impact on support resources as it is currently a manual process and not scalable.

[mkdir-washington-edu]

I don't know what happened to the description here, but I think this issue came from this thread: https://hypothes-is.slack.com/archives/C2BLQDKHA/p1686943086486359?thread_ts=1686934813.780349&cid=C2BLQDKHA

The web app has a number of accounts that are likely created by spammers. While we can manually detect spammers when they post on our site, or when their annotations get flagged, we don't have another way to detect them. Once detected we can NIPSA them (or delete their accounts, but we've never done that as far as I know), but we don't have other tools.

The support team investigated the last time we saw a huge number of new accounts made in a short time and we saw some trends in Spam accounts on our site:

• almost everyone filling out the “URL” field in the user profiles is linking to questionable sites; it seems like most users don't use this field, and most spammers use it to link to spammy sites
• Lots of these accounts also use descriptions that contain web addresses that are questionable

Some thoughts:

• Can we detect when someone creates 100+ accounts with small variations on email address in a short time and auto-nipsa all those accounts? “100+” b/c I can envision reasonable users make 2 or 3 this way
• Can we run the URL user field through the palo alto tool or similar and auto-NIPSA folks linking to spammy sites?
• Can we build SSO registration and then kill username/email registration, and will that solve this issue?