Bulk moderation API

[erkinalp]

Why this is needed

As discussed in #263, a bulk moderation API would be useful for filtering inappropriate content out. Several bulk moderation models will be possible using this API, in addition to currently available post-moderation method of removing things:

• Approval/denial style pre-moderation (Pre-moderator flow)
• Filter based automated pre-moderation (Filter flow)
• Reputations and/or scoring based post-moderation (Points flow)
• Substitution rules (substitute certain character sequences by something else)

Synopsis of the endpoints

• Moderation rule create/update
  • Moderation word and context add (per guild/controlled user/controller group/group chat, but shareable)
    • for guilds: PUT /guilds/{guild.id}/policies/words?word={word}
    • for individual controlled user accounts: PUT /users/{user.id}/policies/words?word={word}
  • Moderation pretrained corpus add (per guild/controlled user/controller group/group chat, but shareable)
    • for guilds: PUT /guilds/{guild.id}/policies/corpuses?corpus={corpus_id}
    • for individual controlled user accounts: PUT /users/{user.id}/policies/corpuses?corpus={corpus.id}
  • Moderation blacklisted work add (global)
    • POST /policies/fingerprints/
  • Premoderator add/update/remove (list of users/roles, per guild/channel)
    • PUT /guilds/{guild.id}/policies/premoderators/{user.id} to add, DELETE to same path to remove
    • PUT /guilds/{guild.id}/policies/premoderators/{user.id} to add, DELETE to same path to remove
  • Premoderator approve/decline (messages, with a timeout to prevent blackout, in timeout situations, messages get flagged worst possible, if the scoring is also active, until approval comes later)
    • POST /channels/{channel.id}/messages/{message.id}/OK to approve
    • DELETE /channels/{channel.id}/messages/{message.id} to decline
• Add/update/remove scorer groups (list of users/roles)
  • PUT /guilds/{guild.id}/policies/scorers/{user.id} to add, DELETE to the same path to remove
  • GET /guilds/{guild.id}/policies/scorers to retrieve the list of scorers
• List moderation rules
  • GET /guilds/{guild.id}/policies
• Retrieve moderation rule
  • GET /guilds/{guild.id}/policies/{policy.id}
• Moderation rule enable/disable (corpus/word filter only, work filters are append only)
  • PUT /guilds/{guild.id}/policies/{policy.id}/disable, DELETE to same path for re-enable
• Create/update/remove corpus group (used for "score messages" functionality)
  • PUT /corpuses/{corpus.id} to create, PATCH to update and DELETE to remove
• Score messages for appropriateness
  • POST /channels/{channel.id}/messages/{message.id}/score with body containing the score value, higher is better
• Moderation rule remove (corpus/word filter only, not applicable to work filter)
  • DELETE /corpuses/{corpus.id}
  • DELETE /policies/words/{policy.id}
  • DELETE /guilds/{guild.id}/policies/words
• Moderation rule peek (send a test message to a moderation rule and return whether it is acceptable, only accessible to users with channel mod access)
  • POST /test?policy={policy.id}
• Appeal moderation rule decision
  • TBD

Flows

Pre-moderator and filter flows

1. user attempts to send a message
2. server looks up for the list of pre-moderators and filters of that channel and approval condition
3. a. if the approval condition is met, then the message is sent as normal b. if it fails, message is blocked and audit log record shall be created
4. if the failed post attempts of a user exceeds a configurable metric, then automatically apply a moderation action (reduce powers/kick/ban from guild/ban from server)

Points flow

1. user sends the message
2. configured scoring group (may be designated individually, or all other users) up- and downvotes the message
3. message is shadowed if it is below a certain configurable score
4. user is banned from guild if it is below a certain configurable score
5. user is banned from server if it is below a certain configurable score

Possible remedies against bulk moderation violations

• Stealth-drop the message
• Stealth-fail the message and log the issue
• Apply a penalty point

[samuelscheit]

I think we should document the whole api in https://github.com/fosscord/fosscord-docs We can fork discord-api-docs as a base as it uses MIT license https://github.com/discord/discord-api-docs/blob/master/package.json#L14 and add our own documentation

[samuelscheit]

What is corpus?

[erkinalp]

The text corpus to be used in NLP-based moderation rules.

[DiegoMagdaleno]

Em couldn't a bot be used for this?, anyway is getting added to community ideas, and removed the [EPIC] if you continue doing so, the issue will be instantly closed, is not [EPIC]

[erkinalp]

Em couldn't a bot be used for this?

This is for pre-moderation (with approvers or with automated rules). Bots cannot pre-moderate a channel.

[samuelscheit]

We would have multiple options: either allow bots/server plugins to intercept events before they get delivered (which would drastically increase the latency) or directly integrate moderation into fosscord or leave it as it as (bots do post-moderation).

[erkinalp]

intercept events before they get delivered

Some of the routes mentioned above cover that kind of system (approve/decline message endpoint).

directly integrate moderation into fosscord

That is covered in the aforementioned routes too (word filter, corpus based context aware filter, fingerprint filter, points based reputation system).

And trade-off is on the channel/guild owners: either pre-moderate and accept the latency and risk of server tampering, or post-moderate and accept the existence of cases that would be missed.