funnel icon indicating copy to clipboard operation
funnel copied to clipboard

Content moderation data models

Open jace opened this issue 2 years ago • 0 comments

There is a growing need for content moderation and automatic classification into spam, or into topics (from #1421). #669 introduced CommentModeratorReport model that is applied to spam in comments. It allows anyone to report a comment as spam, and multiple designated comment moderators to second that report or contest it, allowing for spam moderation on the basis of consensus. This has worked decently enough but is nowhere near adequate:

  1. Content moderation should be possible for all types of contents (a list is #1050), and
  2. Content moderation should have some automation, allowing an ML model to make an initial guess and to learn from a reviewer's decision.

There are four broad areas to tackle:

  1. An automatic classification system should be able to record guesses on the language of content (English, other), and spam or other offence classification, using machine-learning models. These reports should be tagged with the model and its version, as models may be replaced or upgraded.

  2. A user should be able to make a spam/offence report on any piece of content. This could be a ContentReport model that subsumes CommentModeratorReport, and includes a flag marking it as resolved or not (possibly the existing resolved_at)

  3. A content reviewer's opinion on either of the above. To avoid the risk of ballot stuffing, a content reviewer's report has distinct weightage from a general user's report. The existing CommentModeratorReport merges both use cases, but this can be resolved by adding a flag based on whether the user is also a comment moderator and populating historical records.

  4. An appeal or notification system that is sent to the offending user. This is not important for spam, but will be necessary for offence reports.

The first three can all be represented using a single model, but only the consensus status from content reviewer's reports can be used to feed an ML model.

The original content models will also need columns to record the decision of the content classification system (between ML and reviewer).

Since the content models are mutable, they will also need a revision id that is recorded in the moderation reports, to help identify when older reports are no longer valid. We currently use an automatic versionid column in some of these models, but it's revised on any change to the row, not just in the content columns. We'll possibly need another that is only updated when content columns change.

jace avatar Dec 19 '22 10:12 jace