LAION-AI
Tools for "batch" moderation
Moderators want the ability to search and filter through the data, this should make the moderation process easier and faster, and perhaps a bit more automated.
As an example: allow regex searches through all messages.
Some mods have experience with SQL and could in theory achieve this relatively quickly if they had access to the database.
This has the potential problem of exposing our database to the outside world and reducing its security.
We could allow access through the website by implementing a UI for the database, we have to evaluate the effort and the features required for achieving this.
Another easier option could be to have a database dump (or maybe just trollboard dump) from a specific date onwards that contains the information relevant for moderation in a common format (example: CSV) These dumps will be slightly outdated.
One option could also be to set up some sort of CDC solution and then analytics and other operational stuff like this could maybe be done on the replication data and so not touching production db and can maybe be locked down or have views etc built on it to serve whatever is needed.
CDC could be a bit complex to do though, not sure.
https://materialize.com/docs/integrations/cdc-postgres/#:~:text=Change%20Data%20Capture%20(CDC)%20allows,on%20top%20of%20CDC%20data.
Airbyte seems to have some sort of support that maybe could be of use: https://docs.airbyte.com/understanding-airbyte/cdc/
Maybe CDC is a bigger decision in terms of if a near real time replication of the production data would be useful for lost of different stuff.
Eg could have CDC going to bigquery or snowflake or something and then have this "analytical replication" of production be used for whatever.
Spin off a new issue from this for basic moderator search functionality: #3009
I think we can close this for now, we have basic message search and editing. If there are further proposals we can create more specific issues for them