Evaluate Detoxify to filter out unwanted prompts

[andreaskoepf]

Evaluate if unitaryai/detoxify could be used to automatically filter prompts (e.g. compute for all posts submitted to the db). Or whether it maybe could be used in a security-layer that filters input and output to a live assistant bot in production.

Please write a short report about your findings (or generate a ipynb), including the model sizes, GPU memory-requirements, inference performance, subjective opinion about the filtering quality (if possible provide some examples). Check if their license would allow us to use their model. Check how we could host the model (e.g. huggingface?).

[SzymonOzog]

I would like to take on this issue, my plan is to provide a ipynb notebook containing comparison of different models that includes:

1. Inference speed and memory usage
2. Training speed and memory usage
3. Tests of detoxify on some inputs with different levels and kinds of toxic language
4. The name of the license and main bulletpoints of it
5. Information about hosting options

[jqueguiner]

Maybe it seems that ideally, this would run on nvidia triton for fast inference right?

[SzymonOzog]

The notebook and a readme were posted in this pr: https://github.com/LAION-AI/Open-Assistant/pull/176 If anyone has any feedback or ideas how to expand this work feel free to contact me