detoxify
detoxify copied to clipboard
unitaryai
Detoxify on AWS Lambda
Hi Team, I have been trying to implement my code using Detoxify library on the AWS lambda function. For this, I am downloading the 'whl' file of the library and then zipping it to put it into a Lambda layer to get it used with the Lambda function, also ensuring that detoxify is installed on my local system. This process has been working with other Python libraries, that I mentioned above to use libraries with the Lambda function. But it's not happening with Detoxify library. Kindly let me know the reasons or suggestions to get it worked, if any.
Regards, Parth Sharma
Hi Parth,
Thanks for raising this! Could you post a screenshot of the error you're getting or give some more details about where this process is going wrong? Would be great if you could give some code examples as well so we can try to reproduce this.
Hi Parth,
Thanks for raising this! Could you post a screenshot of the error you're getting or give some more details about where this process is going wrong? Would be great if you could give some code examples as well so we can try to reproduce this.
Here I'm attaching a screenshot of the error that I'm getting. When coming to a code example, there's hardly anything that I can share for that because I'm not even able to import detoxify library within my lambda function. I have tried importing other python libraries - pandas & numpy in similar manner, it worked though. Thus, I'm bit surprised why it's not happening with Detoxify. Kindly look into this issue and share your views.
You could try just making a copy of the detoxify folder from github. But I think you'd also have to do something about the other dependencies including pytorch and transformers and their own dependencies. There is also the issue of model and tokenizer weights, which you probably don't wont to download every time it starts. Perhaps using a docker container for the lambda would work better https://docs.aws.amazon.com/lambda/latest/dg/images-create.html
