sentiment.ai icon indicating copy to clipboard operation
sentiment.ai copied to clipboard

add a warning about text length?

Open erdnaxel opened this issue 1 year ago • 0 comments

Hello: I've been using your package to get the sentiment of tweets. It's been working very well, so thank you!

Recently, the data source I use to retrieve tweets has started providing the full text of tweets, instead of a truncated version when the tweets are long. (Some tweets are very, very long.)

Not knowing better, I continued to analyze tweet sentiment as before, but I started getting tensorflow warning messages about using more than 10% of memory. After these messages, the model would just stall and never finish the job.

I tried running the sentiment_score() function on smaller batches of tweets and that worked for a few batches, but then the problem arose again for a single small batch.

I knew the number of tweets definitely wasn't the problem -- I was only doing 100 at a time -- so I figured the length of the text must be the issue. I added a step to truncate tweets to 400 characters and that completely solved the problem.

My suggestion would be to add a warning about the length of the text affecting the performance in the Troubleshooting section of the page.

erdnaxel avatar Feb 04 '24 21:02 erdnaxel