OkCupid-Message-Downloader
OkCupid-Message-Downloader copied to clipboard
Update scraper for new message features and HTML format
As @5u2ie noted in https://github.com/lehrblogger/OkCupid-Message-Downloader/issues/43#issuecomment-414201276, the scraper no longer seems to be working. the error message is:
ERROR: There was an error queuing the threads to download - are you sure your username and password are correct? INFO: Done.
Thats logged by a somewhat-blind try except
here, so it's hard to say exactly what the problem is. OkCupid has shipped several iterations of its messaging feature since this scraper was first created, so some of the assumptions about message types and formats might not be valid.
I got a bit bored tonight so I figured I would look at this for a bit. It seems that the code is failing on line 137 of arrow_fetecher.py due to the following error:
"'NoneType' object has no attribute 'find_all'"
Seems that soup.find('ul', {'id': 'messages'}) no longer does what is intended. I am going to mess around a bit longer but can't make any promises that I will find the solution tonight.