ciderpunx
mobile_twitter_to_rss not working
My local install has been running mobile_twitter_to_rss just fine for months, but broke recently (within the last 2 days).
I did some digging and was getting a 400 error until I changed line 28 of the current master to use a full User Agent string:
$browser->agent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:84.0) Gecko/20100101 Firefox/84.0');
I now get a 404, which is a redirect after getting a result in the perl debugger with the $response->content containing "We've detected that JavaScript is disabled in this browser." I'm guessing the page content that LWP was pulling in was non-JS, and that's gone now. I've reached my limit of digging for now. I may dig further later, but I have a question before I try:
Were the old LWP results pulling in a non-JavaScript version from mobile.twitter.com?
If not, this may be heavy lifting and not worth my efforts. (I may try working with nitter if the non-JS was what it was using, and twitter has killed that. but I like how twitrrsme is exactly what I want, and nothing more.)
What is the "external" symptom of the error you describe? I have a local installation of TwitRSS.me too, which has been working fine until a few days ago. Now, all requests fail with an error "ERR: Can’t screenscrape Twitter". I wanted to open an issue for that, but I wouldn't want to create a duplicate.
Here is the full error I get when running it on the command line:
Content-type: text/html
Status: 404
Cache-control: max-age=86400
Refresh: 10; url=http://twitrss.me
<html><head></head><body><h2>ERR: Can’t screenscrape Twitter</h2><p>Redirecting you back to <a href="http://twitrss.me">TwitRSS.me</a> in a few seconds. You might have spelled the username wrong or something</p></body></html>
Is this the end of the road for twitrss.me? My local installation has been down since 12/16/2020. "We've detected that JavaScript is disabled in this browser. Please enable JavaScript or switch to a supported browser to continue using twitter.com. You can see a list of supported browsers in our Help Center." We could probably rebuild this using selenium webdriver with a headless browser to pull the pages for scraping, if there is no quicker solution out there.
https://github.com/zedeus/nitter is a solution that is already built, but it's a full Twitter interface without java script that has RSS support. It's a cudgel if you just want RSS that links to twitter, but it is what I'm currently using as a workaround to twitrssme breaking with Twitter's update. They have links to hosted versions of it, so you can just make a new feed to one of those servers. If you want to interact with the individual twitter posts (like comment on one), you have to open the link in the RSS post (which displays the post on the hosted Nitter site), then click on the twitter icon to go the original page.
I'm thinking of a script to have the https://github.com/FreshRSS/FreshRSS installation I'm using on my server to change the links to the hosted Nitter site to point to the original Twitter post, but that would be something I'd add for myself, possibly cleaning it up to make a PR for FreshRSS.
https://github.com/RSS-Bridge/rss-bridge may work, but I just found it again, so I haven't set it up.
Nitter works like a champ. Had to switch to using nitter.dark.fail as the source URL because the main site was overloaded, but it all pairs very nicely with my RSS reader (The Old Reader) with the same feature set I was used to from twitrssme.