nitter
nitter copied to clipboard
zedeus
looks like X/twitter(?) broke something again
Also, the syndication api for showReplies=true does not work anymore:
https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=true
but showReplies=false still works, showing the tweets ordered by like count...
https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=false
Is there a online/CLI tool converting https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk to RSS feed? Then we could individually download the HTML from a logged in profile and do the conversion in a second step.
Also, the syndication api for
showReplies=truedoes not work anymore: https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=true butshowReplies=falsestill works, showing the tweets ordered by like count... https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=false
Not really. showReplies=false shows years-old content when not logged in.
Also, the syndication api for
showReplies=truedoes not work anymore: https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=true butshowReplies=falsestill works, showing the tweets ordered by like count... https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=falseNot really. showReplies=false shows years-old content when not logged in.
That's because in that specific example, those tweets were years ago. Look again at the like count, notice anything?
Is there a online/CLI tool converting https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk to RSS feed? Then we could individually download the HTML from a logged in profile and do the conversion in a second step.
We can just search for the first { from begin and first } from end and then parse as json.
If a user has not so many tweets (500-1000) then the chance is quite good that also newer tweets are within the most popular 100.
But of course it's not a very good solution. At least, nitter should have it as a backup when nothing else works, this method can be used.
Is there a online/CLI tool converting https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk to RSS feed? Then we could individually download the HTML from a logged in profile and do the conversion in a second step.
We can just search for the first
{from begin and first}from end and then parse as json. If a user has not so many tweets (500-1000) then the chance is quite good that also newer tweets are within the most popular 100. But of course it's not a very good solution. At least, nitter should have it as a backup when nothing else works, this method can be used.
Indeed
#!/usr/bin/python3
import requests
import re
import urllib
url = "https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk"
with urllib.request.urlopen(url) as response:
encoding = response.info().get_param('charset', 'utf8')
html = response.read().decode(encoding)
result = re.search('script id="__NEXT_DATA__" type="application\/json">([^>]*)<\/script>', html)[1]
print(result)
Indeed
Interesting, but this doesn't return RSS with 'item', 'pubDate' etc. tags. Maybe a script using https://github.com/lkiesow/python-feedgen would do the job?
Indeed
Interesting, but this doesn't return RSS with 'item', 'pubDate' etc. tags. Maybe a script using https://github.com/lkiesow/python-feedgen would do the job?
Not sure I understand ? It expose far more informations than needed and it does expose the date and all
Here's an example for one tweet only :
{
"type": "tweet",
"entry_id": "tweet-1519480761749016577",
"sort_index": "1691455400412446720",
"content": {
"tweet": {
"id": 0,
"location": "",
"conversation_id_str": "1519480761749016577",
"created_at": "Thu Apr 28 00:56:58 +0000 2022",
"display_text_range": [
0,
52
],
"entities": {
"user_mentions": [],
"urls": [],
"hashtags": [],
"symbols": [],
"media": []
},
"favorite_count": 4600599,
"favorited": false,
"full_text": "Next I’m buying Coca-Cola to put the cocaine back in",
"id_str": "1519480761749016577",
"lang": "en",
"permalink": "/elonmusk/status/1519480761749016577",
"possibly_sensitive": false,
"quote_count": 171975,
"reply_count": 187438,
"retweet_count": 649833,
"retweeted": false,
"text": "Next I’m buying Coca-Cola to put the cocaine back in",
"user": {
"blocking": false,
"created_at": "Tue Jun 02 20:12:29 +0000 2009",
"default_profile": false,
"default_profile_image": false,
"description": "Blades of Glory",
"entities": {
"description": {
"urls": []
},
"url": {}
},
"fast_followers_count": 0,
"favourites_count": 30569,
"follow_request_sent": false,
"followed_by": false,
"followers_count": 153112066,
"following": false,
"friends_count": 410,
"has_custom_timelines": false,
"highlightedLabel": {
"badge": {
"url": "https://pbs.twimg.com/profile_images/1683899100922511378/5lY42eHs_bigger.jpg"
},
"description": "X",
"userLabelType": "BusinessLabel",
"userLabelDisplayType": "Badge"
},
"id": 0,
"id_str": "44196397",
"is_translator": false,
"listed_count": 126597,
"location": "𝕏Ð",
"media_count": 1659,
"name": "Elon Musk",
"normal_followers_count": 153112066,
"notifications": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/44196397/1690621312",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1683325380441128960/yRsRRjGO_normal.jpg",
"protected": false,
"screen_name": "elonmusk",
"show_all_inline_media": false,
"statuses_count": 29441,
"time_zone": "",
"translator_type": "none",
"url": "",
"utc_offset": 0,
"verified": false,
"withheld_in_countries": [],
"withheld_scope": "",
"is_blue_verified": true
}
}
}
},
EDIT : Maybe you meant a directly usable solution for an end user, and of course it's not, the snippet need to be adapted by a dev.
Indeed
Interesting, but this doesn't return RSS with 'item', 'pubDate' etc. tags. Maybe a script using https://github.com/lkiesow/python-feedgen would do the job?
Not sure I understand ? It expose far more informations than needed and it does expose the date and all
Ok, thank you, I'll try this.
Is there a online/CLI tool converting https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk to RSS feed? Then we could individually download the HTML from a logged in profile and do the conversion in a second step.
calling the syndication URL without being logged in twitter doesn't retrieve the most recent tweets. If I call this url in postman, I retrieve 100 tweets from 10/19/2018 to 07/31/2023; no tweets from august...
Is there a online/CLI tool converting https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk to RSS feed? Then we could individually download the HTML from a logged in profile and do the conversion in a second step.
calling the syndication URL without being logged in twitter doesn't retrieve the most recent tweets. If I call this url in postman, I retrieve 100 tweets from 10/19/2018 to 07/31/2023; no tweets from august...
It retrieves the tweets with the highest like count from that user, which doesnt sound good if your goal is retrieving the most recent tweets, as there's no guarantee new tweets will make it to the top 100 tweets from that user. And even if they did, it might take a considerable amount of time
I've noticed that for smaller accounts that have less than 100 tweets, that syndication URL does not load any tweets.
Also, the syndication api for
showReplies=truedoes not work anymore: https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=true butshowReplies=falsestill works, showing the tweets ordered by like count... https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=falseNot really. showReplies=false shows years-old content when not logged in.
That's because in that specific example, those tweets were years ago. Look again at the like count, notice anything?
No. That is the case for all big accounts. I am interested in the most recent Tweets and this approach will lead to nothing.
Indeed
Interesting, but this doesn't return RSS with 'item', 'pubDate' etc. tags. Maybe a script using https://github.com/lkiesow/python-feedgen would do the job?
Not sure I understand ? It expose far more informations than needed and it does expose the date and all
Here's an example for one tweet only :
{ "type": "tweet", "entry_id": "tweet-1519480761749016577", "sort_index": "1691455400412446720", "content": { "tweet": { "id": 0, "location": "", "conversation_id_str": "1519480761749016577", "created_at": "Thu Apr 28 00:56:58 +0000 2022", "display_text_range": [ 0, 52 ], "entities": { "user_mentions": [], "urls": [], "hashtags": [], "symbols": [], "media": [] }, "favorite_count": 4600599, "favorited": false, "full_text": "Next I’m buying Coca-Cola to put the cocaine back in", "id_str": "1519480761749016577", "lang": "en", "permalink": "/elonmusk/status/1519480761749016577", "possibly_sensitive": false, "quote_count": 171975, "reply_count": 187438, "retweet_count": 649833, "retweeted": false, "text": "Next I’m buying Coca-Cola to put the cocaine back in", "user": { "blocking": false, "created_at": "Tue Jun 02 20:12:29 +0000 2009", "default_profile": false, "default_profile_image": false, "description": "Blades of Glory", "entities": { "description": { "urls": [] }, "url": {} }, "fast_followers_count": 0, "favourites_count": 30569, "follow_request_sent": false, "followed_by": false, "followers_count": 153112066, "following": false, "friends_count": 410, "has_custom_timelines": false, "highlightedLabel": { "badge": { "url": "https://pbs.twimg.com/profile_images/1683899100922511378/5lY42eHs_bigger.jpg" }, "description": "X", "userLabelType": "BusinessLabel", "userLabelDisplayType": "Badge" }, "id": 0, "id_str": "44196397", "is_translator": false, "listed_count": 126597, "location": "𝕏Ð", "media_count": 1659, "name": "Elon Musk", "normal_followers_count": 153112066, "notifications": false, "profile_banner_url": "https://pbs.twimg.com/profile_banners/44196397/1690621312", "profile_image_url_https": "https://pbs.twimg.com/profile_images/1683325380441128960/yRsRRjGO_normal.jpg", "protected": false, "screen_name": "elonmusk", "show_all_inline_media": false, "statuses_count": 29441, "time_zone": "", "translator_type": "none", "url": "", "utc_offset": 0, "verified": false, "withheld_in_countries": [], "withheld_scope": "", "is_blue_verified": true } } } },EDIT : Maybe you meant a directly usable solution for an end user, and of course it's not, the snippet need to be adapted by a dev.
im using this for my bot and it working fine with cookies and headers.
Is there a online/CLI tool converting https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk to RSS feed? Then we could individually download the HTML from a logged in profile and do the conversion in a second step.
calling the syndication URL without being logged in twitter doesn't retrieve the most recent tweets. If I call this url in postman, I retrieve 100 tweets from 10/19/2018 to 07/31/2023; no tweets from august...
Yes. And there is at least one tweet from August with more likes (>807K) than some older tweets which are included (e.g. <680K).
I am aware but couldn't nitter implement a system that aurora uses with lots of accounts that rotate per user?
I am aware but couldn't nitter implement a system that aurora uses with lots of accounts that rotate per user?
That's hard to maintain and simple for twitter to ban by just filtering "if number of accounts per IP > SOME_CONSTANT: ban all of them"
Looks like https://nitter.privacydev.net/ is working
User feeds not working on this
I switched to the privacydevel fork, credentials in but its still 404ing the same endpoint upstream is having problems with
Strange. privacydev (without credentials) works more or less for @elonmusk, but not for other users like for instance @barackobama.
Also, the syndication api for
showReplies=truedoes not work anymore: https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=true butshowReplies=falsestill works, showing the tweets ordered by like count... https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=false
Not logged in, pulling a TL in the normal fashion (eg. twitter.com/elonmusk) supplies results like this as well.
Has this always (recently, since all the major upheavals) been the case?
No. With a browser specific user agent up to a few days twitter/username would just show the profile and one tweet. Using "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" and coming from a tweet of the user the correct and fresh timeline was shown.
Also, the syndication api for
showReplies=truedoes not work anymore: https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=true butshowReplies=falsestill works, showing the tweets ordered by like count... https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk?showReplies=falseNot logged in, pulling a TL in the normal fashion (eg. twitter.com/elonmusk) supplies results like this as well.
Has this always (recently, since all the major upheavals) been the case?
Yes. They switched the same time on the main page and the syndication api with showReplies=false to show the top 100 tweets. First, they changed the googlebot method from show the latest 20 tweets to the top 100. That was in the time, where twitter was completely locked without account and only googlebot method worked. When they found it a good solution to break scraping, they switched the other two to do the same. But since today, also showReplies=true does not work anymore.
Additionally, they removed timeline support completely on the "embed page". Thus, we can expect that timelines on the syndication api will go away completely within the next days or weeks (also top 100).
I assume, within the next 3 months, scraping without an account will not be possible anymore, besides top tweets. The top tweet method is a quite good solution for twitter: "normal users" without an account think Twitter is working perfectly fine, scrapers can't scrape anymore. Account-havers are not realizing any differences.