gallery-dl
gallery-dl copied to clipboard
mikf
[Twitter] Can only download around 1000 files!
The program can only download around 1000 media files, so it's a big issue for accounts with several thousand files. Please help!
Twitter only lets you see the newest 1000 files
So it's not possible to download complete profiles at all?
So it's not possible to download complete profiles at all?
Depends on which profile that is. It is kinda possible for public, non-shadowbanned profiles tweets from whom can appear in search. Here is a rough algorithm:
First you locate the oldest tweet you've managed to download so far and extract its publication date. Given this date, re-run gallery-dl like so:
gallery-dl --cookies .\cookies-twitter-com.txt "https://twitter.com/search?q=from%3A{author_handle}%20until%3A{date}"
where {author_handle} is the author handle without the @, and {date} is the date of that oldest tweet plus maybe a day or so in YYYY-MM-DD format (like 2021-11-30). You may need to run this a couple of times, updating the date if the previous run has resulted in any tweets getting downloaded.
gallery-dl already does something very similar by default. It first grabs all Tweets from /USER/media and then from /search?q=from:USER max_id=LAST MEDIA TWEET ID (See here). If the search doesn't return any results then the ~1000 Tweets from /USER/media is all you're going to get.
gallery-dl already does something very similar by default. It first grabs all Tweets from
/USER/mediaand then from/search?q=from:USER max_id=LAST MEDIA TWEET ID(See here). If the search doesn't return any results then the ~1000 Tweets from/USER/mediais all you're going to get.
Thanks for the info. So is there no way to get all files? If a profile has 12K files, the program can only give you 1/12 of the uploaded files.
gallery-dl already does something very similar by default
Nice, didn't know that. But that still doesn't solve the problem of search results ending abruptly. Just verified on a large profile I've been meaning to download (~1500 media elements): gallery-dl managed to get around 1100 first try, which is better than ~1000 the /media endpoint alone yields, but still not everything. Continuing manually with search queries worked, though. I've also noticed that if the timepoint is far enough in the past, there may be no results at first, but the same query repeated after a few seconds yields some more useful results.
