gallery-dl icon indicating copy to clipboard operation
gallery-dl copied to clipboard
verified publisher icon mikf

[twitter] "Unavailable" when using twitter even with username and password

Open Deer-Spangle opened this issue 1 month ago • 3 comments

It seems like downloading from twitter is no longer working? Whether or not I provide a username and password. I'm not sure if it's still working with cookies, but it seems not to work in the other ways. I'm not certain whether this might be due to trying from the UK, but I saw similar or excessive rate limit warnings when I tried some VPN endpoints.

I'm using the absolute latest version from the git repository:

$ pip install -U --force-reinstall git+https://github.com/mikf/gallery-dl

And an example:

$ gallery-dl "https://twitter.com/user/status/1889995281200209929" --verbose Output 
$ gallery-dl "https://twitter.com/user/status/1889995281200209929" --verbose
[gallery-dl][debug] Version 1.31.0-dev - Git HEAD: d7a487e
[gallery-dl][debug] Python 3.12.3 - Linux-6.8.0-87-generic-x86_64-with-glibc2.39
[gallery-dl][debug] requests 2.32.5 - urllib3 2.6.1
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/user/status/1889995281200209929'
[twitter][debug] Using TwitterTweetExtractor for 'https://twitter.com/user/status/1889995281200209929'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): x.com:443
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/qxWQxcMLiTPcavz9Qy5hwQ/TweetResultByRestId?variables=%7B%22tweetId%22%3A%221889995281200209929%22%2C%22withCommunity%22%3Afalse%2C%22includePromotedContent%22%3Afalse%2C%22withVoice%22%3Afalse%7D&features=%7B%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticleRichContentState%22%3Atrue%2C%22withArticlePlainText%22%3Afalse%2C%22withGrokAnalyze%22%3Afalse%2C%22withDisallowedReplyControls%22%3Afalse%7D HTTP/1.1" 200 101
[twitter][debug] 
Traceback (most recent call last):
  File "/mnt/md0/samba/private/bot/GalleryDLSubBot/venv/lib/python3.12/site-packages/gallery_dl/extractor/twitter.py", line 1023, in _tweets_single
    self._assign_user(tweet["core"]["user_results"]["result"])
                      ~~~~~^^^^^^^^
KeyError: 'core'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/md0/samba/private/bot/GalleryDLSubBot/venv/lib/python3.12/site-packages/gallery_dl/job.py", line 158, in run
    msg = self.dispatch(extractor)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/md0/samba/private/bot/GalleryDLSubBot/venv/lib/python3.12/site-packages/gallery_dl/job.py", line 209, in dispatch
    for msg, url, kwdict in messages:
  File "/mnt/md0/samba/private/bot/GalleryDLSubBot/venv/lib/python3.12/site-packages/gallery_dl/extractor/twitter.py", line 85, in items
    for tweet in tweets:
  File "/mnt/md0/samba/private/bot/GalleryDLSubBot/venv/lib/python3.12/site-packages/gallery_dl/extractor/twitter.py", line 1025, in _tweets_single
    raise exception.AbortExtraction(
gallery_dl.exception.AbortExtraction: 'Unavailable'
[twitter][error] 'Unavailable'

And an example providing username and password:

$ gallery-dl "https://twitter.com/user/status/1889995281200209929" --verbose -u "" -p ""

Output:

$ gallery-dl "https://twitter.com/user/status/1889995281200209929" --verbose -u "<redacted>" -p "<redacted>"
[gallery-dl][debug] Version 1.31.0-dev - Git HEAD: d7a487e
[gallery-dl][debug] Python 3.12.3 - Linux-6.8.0-87-generic-x86_64-with-glibc2.39
[gallery-dl][debug] requests 2.32.5 - urllib3 2.6.1
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/user/status/1889995281200209929'
[twitter][debug] Using TwitterTweetExtractor for 'https://twitter.com/user/status/1889995281200209929'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): x.com:443
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/qxWQxcMLiTPcavz9Qy5hwQ/TweetResultByRestId?variables=%7B%22tweetId%22%3A%221889995281200209929%22%2C%22withCommunity%22%3Afalse%2C%22includePromotedContent%22%3Afalse%2C%22withVoice%22%3Afalse%7D&features=%7B%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticleRichContentState%22%3Atrue%2C%22withArticlePlainText%22%3Afalse%2C%22withGrokAnalyze%22%3Afalse%2C%22withDisallowedReplyControls%22%3Afalse%7D HTTP/1.1" 200 101
[twitter][debug] 
Traceback (most recent call last):
  File "/mnt/md0/samba/private/bot/GalleryDLSubBot/venv/lib/python3.12/site-packages/gallery_dl/extractor/twitter.py", line 1023, in _tweets_single
    self._assign_user(tweet["core"]["user_results"]["result"])
                      ~~~~~^^^^^^^^
KeyError: 'core'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/md0/samba/private/bot/GalleryDLSubBot/venv/lib/python3.12/site-packages/gallery_dl/job.py", line 158, in run
    msg = self.dispatch(extractor)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/md0/samba/private/bot/GalleryDLSubBot/venv/lib/python3.12/site-packages/gallery_dl/job.py", line 209, in dispatch
    for msg, url, kwdict in messages:
  File "/mnt/md0/samba/private/bot/GalleryDLSubBot/venv/lib/python3.12/site-packages/gallery_dl/extractor/twitter.py", line 85, in items
    for tweet in tweets:
  File "/mnt/md0/samba/private/bot/GalleryDLSubBot/venv/lib/python3.12/site-packages/gallery_dl/extractor/twitter.py", line 1025, in _tweets_single
    raise exception.AbortExtraction(
gallery_dl.exception.AbortExtraction: 'Unavailable'
[twitter][error] 'Unavailable'

Is almost precisely the same

An example on macOS, via a VPN:

% gallery-dl "https://twitter.com/user/status/1889995281200209929" --verbose

Output:

% gallery-dl "https://twitter.com/user/status/1889995281200209929" --verbose
[gallery-dl][debug] Version 1.31.0-dev - Git HEAD: d7a487e
[gallery-dl][debug] Python 3.13.5 - macOS-15.6.1-x86_64-i386-64bit-Mach-O
[gallery-dl][debug] requests 2.32.5 - urllib3 2.6.1
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/user/status/1889995281200209929'
[twitter][debug] Using TwitterTweetExtractor for 'https://twitter.com/user/status/1889995281200209929'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): x.com:443
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/qxWQxcMLiTPcavz9Qy5hwQ/TweetResultByRestId?variables=%7B%22tweetId%22%3A%221889995281200209929%22%2C%22withCommunity%22%3Afalse%2C%22includePromotedContent%22%3Afalse%2C%22withVoice%22%3Afalse%7D&features=%7B%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticleRichContentState%22%3Atrue%2C%22withArticlePlainText%22%3Afalse%2C%22withGrokAnalyze%22%3Afalse%2C%22withDisallowedReplyControls%22%3Afalse%7D HTTP/1.1" 200 101
[twitter][debug] 
Traceback (most recent call last):
  File "/Users/spangle/PycharmProjects/GalleryDLSubBot/venv/lib/python3.13/site-packages/gallery_dl/extractor/twitter.py", line 1023, in _tweets_single
    self._assign_user(tweet["core"]["user_results"]["result"])
                      ~~~~~^^^^^^^^
KeyError: 'core'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/spangle/PycharmProjects/GalleryDLSubBot/venv/lib/python3.13/site-packages/gallery_dl/job.py", line 158, in run
    msg = self.dispatch(extractor)
  File "/Users/spangle/PycharmProjects/GalleryDLSubBot/venv/lib/python3.13/site-packages/gallery_dl/job.py", line 209, in dispatch
    for msg, url, kwdict in messages:
                            ^^^^^^^^
  File "/Users/spangle/PycharmProjects/GalleryDLSubBot/venv/lib/python3.13/site-packages/gallery_dl/extractor/twitter.py", line 85, in items
    for tweet in tweets:
                 ^^^^^^
  File "/Users/spangle/PycharmProjects/GalleryDLSubBot/venv/lib/python3.13/site-packages/gallery_dl/extractor/twitter.py", line 1025, in _tweets_single
    raise exception.AbortExtraction(
        f"'{tweet.get('reason') or 'Unavailable'}'")
gallery_dl.exception.AbortExtraction: 'Unavailable'
[twitter][error] 'Unavailable'

I'm not sure if others see this, or if I need to migrate to using cookies, but it's been broken for a couple weeks on my side

Deer-Spangle avatar Dec 09 '25 22:12 Deer-Spangle

It works just fine, but you need to pass authenticated cookies to access Tweets with "Age-restricted adult content":

$ gallery-dl -v https://twitter.com/user/status/1889995281200209929
[gallery-dl][debug] Version 1.31.0-dev - Git HEAD: 8e5d8d8c
[gallery-dl][debug] Python 3.13.7 - Linux-6.17.8-arch1-1-x86_64-with-glibc2.42
[gallery-dl][debug] requests 2.32.5 - urllib3 2.5.0
[gallery-dl][debug] Configuration Files ['${HOME}/.gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/user/status/1889995281200209929'
[twitter][debug] Using TwitterTweetExtractor for 'https://twitter.com/user/status/1889995281200209929'
[twitter][debug] cookies: Loading cookies from '~/cookies-x-com.txt'
[twitter][info] Initializing client transaction keys
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): x.com:443
[urllib3.connectionpool][debug] https://x.com:443 "GET / HTTP/1.1" 200 None
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): abs.twimg.com:443
[urllib3.connectionpool][debug] https://abs.twimg.com:443 "GET /responsive-web/client-web/ondemand.s.deec4d3a.js HTTP/1.1" 200 11598
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/iFEr5AcP121Og4wx9Yqo3w/TweetDetail?variables=%7B%22focalTweetId%22%3A%221889995281200209929%22%2C%22referrer%22%3A%22profile%22%2C%22with_rux_injections%22%3Afalse%2C%22includePromotedContent%22%3Afalse%2C%22withCommunity%22%3Atrue%2C%22withQuickPromoteEligibilityTweetFields%22%3Afalse%2C%22withBirdwatchNotes%22%3Atrue%2C%22withVoice%22%3Atrue%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticleRichContentState%22%3Atrue%2C%22withArticlePlainText%22%3Afalse%2C%22withGrokAnalyze%22%3Afalse%2C%22withDisallowedReplyControls%22%3Afalse%7D HTTP/1.1" 200 3133
./twitter/Nellies_Nest/1889995281200209929_1.jpg

Whether or not I provide a username and password

No longer supported since it's basically blocked for everyone. (c1d21e8cb9fab9c3c19ecdfede65db3722e69fb9)

mikf avatar Dec 10 '25 07:12 mikf

Ohhh, I had seemed to be using username and password successfully until November 13th.. Though, perhaps I hadn't updated gallery-dl for a while at that point.

Deer-Spangle avatar Dec 10 '25 11:12 Deer-Spangle

It was still using the cached login session for your username, but the cache entry most likely expired.

What you could do is modify gallery-dl's cache file with some SQLite tool like https://sqlitebrowser.org/ and set the expires timestamp value of the ….twitter._login_impl.… row to some time in the far future, so it will get used.

Or just use cookies...

Maybe I should add an option to ignore an SQLite file cache's expires timestamp...

mikf avatar Dec 10 '25 11:12 mikf

Oh, that's a neat idea, yeah. I have switched to using cookies now, and can confirm it all works as expected when I do that! I'm somewhat surprised it didn't complain about some unused variables being set in configuration when I was setting username and password, perhaps?

Are these cookies I'll have to swap out regularly, do you know? (like the s3nd.pics jwt which only lasts a week) Or will it update or some such?

Deer-Spangle avatar Dec 12 '25 07:12 Deer-Spangle

I'm pretty sure these cookies will stay valid for a long time, like several months or even years.

I'm somewhat surprised it didn't complain about some unused variables being set in configuration when I was setting username and password, perhaps?

It should output an error message when trying to login with username & password and the cache entry has expired: https://github.com/mikf/gallery-dl/blob/ab2c03b39eacc897736046c7d67ea3f4a1d7efbc/gallery_dl/extractor/twitter.py#L2270-L2272

Or will it update or some such?

https://gdl-org.github.io/docs/configuration.html#extractor-cookies-update (I'm using them with "cookies-update": false to avoid disk writes, and they still stay valid)

mikf avatar Dec 12 '25 08:12 mikf