Response status: 404 on many request

[Sontranwakumo]

Thanks for the great project!

Lately, I have had issues when fetching profile, getUserIdByScreenName, getTweet, all of them have encoutered the same error: Response status: 404

About a week ago, this problem have never appeared. I updated latest version and still getting the same issue, I guess that Twitter have updated these API endpoints.

[videvian]

We see the same issue. None of the requests succeed now and fail both locally and on deployed environment for resolving profile by the user name getProfile.

I've noticed that for getProfile this endpoint is specified in code: https://api.x.com/graphql/96tVxbPqMZDoYB5pmzezKA/UserByScreenName. But Twitter is using https://x.com/i/api/graphql/ZHSN3WlvahPKVvUxVQbg1A/UserByScreenName. I tried to use this URL locally but I get this error (we don't use credentials to log in):

AuthenticationError: {"errors":[{"message":"Sorry, that page does not exist","code":34}]}
    at TwitterGuestAuth.updateGuestToken (/Users/.../node_modules/@the-convocation/twitter-scraper/src/auth.ts:259:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async TwitterGuestAuth.installTo (/Users/.../node_modules/@the-convocation/twitter-scraper/src/auth.ts:173:7)
    at async requestApi (/Users/.../node_modules/@the-convocation/twitter-scraper/src/api.ts:63:9)
    at async getProfile (/Users/.../node_modules/@the-convocation/twitter-scraper/src/profile.ts:192:15)
    at async Scraper.getProfile (/Users/.../node_modules/@the-convocation/twitter-scraper/src/scraper.ts:126:17)

Curious if it's the same issue now as recently it was with 403. We tried adding cycletls and it didn't help.

[vmo0t2xw00t]

Getting the same error as of a day or so ago.

[cjmaxik]

All the GraphQL IDs were changed. Also, the requests now require User-Agent header present.

[karashiiro]

Updated most of the GQL endpoints just now (v0.19.1), there are a bunch of test failures on the user auth test suite I haven't figured out yet, though. Still, this should resolve the 404s, at least. I also added user-agent to the installed headers in the header update methods, just to be sure it's passed consistently.

[MoralCode]

Could this be related to https://devcommunity.x.com/t/deprecation-of-the-1-1-guest-activate-endpoint-in-the-twitter-api/200591?

Seems like this endpoint has been deprecated since 2023 and maybe just got removed?

its used in the code here: https://github.com/the-convocation/twitter-scraper/blob/abbd77f2b0d8c73717988a5c6685dbe63ce84aca/src/auth.ts#L245

[karashiiro]

That seems unlikely, or else it wouldn't be possible to view tweets unauthenticated anymore (unless they introduced a new equivalent, of course).

[MoralCode]

#166 fixed my issue (i was seeing the "that page does not exist" code 34 error as others here)!

Edit: its back Edit edit: ok it seems to be intermittent

[videvian]

I still get {\"errors\":[{\"message\":\"Sorry, that page does not exist\",\"code\":34}]} with v0.19.1 from TwitterGuestAuth.updateGuestToken

[mickael-btc]

It happens sometimes when the header "x-client-transaction-id" is missing. I use this lib to generate it: https://github.com/Lqm1/x-client-transaction-id Seems like they check more and more on headers, "x-xp-forwarded-for" might be the next one to be required

[karashiiro]

Added experimental support for those headers in v0.20.0. I've gated those under options for now, as I couldn't figure out how to test them thoroughly enough to have confidence in the implementations:

const scraper = new Scraper({
  experimental: {
    xClientTransactionId: true,
    xpff: true,
  },
});

XPFF implementation: https://github.com/the-convocation/twitter-scraper/blob/main/src/xpff.ts X-Client-Transaction-Id implementation (uses that library with some copied code/tweaks to keep it unloaded by browsers): https://github.com/the-convocation/twitter-scraper/blob/main/src/xctxid.ts

I've also changed the guest token activation logic to just skip it and continue if it fails, since based on further testing on the website, it seems to work fine if the guest token cookie/header isn't present - it hit 429s almost every time it called /activate.json, and could still pull the TweetResultByRestId endpoint without any obvious issues.

[ghostrick]

I still get 404 with v0.20.0 from getProfile("elonmusk")

[logesh2496]

There's another error after updating to the latest version:

**Error fetching ondemand file: TypeError: fetch failed**
    at node:internal/deps/undici/undici:13484:13
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at ClientTransaction.getIndices (node_modules/x-client-transaction-id/script/transaction.js:129:46)
    at ClientTransaction.initialize (node_modules/x-client-transaction-id/script/transaction.js:85:72)
    at Function.create (node_modules/x-client-transaction-id/script/transaction.js:110:9)
    at generateTransactionId (node_modules/@the-convocation/twitter-scraper/src/xctxid.ts:170:23)
    at requestApi (node_modules/@the-convocation/twitter-scraper/src/api.ts:71:27)
    at TwitterUserAuth.isLoggedIn (node_modules/@the-convocation/twitter-scraper/src/auth-user.ts:243:17)
    at getSearchTimeline (node_modules/@the-convocation/twitter-scraper/src/search.ts:89:9)
    at fetchSearchTweets (node_modules/@the-convocation/twitter-scraper/src/search.ts:54:20) {
  [cause]: ConnectTimeoutError: Connect Timeout Error (attempted address: abs.twimg.com:443, timeout: 10000ms)
      at onConnectTimeout (node:internal/deps/undici/undici:2602:28)
      at Immediate._onImmediate (node:internal/deps/undici/undici:2583:11)
      at processImmediate (node:internal/timers:491:21) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}
**Failed to initialize ClientTransaction: Error: Couldn't get KEY_BYTE indices**
    at ClientTransaction.getIndices (node_modules/x-client-transaction-id/script/transaction.js:146:19)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at ClientTransaction.initialize (node_modules/x-client-transaction-id/script/transaction.js:85:72)
    at Function.create (node_modules/x-client-transaction-id/script/transaction.js:110:9)
    at generateTransactionId (node_modules/@the-convocation/twitter-scraper/src/xctxid.ts:170:23)
    at requestApi (node_modules/@the-convocation/twitter-scraper/src/api.ts:71:27)
    at TwitterUserAuth.isLoggedIn (node_modules/@the-convocation/twitter-scraper/src/auth-user.ts:243:17)
    at getSearchTimeline (node_modules/@the-convocation/twitter-scraper/src/search.ts:89:9)
    at fetchSearchTweets (node_modules/@the-convocation/twitter-scraper/src/search.ts:54:20)
**Error: Couldn't get KEY_BYTE indices**

Any idea how to fix the issue above?

Thanks!

[karashiiro]

@logesh2496 Is that with or without enabling the new options? Also, are you using CF Workers or some other edge runtime (if so, it could be memory-related)?

[karashiiro]

@mickael-btc Did you mean that you've used that lib with the scraper successfully before? Do you have any reference code available I can compare against? Not sure what else might be missing, here.

[mickael-btc]

@karashiiro I have a custom implementation, but here’s what I’ve found so far:

In this repo, you are using the following bearer token: AAAAAAAAAAAAAAAAAAAAAFQODgEAAAAAVHTp76lzh3rFzcHbmHVvQxYYpTw%3DckAlMINMjmCwxUcaXbAN4XqJVdgMJaHqNOFgPMK0zN1qLqLQCF (let’s call this bearer 1).

However, when inspecting requests in the debugger, I noticed Twitter is actually uses another bearer: AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA (let’s call this bearer 2).

From my tests:

• Bearer 1 works for the login (username/password) flow but is unstable for other requests.
• Bearer 2 seems more recent and more stable overall — I use it for all guest and cookie-based requests.
• I only use bearer 1 for the login endpoint, since that route appears to support bearer 1 only.

Before making any request, ensure you generate a guest token with the bearer 2 . Otherwise, you’ll get:

{
  "errors": [
    {
      "message": "Bad guest token",
      "code": 239
    }
  ]
}

I don’t have the code on me right now but should have it once I get my laptop this week. The 404 errors you’re seeing are likely from the new GraphQL routes that only work with bearer 1.

Hope this helps !

[karashiiro]

Very interesting, this sounded like something I had seen before, and it turns out that it was in fact something I had already seen: https://github.com/n0madic/twitter-scraper/blob/296710769dd8d1634ecded7895962b55755b6da7/auth.go#L23

I'll give this a try, thanks! I'm not sure why I never ported this over before, or why it (apparently) wasn't necessary until now.

[karashiiro]

Released more fixes in v0.20.1 to use the alternate bearer token on a few endpoints.

[videvian]

Released more fixes in v0.20.1 to use the alternate bearer token on a few endpoints.

Thank you for the update!

The getProfile still doesn't work. It's 403, and when I apply cycletls, it's 404. I tried new experimental flags, but they fail for me because our project is ESM, and the new code has require.

[karashiiro]

Updated getProfile as well in v0.20.2, will adjust the experimental flags separately

[karashiiro]

Fixed the experimental flags in v0.20.3.

[artificialrobot]

I was able to get things working again for fetching by username with one more change: https://github.com/the-convocation/twitter-scraper/pull/173 It does also require enabling the xClientTransactionId experimental flag. If I get a chance I will try and grab the other URLs but wanted to share this back in case it helps others as it is easy to patch in.

[thetrueyoshifan]

searchTweets also produces 404 errors, I believe for similar reasons, but so far is the only one from our project that encounters 404 now with v0.21.0

EDIT: It does behave more stably with the xClientTransactionId flag. I seem to occasionally still get the error, but it's less than before.