Replace tokens with guest accounts, swap endpoints

[zedeus]

This replaces the old method of using guest tokens with using "guest accounts". For more information see this comment https://github.com/zedeus/nitter/issues/983#issuecomment-1683134811

Live now on nitter.net for testing. If you'd like to test and don't have any guest accounts, feel free to reach out to me on Matrix. A way to acquire guest accounts will be build and shared very soon.

[dawnerd]

RSS returns forbidden, was this disabled?

[veilm]

RSS returns forbidden, was this disabled?

https://github.com/zedeus/nitter/issues/937

[DrSocket]

hi @zedeus, this is amazing, just a question, how is guest_accounts.json formatted?

[zedeus]

It's a JSON array of the raw object you get from guest account flow, looks like this (you can use this one for testing if you want):

[{"user":{"id":1693553468835831800,"id_str":"1693553468835831808","name":"Open App User","screen_name":"_LO_082193WA35n","user_type":"Soft"},"next_link":{"link_type":"subtask","link_id":"next_link","subtask_id":"OpenAppFlowStartAccountSetupOpenLink"},"oauth_token":"1693553468835831808-TBis5qLp17jenKVyu9jeRjnTcTKjpL","oauth_token_secret":"BvbmNeJucYaUAWoEzrSUzTjkWxifoYaqAnjXTevL6ICxU","attribution_event":"signup"}]

[DrSocket]

Great! thanks a lot, it's the format from BANKA's script in an array.

[DrSocket]

I'm getting an error on compilation with this branch though, do you know anything about it? parserutils.nim(39, 13) Error: undeclared field: 'notNull' for type packedjson.JsonNode [type declared in /Users/user/.nimble/pkgs/packedjson-#9e6fbb6/packedjson.nim(115, 3)]

[zedeus]

This happens with Nim versions older than 1.6.14, I'll fix it soon, for now you can work around it by updating to 1.6.14 or preferably 2.0

[pbregener]

For the record: I successfully built a Docker image for aarch64 from this branch and it works like a charm :)

I changed the base image to use alpine 3.18 which has Nim 1.6.14

[DrSocket]

Sorry for the long post, couple of questions just for my own learning:

1. Do you using Android Studio to reverse engineer the android endpoints? If not how do you recommend going about it?
2. Would it be complicated to add an option to use account credentials from a json instead of guest account in case they close the guest accounts endpoint and just have a lot of them. I'm still pretty rusty with nim code, maybe you have recommendations about a good place so start learning so I can actually help contributing to the project (currently use C,C++,python,TS). i.e. (this account is blocked now) -> {"cookie":"_twitter_sess=xyz; kdt=xyz; auth_token=xyz; ct0=xyz; personalization_id="xyz"; twid="xyz, "x-csrf-token": "xyz", "authorization": "Bearer AAAAAAAAAAAAAAAAAAAAAxyz", "x-twitter-auth-type": "OAuth2Session", "x-twitter-active-user": "yes"}

[zedeus]

I used a virtual machine to listen to network requests, but most of the reverse engineering work was done by other folks.

use account credentials from a json instead of guest account in case they close the guest accounts endpoint

I'm not sure what you mean since that's how it already works, storing guest accounts in a file. No endpoints are being used beyond the initial creation of the account, which expires after 30 days. When you say blocked, do you mean you get an error code 88 (rate limit exceeded)? That block goes away 24 hours later.

[DrSocket]

Would you mind sharing what the kind of vm you use for this? something like wireshark or mitmproxy? That's an actual burner account, not a guest account. It would be great if we could switch from guest to burner accounts like PrivacyDevel uses. In case it gets shut down again just have a few burners to still be able to use Nitter while finding a new solution. By locked I mean: "Twitter responded with an error: AuthorizationError: Authorization: Denied by access control: To protect our users from spam and other malicious activity, this account is temporarily locked. Please log in to https://twitter.com to unlock your account."

[zedeus]

I used the free trial from Corellium to run Twitter and inspect network logs. I tried running BlissOS via qemu to no avail first.

I just noticed the thing you posted is a normal account, didn't look close enough. What I said still stands for guest accounts, they don't get blocked though just limited for 24 hours. The problem with burner accounts is you need a lot for any meaningfully big instance, so it's not really feasible especially since they just get locked. It's extremely easy to get guest accounts, just takes a lot of proxies which are easy to get.

[DrSocket]

Great thank you! Yes the problem with guest accounts solution is that twitter can still kill this method no?

[zedeus]

Indeed, but that's the case with every solution. With accounts they can start banning or further restricting, there is no perfect solution.

[felipefalkiner]

How to upgrade to this version? Git clone this branch and run the following commands again?

$ nimble build -d:release
$ nimble scss
$ nimble md

[zedeus]

Correct, but you also need to get one or more guest accounts, otherwise nothing will work

[dawnerd]

I have this running on my instance and so far it's running great, haven't seen any issues so far. Getting gust tokens definitely the trickiest part, but being as Im just running a private/relay instance one guest user should cover it.

Thanks for getting this put together!

[somini]

Is it possible to run this branch with an actual account? What are the tokens I need to extract from the real account to create the JSON file?

[zedeus]

It might be possible but I haven't tried, there's no code in this branch for regular accounts as they use cookies and bearer tokens etc instead of oauth.

[cmj]

Do you happen to know the rate limits for with_replies? I am burning through the limit fast with replies but the timelines seems stable for my private instances. Overall this is working great.

[zedeus]

I know what the rate limit is supposed to be, but what it actually is I don't know. The same question was just asked here: https://github.com/zedeus/nitter/issues/983#issuecomment-1687182471

[cmj]

Thanks. I'll keep it on timelines for now until I can get a proxy system setup.

[felipefalkiner]

There's any guide on how to get guest accounts?

[zedeus]

@felipefalkiner see here: https://github.com/zedeus/nitter/issues/983#issuecomment-1681199357 TL;DR you have to run a script which will create a guest account for you. You can reliably get 1 per day per IP. You should change the last line to console.log(JSON.stringify(account)), then you can put it in a JSON array (looks like [{"user": ...}]) in the guest_accounts.json file Nitter expects in the directory you run it from.

[unixfox]

For local/private instance, I propose to have nitter automatically create guest accounts and store it somewhere in the disk.

I think it should be enough for local usage to have at least just one guest account.

And for bigger instance (public ones), the owner can provide a guest accounts file, like already implemented in this PR.

[zedeus]

That's the plan, except I'll host a guest account pool service that public instances (those in the wiki) can automatically connect to, so they don't need to worry about proxies or anything.

[somini]

@felipefalkiner see here: #983 (comment) TL;DR you have to run a script which will create a guest account for you. You can reliably get 1 per day per IP. You should change the last line to console.log(JSON.stringify(account)), then you can put it in a JSON array (looks like [{"user": ...}]) in the guest_accounts.json file Nitter expects in the directory you run it from.

Thanks for that.

In case anyone know nothing about node:

• Create an empty directory
• Create a file guest.js and paste that script's content (including JSON.stringify change))
• Create a file package.json with the extact contents: {"type":"module"}
• cd to that directory and run node guest

That will create the JSON object. Adapt it to a JSON array (maybe run the script from several IPs?), and add it to guest_accounts.json on nitter's working directory.

[cmj]

Adding onto this, I know next to nothing about node, but was able to get this working passing the proxy host:port to the script. In theory it should work but all proxies I tested today didn't work: https://gist.github.com/cmj/79501b374e387aebe3a717b9e8edda04

And because node seems to still be foreign to me I wanted to break it out into a simple shell script: https://gist.github.com/cmj/6e6f6cae51c28cf6e161ceba8d108dda

Here is the same but with using proxy option: https://gist.github.com/cmj/6cb90eae1ed299ee050a2396e7ceaaaa

[somini]

@cmj Thanks, that's much easier. That script should be included on the repository itself.

[mhkhung]

I have been running this branch with a few tokens for a few days already.. seems to be working well other than hitting "Instance has been rate limited." when viewing individual tweets.. is having more tokens going to solve this problem? Or are they rate-limiting my server IP?

If each token's going to last about a month, we probably should tag them with a generation date so we can rotate them out easily/automatically.