twscrape
twscrape copied to clipboard
vladkens
Discussion: Accounts survival rate
Over the past month or so I've had issues with keeping accounts reliability logged in. I'm authenticating via username/email and password when I do log in and things tend to work for a while.
I've got a script that runs on a cronjob. I've tried reducing the frequency of running, etc but it is all getting the accounts de-authed eventually.
Because my account is following a few private accounts, the idea of just generating new accounts doesn't work.
Has anyone been able to reliability keep accounts 'alive' without reauthing or code for automated re-auth?
It seems like it's gotten much harder. Previously I was losing a few accounts/day, and in the past few days I think I went through close to 1,000. 👎
Automated re-auth would be great.
@takabinance Same for me - looks like I'm not the only one. Since yesterday my accounts constantly gets banned, and every time I tried to re-login it gets banned. This hardly happened before. Are you experiencing the same?
I have not tried to re-login... it's with a large number of accounts. I'm not sure how I would do this. But, yeah, it seems like something changed very recently.
It seems that X has made some changed in recent days and start suspending accounts more frequently, all of my old and new accounts are all suspended (read only) yesterday.
It's been around a month or so since X has started systematically detecting automated actions and suspending them, even asking for Arkose Challenges as often as every day, I wonder if there's a work around for this, because for now I'm forced to check my account every couple hours for Arkose Challenges so my automation keeps on going... (one workaround I know of is buying premium, which I will not do), any ideas?
@TMDR I believe we are talking about different issues. The one you mentioned has exist for quite some time, while the issue we discussed just emerged a few days ago, when dealing with a large number of accounts.
same here. I got around 3k accounts and used them daily in a rotating system. Around 100 per day. Over 95% are suspended now (read only). Some of them I have not even used yet. I am wondering how they detect such accounts and what we can do to keep them alive...
Has anyone had luck with re-logging in (twscrape relogin
twscrape.accounts_pool:login:162 - Failed to login 'accountname': 429 - {"code":88,"message":"Rate limit exceeded."}
Have tried with multiple accounts, multiple machines and always same
Also, has anyone tried with premium accounts. I'd happily pay for 10-20 accounts to avoid this.
An small update about suspended (read only) account.
Turns out that this kind of account is literally "read only", with no write permission and limited read permission. This account are still able to do reading things like searching or loading user profile, but with a really low threshold on rate limit compared to a normal account.
According to X's docs, the x-rate-limit-limit , x-rate-limit-remaining and x-rate-limit-reset in response headers are supposed to indicate parameters about rate limit, but they are no longer functional under read only mode, which means you can still get rate limited even if x-rate-limit-remaining is not 0, and the account maybe still (reality is very likely) unavailable after x-rate-limit-reset.
I make a small test on SearchTimeline on 5 accounts. I can requests around 11-13 times with 10s interval before the account get limited. I am confident about these accounts' read operation is recoverable because the accounts I test are just dead 2 days ago, but the reset time is uncertain. I will update further research at tomorrow.
A conclusion I can made so far is if your business only needs read operation, then it is theoretically possible to do scraping with a fair large amount numbers of read only account, but be aware of ip banning. And I also want twscrape can make a feature about testing if these read only account are still available or is recovered, instead of just marking them as inactive.
@ExHix Thanks for your investigation! Are you suggesting the rate-limited account can still be used for read-only operation? From my experience very little tweets can be loaded for these accounts, maybe I didn't test it after enough intervals. But looking forward to your further update!
@caterpillar1219 It is theoretically possible at least on searching if my guess is correct, but I need more data about rate limit behavior on read only accounts. If anyone also want to investigate it, this is my script to collect requesting data on X's search API. Also if there is existing research on X's read only account, please share it!
When checking the usage, despite several accounts being added to the db, the same 20 accounts or so get used over and over, has anyone else noticed it? Wouldn't a better rotation between available accounts be better? so each account does actions less often
@BrokedTV There is some accounts pool setting you can modify in the code, to either select available accounts by alphabetic order(default), or random. I tried the later, but unfortunately it does not help. My guess is the rate limit is not only on the account level, but somehow on the proxy level.
@ExHix fwiw, the normal account req limit is 49 - then it is locked for 15min and limit refreshed. But as I looked into those read-only account, it seems when read req reaches 8 or 9, then it is turned into an irreversible "Rate Limit Exceeded" state.
When checking the usage, despite several accounts being added to the db, the same 20 accounts or so get used over and over, has anyone else noticed it? Wouldn't a better rotation between available accounts be better? so each account does actions less often
Add this to your code:
AccountsPool._order_by = "RANDOM()"
The AccountsPool will then randomly select accounts instead of going through by username.
@BrokedTV There is some accounts pool setting you can modify in the code, to either select available accounts by alphabetic order(default), or random. I tried the later, but unfortunately it does not help. My guess is the rate limit is not only on the account level, but somehow on the proxy level.
@ExHix fwiw, the normal account req limit is 49 - then it is locked for 15min and limit refreshed. But as I looked into those read-only account, it seems when read req reaches 8 or 9, then it is turned into an irreversible "Rate Limit Exceeded" state.
Very useful findings! So does this mean if we set the limit to 8 or 9, it's less likely for the accounts to be suspended?
An small update about suspended (read only) account. Turns out that this kind of account is literally "read only", with no write permission and limited read permission. This account are still able to do reading things like searching or loading user profile, but with a really low threshold on rate limit compared to a normal account. According to X's docs, the
x-rate-limit-limit,x-rate-limit-remainingandx-rate-limit-resetin response headers are supposed to indicate parameters about rate limit, but they are no longer functional under read only mode, which means you can still get rate limited even ifx-rate-limit-remainingis not0, and the account maybe still (reality is very likely) unavailable afterx-rate-limit-reset. I make a small test onSearchTimelineon 5 accounts. I can requests around 11-13 times with 10s interval before the account get limited. I am confident about these accounts' read operation is recoverable because the accounts I test are just dead 2 days ago, but the reset time is uncertain. I will update further research at tomorrow. A conclusion I can made so far is if your business only needs read operation, then it is theoretically possible to do scraping with a fair large amount numbers of read only account, but be aware of ip banning. And I also want twscrape can make a feature about testing if these read only account are still available or is recovered, instead of just marking them as inactive.
Thank you for your findings. I set the limit to 5 requests for each 15 mins but unlucky my accounts still got banned.
Thank you for your findings. I set the limit to 5 requests for each 15 mins but unlucky my accounts still got banned.
It may because X has implemented more radical abnormal behavior detection, no matter how low frequency you scrape.
I will update further research at tomorrow. A conclusion I can made so far is if your business only needs read operation, then it is theoretically possible to do scraping with a fair large amount numbers of read only account, but be aware of ip banning.
Based on two days of testing data, I can make a reliable guess that the rate limit reset window for each read only account is 24 hours, and for searching, each account has 11 to 13 times available in every window. Of course, this is not enough for the business which needs always scraping, but is enough for light users. I guess other APIs may have similar behaviors, but I won't dig into it further because the project is bit huge, if anyone is interested in it you can try it.
It seems like it's gotten much harder. Previously I was losing a few accounts/day, and in the past few days I think I went through close to 1,000. 👎
Automated re-auth would be great.
@takabinance Hello, I would like to ask: How to achieve Automated re-auth? Look forward to your answer, thank you very much.
@ExHix Yeah 11~13 req is too few. Normal accounts has 50 req every 15min.
My current configuration is:
- random account selection
- low daily volume - around 20-30 searches per day per account
- an 8 hour delay if limit has been hit (although this rarely happens with random)
- residential rotating proxies
I'm back to losing 1-2% per day over the past few days. Will update in several days to see if this holds.
@takabinance Hello, I would like to ask: How to achieve Automated re-auth? Look forward to your answer, thank you very much.
I actually don't even know how to log back in once. I thought it was a simple twscrape relogin command, but this hasn't worked for me as I get a 429 every time. You can see that 2fa recovery was added to the library (but maybe not documented yet) but haven't dug into it to see how to use it yet.
I saw somewhere else that a header for 'ui_metrics' could have been implemented recently. Twscrape hits the GQL directly hence not scoring high on the UI use - maybe this could be worth investigating?
@takabinance Thanks for sharing! I also notice that I started to lose less accounts since yesterday. I'm not sure if it's because some changes I made(e.g. no. accounts, search freq), or it's the change from Twitter side. When you mentioned 20-30 searches, is it technically 20-30 searches(1 search could be ~10 req), or 20-30 req? Also may I ask how many accounts you are losing per day? like tens or hundreds?
I haven't lost an account in a few days. I'm now only running around 100 accounts and maybe 40-50 requests per account per day (I only get the first page of search results so for me 1 search = 1 request).
I haven't lost an account in a few days. I'm now only running around 100 accounts and maybe 40-50 requests per account per day (I only get the first page of search results so for me 1 search = 1 request).
can I ask how you managed to create so many accounts?
I haven't lost an account in a few days. I'm now only running around 100 accounts and maybe 40-50 requests per account per day (I only get the first page of search results so for me 1 search = 1 request).
can I ask how you managed to create so many accounts?
There are websites selling automatically registered accounts, no need to create manually.
