Error: twint.token.RefreshTokenException: Could not find the Guest token in HTML

[weslleylira]

Issue Template

Error: twint.token.RefreshTokenException: Could not find the Guest token in HTML

Initial Check

If the issue is a request please specify that it is a request in the title (Example: [REQUEST] more features). If this is a question regarding 'twint' please specify that it's a question in the title (Example: [QUESTION] What is x?). Please only submit issues related to 'twint'. Thanks.

Make sure you've checked the following:

• [] Python version is 3.6;
• [] Updated Twint with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;
• [] I have searched the issues and there are no duplicates of this issue/question/request.

Command Ran

Please provide the exact command ran including the username/search/code so I may reproduce the issue.

Description of Issue

when running Twint I get the following error: raceback (most recent call last):

File "twitter.py", line 18, in twint.run.Search(c) File "/home/ssm-user/.local/lib/python3.8/site-packages/twint/run.py", line 410, in Search run(config, callback) File "/home/ssm-user/.local/lib/python3.8/site-packages/twint/run.py", line 329, in run get_event_loop().run_until_complete(Twint(config).main(callback)) File "/home/ssm-user/.local/lib/python3.8/site-packages/twint/run.py", line 36, in init self.token.refresh() File "/home/ssm-user/.local/lib/python3.8/site-packages/twint/token.py", line 68, in refresh raise RefreshTokenException('Could not find the Guest token in HTML') twint.token.RefreshTokenException: Could not find the Guest token in HTML

Environment Details

Using Ubuntu Server

[himanshudabas]

this seems to me a duplicate of #957

[aburak256]

I have a similar issue. I am trying to collect some data with twint for time series analysis. To make that, I send searches hour by hour starting from january 2014. After 1000 search I get the same error.

I am using Colab [] Updated Twint with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;

[BonfaceKilz]

Getting the same error. Anyone able to come with a work-around?

[orestislampridis]

Is there a general problem with Twint currently? I seem to get the same error.

[theshouryagupta]

I am having the same issue right now. It was working fine 2 hours ago.

[BonfaceKilz]

I speculate there's yet another internal change within twitter that we aren't aware of...

[theshouryagupta]

Yeah @BonfaceKilz could be a possibility. Which os were you using ? I encountered this on Ubuntu 20.04.01 LTS.

[BonfaceKilz]

@theshouryagupta I'm running ArchLinux; though I've been running twint in a Guix[0] container

[0] https://github.com/pjotrp/guix-notes/blob/master/CONTAINERS.org

[orestislampridis]

I am using Windows 10. I don't think this is an os related problem.

[data-z]

I am getting the same error. It was working fine last night. The I tried to clone the branch form above -himanshudabas commented 9 days ago this seems to me a duplicate of #957 and am getting a different error

[theshouryagupta]

@orestislampridis yes you are right. I tried it on macOS too. Same issue...

[data-z]

how do we install the new commits?

[sukioral]

Getting same problem again

[orestislampridis]

@data-z "pip uninstall twint" and then "pip install git+git://github.com/ajctrl/twint@patch-1". It works for me now!

[sukioral]

RefreshTokenException: Could not find the Guest token in HTML

[sukioral]

@data-z "pip uninstall twint" and then "pip install git+git://github.com/ajctrl/twint@patch-1". It works for me now!

Works for me! Thanks!

[data-z]

@sukioral Thanks works for me too.

[xiaozhouliu]

@orestislampridis Thanks, it works!

[irisdemented]

@data-z "pip uninstall twint" and then "pip install git+git://github.com/ajctrl/twint@patch-1". It works for me now!

This worked for me too! Thank you 👍

[WjcSoso0928]

@data-z "pip uninstall twint" and then "pip install git+git://github.com/ajctrl/twint@patch-1". It works for me now!

I use this method, but I get new error Error Code:

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.twitter.com:443 ssl:True [Network is unreachable]

I don't know how to fix it

[patrickh217]

I tried installing it with pip install git+git://github.com/ajctrl/twint@patch-1 but it did not work at first. Then I simply added the line code directly in the script that @ajctrl changed in his pull request. after this little change, it is working for me.

Many thanks

[data-z]

Thats what I did as well.

Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Androidhttps://aka.ms/ghei36

---

From: Patrick Hermann [email protected] Sent: Thursday, December 17, 2020 3:25:23 AM To: twintproject/twint [email protected] Cc: Carter, Tariq - cartto04 [email protected]; Mention [email protected] Subject: Re: [twintproject/twint] Error: twint.token.RefreshTokenException: Could not find the Guest token in HTML (#1061)

I tried installing it with pip install git+git://github.com/ajctrl/twint@patch-1 but it did not work at first. Then I simply added the line code directly in the script that @ajctrlhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fajctrl&data=04%7C01%7Ccartto04%40uwgb.edu%7C4d652d98472a4573d28508d8a2654d16%7C7fc34f9d1f754f96b5b33cdcaab03aea%7C0%7C0%7C637437903266564493%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CBiA7j4CvnxT285U%2F98QIngBvslOEtZTBcKnxeD5Nk8%3D&reserved=0 changed in his pull requesthttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftwintproject%2Ftwint%2Fpull%2F1075%2Fcommits%2F48e8586cb0a49c93a712b86e0824f7db10ec8f35&data=04%7C01%7Ccartto04%40uwgb.edu%7C4d652d98472a4573d28508d8a2654d16%7C7fc34f9d1f754f96b5b33cdcaab03aea%7C0%7C0%7C637437903266574446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=waNn55EooENCeRhhfXSjHKU5Ygh3MV4jhJwY8K1EXQI%3D&reserved=0. after this little change, it is working for me.

Many thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftwintproject%2Ftwint%2Fissues%2F1061%23issuecomment-747289495&data=04%7C01%7Ccartto04%40uwgb.edu%7C4d652d98472a4573d28508d8a2654d16%7C7fc34f9d1f754f96b5b33cdcaab03aea%7C0%7C0%7C637437903266574446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QtHH67eB1QZAQeBN0xC5Myvj7neKKrPFJAB1X%2Fn8ZhM%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQ3FBIZVCFIE2EQQJTR65OLSVG57HANCNFSM4UQPMRBA&data=04%7C01%7Ccartto04%40uwgb.edu%7C4d652d98472a4573d28508d8a2654d16%7C7fc34f9d1f754f96b5b33cdcaab03aea%7C0%7C0%7C637437903266574446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xSOFH7XWykyuldKtIcwdQq6tuTSRpxOLc2UF8Ja3dcI%3D&reserved=0.

[Vickycats]

Hi Please could someone give a command line instruction for installing the current version with the patch since pip install git+git://github.com/ajctrl/twint@patch-1 no longer works. I have manually added the new version of Token.py from git, without doing a full Twint re-install but I am getting inconsistent results. For example in the code below, the output works fine if the Username is 'realdonaldtrump' or 'kamalaharris' but I get errors for other (existing) users. For example if Username is 'sainsburys' (a large superstore chain in the UK), or most other names I get a keyerror on the url:

_usr.url = ur['data']['user']['legacy']['url'] KeyError: 'url'

Less often it fails with 'cannot find the Guest token'.

Code used:

c = twint.Config() c.Username = 'realdonaldtrump' # works for some not for others. Most not working. c.Store_object = True c.Store_object_users_list = [] c.User_full = True twint.run.Lookup(c)

[rutvikprajapati]

@Vickycats I manually added the new version of Token.py and now got the below error

CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1****

[subasish]

Finding similar error code after updating token.py

[hockeybro12]

@rutvikprajapati I get the same error as you.

[rutvikprajapati]

I found this article on snscrape library and it is working fine.

[BonfaceKilz]

rutvikprajapati [email protected] writes:

I found this article on snscrape library and it is working fine.

FWIW, at one point IIRC, snscrape faced this similar issue; but in a different flavour: https://github.com/JustAnotherArchivist/snscrape/issues/110

-- Bonface M. K. https://www.bonfacemunyoki.com Humble GNU Emacs User / Bearer of scheme-y parens Curator: https://upbookclub.com / Twitter: @BonfaceKilz GPG Key: D4F09EB110177E03C28E2FE1F5BBAE1E0392253F

[mpucci92]

After updating the token.py script with:

class Token: def init(self, config): self._session = requests.Session() self._session.headers.update({'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0'}) self.config = config self._retries = 5 self._timeout = 10

It works now, Thanks!

[agombert]

Same problem on local, even after updating with the patch mentioned above. Anyone got a solution?

[olof98johansson]

Same problem on local, even after updating with the patch mentioned above. Anyone got a solution?

For me, the above update didn't work either, but uninstalling twint and then install via

pip install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

worked now!

[KazimTibetSar]

Same problem on local, even after updating with the patch mentioned above. Anyone got a solution?

For me, the above update didn't work either, but uninstalling twint and then install via

pip install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

worked now!

at python 3.9.0 i wrote pip install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint but it didnt worked

[olof98johansson]

Same problem on local, even after updating with the patch mentioned above. Anyone got a solution?

For me, the above update didn't work either, but uninstalling twint and then install via pip install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint worked now!

at python 3.9.0 i wrote pip install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint but it didnt worked

Yeah ok :( I use python 3.7.4

[mengyao-liu]

I use Python 3.9 on a Mac and I have solved this problem by maybe one of the following two steps:

In twint/token.py, change _self.session.headers.update({'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0'}) to self._session.headers.update({'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:85.0) Gecko/20100101 Firefox/85.0'})

then uninstall and reinstall twint.

It works for me!

[DenisOgr]

Hi all. I got this error only when running my code in the Google Cloud Function. On my Mac, it works well. I fixed it similarly as described above, changed the _self.session.headers. But I changed it to a Linux browser, not Mac or Windows. I changed:

self.token._session.headers.update({'User-Agent': 'Mozilla/5.0 (X11; Linux ppc64le; rv:75.0) Gecko/20100101 Firefox/75.0'})

I guess the Twitter platform checks the client operating system and if it differs from session.headers, it raises an error. My code works well without twint.token.RefreshTokenException on Linux (on GCP) and Mac platforms.

[konnextv]

Heads up: @DenisOgr 's method works for me inside a Google Cloud Function, but only using python37 as the environment.

EDIT: nevermind, doesn't work with 3.7 either, still getting "Could not find the Guest token in HTML"

[HAKANMAZI]

still same error

raise RefreshTokenException('Could not find the Guest token in HTML') 2021-04-18T21:50:03.559085+00:00 app[web.1]: twint.token.RefreshTokenException: Could not find the Guest token in HTML 2021-04-18T21:50:03.562041+00:00 app[web.1]: 10.47.235.178 - - [18/Apr/2021:21:50:03 +0000] "POST / HTTP/1.1" 500 290 "https://snscrape.herokuapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"

[himanshudabas]

Hi, please check this comment. also you can go through the thread of this comment to get a better understanding of why and when this Guest token issue occurs. I have also mentioned a workaround in the comment.

[prhbrt]

@data-z "pip uninstall twint" and then "pip install git+git://github.com/ajctrl/twint@patch-1". It works for me now!

This patch is pulled into twint now. https://github.com/twintproject/twint/pull/1075

[Brata11]

This is the error I am getting using Jupyter Notebook

Collecting git+git://github.com/ajctrl/twint@patch-1 Cloning git://github.com/ajctrl/twint (to revision patch-1) to /tmp/pip-req-build-7ew6pbd5 Running command git clone -q git://github.com/ajctrl/twint /tmp/pip-req-build-7ew6pbd5 WARNING: Did not find branch or tag 'patch-1', assuming revision or ref. Running command git checkout -q patch-1 error: pathspec 'patch-1' did not match any file(s) known to git. WARNING: Discarding git+git://github.com/ajctrl/twint@patch-1. Command errored out with exit status 1: git checkout -q patch-1 Check the logs for full command output. ERROR: Command errored out with exit status 1: git checkout -q patch-1 Check the logs for full command output.

[magikarp171]

This is the error I am getting using Jupyter Notebook

Collecting git+git://github.com/ajctrl/twint@patch-1 Cloning git://github.com/ajctrl/twint (to revision patch-1) to /tmp/pip-req-build-7ew6pbd5 Running command git clone -q git://github.com/ajctrl/twint /tmp/pip-req-build-7ew6pbd5 WARNING: Did not find branch or tag 'patch-1', assuming revision or ref. Running command git checkout -q patch-1 error: pathspec 'patch-1' did not match any file(s) known to git. WARNING: Discarding git+git://github.com/ajctrl/twint@patch-1. Command errored out with exit status 1: git checkout -q patch-1 Check the logs for full command output. ERROR: Command errored out with exit status 1: git checkout -q patch-1 Check the logs for full command output.

I am getting the same error here

[febrizky]

Hi, I'm still getting this error RefreshTokenException: Could not find the Guest token in HTML, try to change __self.session.headers based on here still doesn't work. Does anybody know which user agent string that I should use for Windows 10 x64, and Google Chrome version 96.0.4664.110 ? Thank you.

[philliplumod]

Same problem on local, even after updating with the patch mentioned above. Anyone got a solution?

For me, the above update didn't work either, but uninstalling twint and then install via pip install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint worked now!

at python 3.9.0 i wrote pip install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint but it didnt worked

still not fixt for me python 3.8

[M4573RN04H]

Has anyone found a workaround yet? It randomly works 50% of the time 10% of the time. The other 90% it does not work 100% of the time.

[moxak]

I modified token.py and it worked without error. please try it below.

1. download token.py via fixed token.py
2. open directory of twint lib
3. move new one to twint lib dir
4. replace it

[postscript] Since it appeared that getting the guest token using html was failing, I tried to get the guest token using another method.

Even if it works, there may be a slight change in behavior of twint.

[kk0walski]

1. fixed token.py

It worked for me.

[moxak]

added lines for catching RefreshTokenException

[M4573RN04H]

If anyone still has an issue there is a whole thread with multiple solutions here #1320

[ljhOfGithub]

What's the use of this exception? Can I comment it out？ '# raise RefreshTokenException('Could not find the Guest token in HTML')'