CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)

[linglin506]

This error will be reported from 0:00 on September 18, 2020. It is found through debug that the https://twitter.com/i/search/timeline page does not exist. Can you fix it?By changing the IP, it can be determined that Twitter is not banning the IP

CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs

[sukioral]

I experienced the same problem today as well. Yesterday was still fine.

[zhixideyu]

今天用就有事

[hcanalesmx]

I am having the same issue

[bigslick87]

Glad to see I am not the only one. I just discovered this today and was excited to use it. I hope they can fix it ASAP

[hj5992]

getting the same error!

[gab-santi]

Getting the same error as well. Hope it gets fixed ASAP

[tmantynen]

I guess Twitter has deleted https://twitter.com/i/search/timeline

Twint is not the only Twitter scraper having the same problem. See: https://github.com/jonbakerfish/TweetScraper/issues/92

[Buaasinong]

bad news

[zhixideyu]

我想Twitter已经删除了https://twitter.com/i/search/timeline

Twint并不是唯一遇到相同问题的Twitter抓取工具。请参阅：jonbakerfish / TweetScraper＃92

Does that mean the project is going to fail

[yemen2016]

I am also having the same issue. The system was working fine yesterday! I hope the problem will be solved soon.

[Sebastokratos42]

I guess Twitter has deleted https://twitter.com/i/search/timeline

Twint is not the only Twitter scraper having the same problem. See: jonbakerfish/TweetScraper#92

Problem also appears using GetOldTweets3.

[shelu16]

https://twitter.com/search?f=live&q=bug%20min_retweets%3A100%20until%3A2020-09-18%20since%3A2020-09-11&src=typed_query but it can work can we do this search here

[vishal7892]

Even I am facing the same issue. Any luck?

[TryAndErr0r]

We need to find a new endpoint. I loaded the site and saw this link New Api link?. It is important to note that if you copy the request and past it to notepad you can see that the request is provided with headers and you can see that it provides a x-csrf-token and x-guest-token which are required. You can find these tokens in the cookie fields "gt" and "ct0".

This is somewhat the official twitter api and you need a bearer token to access this. The bearertoken is provided in the https://abs.twimg.com/responsive-web/client-web/main.6b1bdbc5.js file. Just search for "AAAAAA" and you can find it.

Now you can craft a GET request in postman with the x-csrf-token, the x-guest-token and the bearer token for authentification.

[yunusemrecatalcam]

Booom! I found a new endpoint, implemented the new parser for it. Check this out! https://github.com/twintproject/twint/pull/917 @pielco11

[estebanpdl]

Having the same issue as well. Just tried yesterday (2020-09-17) and everything worked fine. I tried Twint today (2020-09-18) and I got the same loop.

CRITICAL:root:twint.get:User:
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 15 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 15 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 27.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 64.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 125.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 216.0 secs

[boshez]

I thought I was the only person who was facing this. Hope they fix this.

[hasobi]

i'm having the same problem here.

[guilherme-sal]

Same problem here. Yesterday was working fine...

[jeremy1392]

Everything is working except to fetch tweets. Yesterday was working fine

[ivanlewin]

Hi I just want to add that I also had no problems yesterday but today I can't get any data.

Also, I was trying other tool to see if it worked and they seem to be having similar issues

[aradiayu]

I think I'm the only one facing this 'twint' problem. I hope it is fixed soon

[Dbroug01]

I am also facing this issue

[tmantynen]

Booom! I found a new endpoint, implemented the new parser for it. Check this out! #917 @pielco11

Big thanks to @yunusemrecatalcam

I'm not using Twint but instead my own similar Twitter scraper program writen in PHP. So I can't comment directly on @yunusemrecatalcam 's changes to Twint.

Anyway. Couple of days ago my own scraper stopped working just like all Twitter scrapers using i/search/timeline endpoint did. Following @yunusemrecatalcam 's example, I implemented similar changes to my program that @yunusemrecatalcam did to Twint in PR #917.

I haven't had the time to finished all changes, but preliminary impressions are mixed. The new mobile endpoint appears to function similarly to the old one, but Twitter appears to return a lot fewer tweets as a result to queries than previously. Queries that resulted hundreds of tweets earlier now result just tens of tweets. I can't yet comment on speed or rate limits etc. And again, these are just preliminary impressions.

(Edit: I incorrectly tagged @pielco11 when I should have tagged @yunusemrecatalcam. My apologies.)

[adrival13]

Same problem CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 15 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 15 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 27.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 64.0 secs

[rayhannr]

got the same problem though fetching followers and followings are just fine. anyway, is there any good alternative to twint?

[shizia]

今天用就有事

以前是都能用吗？才刚发现这个就发现好像出问题了

[lukaspistelak]

they changed end point url to :+1:

https://twitter.com/explore/

[akuma527]

they changed end point url to 👍

https://twitter.com/explore/

What do you mean exactly? How to get it to run?

[estebanpdl]

they changed end point url to 👍

https://twitter.com/explore/

isn't it /explore/ the URL for trending topics?

[fwgood]

今天用就有事

以前是都能用吗？才刚发现这个就发现好像出问题了

是的，我用了几个月了，前几天突然就不行了

[fwgood]

they changed end point url to 👍 https://twitter.com/explore/

isn't it /explore/ the URL for trending topics?

@estebanpdl “/explore” indeed is the trending topics，but you can use advanced search in this page. If you switch to the second tab you will fetch the same result as before. hopefully there are someone can write the new parser ASAP

[zhixideyu]

How on earth can I crawl my way to a tweet, which is important to me。

[imfht]

@zhixideyu looks like twitter api is a way.

[zhixideyu]

Please recommend a project that can use

[lukaspistelak]

for JS render here is nice lib :+1:

https://github.com/mherrmann/selenium-python-helium

[rodolfopardo]

Twint is broken? OMG

Any solution??

[tahmidrashid]

Twint is broken? OMG

Any solution??

Try this: https://github.com/twintproject/twint/pull/917

[shadenshabayek]

I am facing the same issue too!

[ETedward]

also still facing the same issue

[DavidPerea]

What is the solution for this?

[simahashemi]

the same for me. Is there any solution or do you know any other scrapper as good as Twint?

[Wendy-Chang]

same here! I'm struggling.

[Tasaqi]

same for me..

[eabanoz]

Hi, I would like to use twint package (2.1.20) with Python 3.8 via pycharm IDE and I have gotten 'CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs'. I couldn't find a solution about it. PS: I have already read all posts which are related with this issue.

[jiwoochoi-kor]

I'm facing the same issue...

[hnodaro]

Any idea to resolve this issue ?

[himanshudabas]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

[hnodaro]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Thank you so much !!

[estebanpdl]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Does this version install the command-line tool as well?

[himanshudabas]

@estebanpdl Yes it does!

[estebanpdl]

@himanshudabas Great. Just saw the warning about twint.exe. Thanks so much.

[fattoooomah]

We need to find a new endpoint. I loaded the site and saw this link New Api link?. It is important to note that if you copy the request and past it to notepad you can see that the request is provided with headers and you can see that it provides a x-csrf-token and x-guest-token which are required. You can find these tokens in the cookie fields "gt" and "ct0".

This is somewhat the official twitter api and you need a bearer token to access this. The bearertoken is provided in the https://abs.twimg.com/responsive-web/client-web/main.6b1bdbc5.js file. Just search for "AAAAAA" and you can find it.

Now you can craft a GET request in postman with the x-csrf-token, the x-guest-token and the bearer token for authentification.

Thank you.

Can you give an example on how to to do it? Can we use the endpoint 'https://twitter.com/search?' with your suggestion?

Thank you

[YesimSem]

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This works!!

[carlosiruiz]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

thank you so much!!! it worked

[eabanoz]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Thank you so much. It worked for me :)

[lhfbc]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Does this version install the command-line tool as well?

I have the same question. After reinstallation, I cannot find the command 'twint'. Any body can help me ?

[lhfbc]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Does this version install the command-line tool as well?

I have the same question. After reinstallation, I cannot find the command 'twint'. Any body can help me ?

Ubuntu 18.04 Python 3.6.9

[lhfbc]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Does this version install the command-line tool as well?

I have the same question. After reinstallation, I cannot find the command 'twint'. Any body can help me ?

Ubuntu 18.04 Python 3.6.9

DONOT uninstall twint， just upgrade will be OK!

[zelinnn]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Does this version install the command-line tool as well?

I have the same question. After reinstallation, I cannot find the command 'twint'. Any body can help me ?

Ubuntu 18.04 Python 3.6.9

DONOT uninstall twint， just upgrade will be OK!

Thank youu!!

[muHashh]

I'm having the same issue even after installing this one: git+https://github.com/twintproject/twint.git@origin/master#egg=twint

[spongycode]

I'm having the same issue even after installing this one: mgit+https://github.com/twintproject/twint.git@origin/master#egg=twint

Try this sequence once, it worked for me on Google Colab.

!pip install twint !pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint import twint import nest_asyncio c = twint.Config() c.Username = "github" c.Search = "code" nest_asyncio.apply() twint.run.Search(c)

[aurigandrea]

I got the same issue and the previous solution doesn't work now

[aravind-naidu]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Thank you so much !!

This isn't working as of today!

[waynemaranga]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

This worked. In case it throws an error in a virtual environment, run again without --user

[juheesinghal9]

This worked :)

[miadp]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Thank you, it worked for me!

[wildangunawan]

Jupyter notebook workaround for this issue:

!pip3 uninstall twint -y
!pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

Then continue as usual

import twint
import nest_asyncio
nest_asyncio.apply()

c = twint.Config()
c.Search = "code"
c.Limit = 10

twint.run.Search(c)

EDIT: This somehow doesn't work in Colab. Use Kaggle Notebook instead. You only need to install the package anyway, no need to uninstall it first. So the script will look like this

!pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

import twint
import nest_asyncio
nest_asyncio.apply()

c = twint.Config()
c.Search = "code"
c.Limit = 10

twint.run.Search(c)

[debjyoti003]

after installing the new one, I am getting

~/.local/lib/python3.7/site-packages/twint/format.py in Tweet(config, t) 21 output = output.replace("{hashtags}", ",".join(t.hashtags)) 22 output = output.replace("{cashtags}", ",".join(t.cashtags)) ---> 23 output = output.replace("{replies}", t.replies_count) 24 output = output.replace("{retweets}", t.retweets_count) 25 output = output.replace("{likes}", t.likes_count)

TypeError: replace() argument 2 must be str, not int

[debjyoti003]

Is there any specification for installing it on ubuntu?

[wildangunawan]

Is there any specification for installing it on ubuntu?

I have no problem installing it on Ubuntu 16.04 under Conda environtment. I used this command:

pip install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

[vesuvius13]

I got the same issue and the previous solution doesn't work now

create a new virtual environment in anaconda and install python 3.6 in there. conda create -n yourenvname python=3.6 anaconda Then, follow these steps:-

git clone https://github.com/twintproject/twint.git cd twint pip3 install . -r requirements.txt pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint

Then try to run the script:-

!pip install nest_asyncio import nest_asyncio nest_asyncio.apply()

import twint config = twint.Config() config.Search = "Dogecoin" config.Limit = 10 config.Store_csv = True config.Output = "doge.csv" twint.run.Search(config)

[romellfudi]

Hi everyone, when I run the following command:

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

Does it work on MAC OS? I have been trying to use it since 2019

[lemiemie]

Hi everyone, when I run the following command:

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

Does it work on MAC OS? I have been trying to use it since 2019

@romellfudi Yes it works on my macOS. It gave me error message at the beginning but after I deleted the "--user" part it works

[zachlagden]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

thank you so much!!! it worked

worked for me

[hglite69]

Me Too ^-^

[jpblopez]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

thank you so much!!! it worked

Also worked for me, as of September 2021

[asdrubalivan]

^ This worked for me too!

[tsaaii]

upgrading twint works.

[yehfanju]

pip3 uninstall twint

@tsaaii can you normally scrape all the tweets without encountering limit/date constraints? Thanks!

[tsaaii]

it limits tweets on time line as mentioned in documentation. I haven't seen any date constraints so far.

[yehfanju]

@tsaaii Thanks for your reply! Did you clone the git repo as well or just pips upgrade? I can only get a few tweets after uninstall and reinstall.

[tsaaii]

clone the repo first, upgrade it next. It works for me. Make sure python version is 3.6.

[yehfanju]

@tsaaii Thanks! the 3.6 version and upgrade works for me but only for the latest tweets. Twint does not return anything when specifying since and until. Just want to know if you also cannot retrieve historical tweets or it's just me!

[olivermj3]

This issue fixed after running this command: pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

However, following this upgrade, it resumed collect briefly (16 tweets recovered then the following print out before the error:

[!] No more data! Scraping will stop now. found 0 deleted tweets in this search.

RefreshTokenException Traceback (most recent call last) /var/folders/7z/866jzymn52zcbnymflj7xhbw0000gn/T/ipykernel_6392/906502173.py in 7 filename = str(str(target)+".csv") 8 c.Output = filename ----> 9 twint.run.Search(c)

~/.local/lib/python3.8/site-packages/twint/run.py in Search(config, callback) 408 config.Followers = False 409 config.Profile = False --> 410 run(config, callback) 411 if config.Pandas_au: 412 storage.panda._autoget("tweet")

~/.local/lib/python3.8/site-packages/twint/run.py in run(config, callback) 327 raise 328 --> 329 get_event_loop().run_until_complete(Twint(config).main(callback)) 330 331

~/.local/lib/python3.8/site-packages/twint/run.py in init(self, config) 34 # USAGE : to get a new guest token simply do self.token.refresh() 35 self.token = token.Token(config) ---> 36 self.token.refresh() 37 self.conn = db.Conn(config.Database) 38 self.d = datelock.Set(self.config.Until, self.config.Since)

~/.local/lib/python3.8/site-packages/twint/token.py in refresh(self) 67 else: 68 self.config.Guest_token = None ---> 69 raise RefreshTokenException('Could not find the Guest token in HTML')

RefreshTokenException: Could not find the Guest token in HTML

[fwgood]

已收到您的邮件，会尽快回复

[rangersmyth74]

!pip install twint !pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint import twint import nest_asyncio c = twint.Config() c.Username = "github" c.Search = "code" nest_asyncio.apply() twint.run.Search(c)

Hey all... I am having this problem now in Dec 2021. I have tried all the fixes, but none worked for me. I used Google Cloud Shell and I was able to get twint to work, but then I was getting the same errors.

I was wondering how to use the above date? Do I run it on the command line? Just wondering.

Thanks.

[machlovi]

I am facing the same issue ..I have tried all the solutions. Can anyone help me

[fwgood]

已收到您的邮件，会尽快回复

[ArshdeepSandhu]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Thank you very much Himanshu! Was really struggling with this.

[ArshdeepSandhu]

I am facing the same issue ..I have tried all the solutions. Can anyone help me

Have you tried what Himanshu suggested?

[skswldndi]

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Thank you very much Himanshu! Was really struggling with this.

This worked for me! Thanks.

[fwgood]

已收到您的邮件，会尽快回复

[palavaryunusemre]

@eabanoz @jiwoochoi-kor @hnodaro Aşağıdaki komutu kullanarak önceki twint kurulumunu kaldırın

pip3 uninstall twint

Ardından aşağıdaki komutu kullanarak twint'i yeniden yükleyin

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

Bu çalışmalı.

Himanshu'ya çok teşekkür ederim! Bu konuda gerçekten zorlanıyordu.

this didn't work for me

[fwgood]

已收到您的邮件，会尽快回复

[palavaryunusemre]

已收到您的邮件，会尽快回复

thanks