WebbCompare icon indicating copy to clipboard operation
WebbCompare copied to clipboard

Not able to scrape

Open Emilianocm23 opened this issue 3 years ago • 7 comments

Hello I think this is not working well, its not able to scrape by Author/Hashtag I dont know how to code but i will donate to thank the creator of this fantastic open source bot

Emilianocm23 avatar Nov 09 '20 01:11 Emilianocm23

Hi, Yes I am currently in the process of updating the code to support the latest version of the TikTokAPI. Some things are broken for the latest version of TikTokAPI 3.6.2, I will be fixing it shortly, but in the meantime if you want it to work install TikTokAPI 3.5.8

HA6Bots avatar Nov 09 '20 12:11 HA6Bots

I cant get it to work, installed 3.5.8 version but its not woirking for me. I guess i will wait for update, thank you

Emilianocm23 avatar Nov 09 '20 18:11 Emilianocm23

well, you dont have to wait for the Update by @HA6Bots the error is very straight forward and its very easy to fix in 3 or 4 lines of code ... you just have to read the code,

The new tiktokapi requires us a user agent and a cookie with the same identity, so each time request something it has to have the same cookie, so the problem its mainly on the way we request the data

@HA6Bots implemented a request type using python requests with these headers [('User-agent', 'okhttp'), ('referer', 'https://www.tiktok.com/')]

You just have to edit the headers on the request to fit the random generated cookie which is in this case the DID interpolated variable and the user agent

headers={
                "Accept": "*/*",
                "Accept-Encoding": "identity;q=1, *;q=0",
                "Accept-Language": "en-US;en;q=0.9",
                "Cache-Control": "no-cache",
                "Connection": "keep-alive",
                "cookie": "tt_webid_v2=" + b.did,
                "Host": url.split("/")[2],
                "Pragma": "no-cache",
                "Range": "bytes=0-",
                "Referer": "https://www.tiktok.com/",
                "User-Agent": b.userAgent,
            },

or just use get_bytes or get_video_by_url as the api tells you... its up to you

Arukaito avatar Nov 10 '20 03:11 Arukaito

On what file? tiktok.py?

the variable "b.did" whats its value?

Emilianocm23 avatar Nov 10 '20 03:11 Emilianocm23

b.did its a random integer as specified in #311

did = str(random.randint(10000, 999999999))

Arukaito avatar Nov 10 '20 03:11 Arukaito

@Arukaito Thank you. Yes sorry guys - been extremely busy lately, I should be able to get it updated by the end of this week.

HA6Bots avatar Nov 10 '20 11:11 HA6Bots

API Fix! Please try again now with new server code.

HA6Bots avatar Jan 26 '21 22:01 HA6Bots