No more posts while gathering *lot* of posts based on hashtags

[mandys]

Describe the bug The script works fine if I am trying to pull around 500 hashtags ( most of the time ). But, if the hashtags have let's say about 1000+ results, the response is unpredictable. The script is looping fine, 30 results at a time.

For my use case: the hashtag has more than 10,000+ results. I have set number=1000. Every single time it's looping, it stops around - https://m.tiktok.com/api/challenge/item_list/?challengeID=&count=30&cursor=800&aid=1988&verifyFp=&_signature=

{ "statusCode": 0, "cursor": "800", "hasMore": false }

I think in TikTok.js it reaches here if ((updatedApiResponse && !result.itemList) || (!updatedApiResponse && !result.items)) { throw new Error('No more posts'); }

In this scenario -

1. The script should return the results since it went through 26 cycles before encountering no itemList. At least whatever it scraped should be returned ? Right now it just aborts saying "No more posts".

2. Why is this stopping around 799 ( total ) results when the hashtag has 10,000+ results. In fact even on tiktok web interface, I wasn't able to consistently scroll past 500 odd results. They stopped loading post that. So, does tiktok not give back all the results ? Only upto a particular date ? Any idea ?

If fields below will be empty then issue will be Ignored and Closed

To Reproduce Steps to reproduce the behavior:

• Used OS [ Mac OS X ]

• NodeJs Version (node -v) [ 12.16.1]

• Scraper Version (tiktok-scraper --version) [1.4.36]

• CLI or Module [ Module ]

• Used method TikTokScraper.hashtag

• Used options const posts = await TikTokScraper.hashtag(hashtag, { number: 700, sessionList: ['sid_tt=123;'], });

• Do you run the scraper through Proxy, VPN, VPS? VPN and also through an AWS box

• If scraping from a profile. Did you checked if profile is Public? Scraper won't work with Private profiles! [ Scraping hashtags which are public ]

• Preferably provide URL to the profile,hashtag,video and etc page https://www.tiktok.com/tag/pandya

Screenshots If applicable, add screenshots to help explain your problem.