ntscraper icon indicating copy to clipboard operation
ntscraper copied to clipboard

High number of retries detected. Testing all instances... New working instances:

Open MadAim123 opened this issue 1 year ago • 5 comments

I always get this message after a while: High number of retries detected. Testing all instances... New working instances:

It's an infinity loop. Only breaking the script works. There is no new working instances named (and used). Only thousand times the same message.

MadAim123 avatar Nov 01 '23 15:11 MadAim123

Should be fixed, now if there are no working instances it will end the scraping without errors.

bocchilorenzo avatar Nov 01 '23 16:11 bocchilorenzo

Now there is another infinity loop:

Current stats for Bildung+vernetzt: 3 tweets, 0 threads...
Current stats for Bildung+vernetzt: 3 tweets, 0 threads...
Current stats for Bildung+vernetzt: 3 tweets, 0 threads...
Current stats for Bildung+vernetzt: 3 tweets, 0 threads...

Until it reaches the max number of retries or the IP is blocked: All instances are unreachable. Check your request and try again.

Can you try it for your own for the term "Bildung vernetzt"? Parameters:

  • Instance (randomly choosen): https://nitter.ktachibana.party
  • term: Bildung vernetzt
  • since: 2022-12-25

The created link would be: https://nitter.ktachibana.party/search?f=tweets&q=Bildung+vernetzt&since=2022-12-25 The first tweet is always visible (on every page after click on "Load more").

MadAim123 avatar Nov 02 '23 12:11 MadAim123

Perhaps they would stop going down if you stopped scraping them.

zedeus avatar Nov 06 '23 21:11 zedeus

This scraper does not abuse the instances. I limited it to 1 request at least every 2 seconds, you can see it in the code that it sleeps for 1-2 seconds on every search and then another 2 seconds before every retry. I hope such a low request frequency is not an issue.

bocchilorenzo avatar Nov 07 '23 07:11 bocchilorenzo

Same problem with infinite testing on version 0.2.9

scraper.get_tweets('#TESLA', mode='hashtag', number=256)

изображение

thelollipopka avatar Nov 16 '23 01:11 thelollipopka