DistriNet
List generation fails when enabling Google Safe Browsing
I have tried generating 2 lists differing only in checking the box "Only include domains not flagged as dangerous by Google Safe Browsing". The one with the Google Safe Browsing option enabled time-outed.
- Failed list: https://tranco-list.eu/list/NNYWW/300000
- Successful list: https://tranco-list.eu/list/7N7LX/300000
My experience with Google Safe Browsing is that it allows only 10k queries per day, so unless Google granted Tranco an increased quota, this option should not be configurable for so large lists. Or maybe you got an exception from Google and it expired?
Seems like this issue is the same as https://github.com/DistriNet/tranco-list/issues/10, suggesting it is around for 3 years. I just recommend removing "Only include domains not flagged as dangerous by Google Safe Browsing" checkbox. In our crawls, we anyway run the safe browsing check just before the crawl and we typically filter less than 1 website per 10k of list (when using CrUX), it is not so important feature.