domains
domains copied to clipboard
tb0hdan
World’s single largest Internet domains dataset
Yesterday at 18:17 CEST we noted a SYN flood caused by the project crawler. Please implement request limits.
Hey tb0hdan, glad to see that you're still up and about, I've found a large source of domains at https://ipsniper.info/numbers.html, apparently they have around 286 million domains and it's being...
Hi! You may be interested by this dataset: https://github.com/etalab/noms-de-domaine-organismes-publics If I get my shell-fu right (`comm -13 theirs mine | wc -l`), it may contain 22673 domains that you don't...
It is sometimes interesting to see how much size those domains take[^1]. It would be cool if you guys keep track this total size, in gigabytes, of all data packed/unpacked...
I found a website with large lists of newly registered domains updated on a daily basis, perhaps you would consider integrating it into your collection: https://www.cubdomain.com/domains-registered-dates/1 You'll need to do...
Hi, Thanks for this nice project. I found that some domains are missing from your dataset, such as "aaaa.com" or "azjj.com". Is there a reason for this? Thanks.
Downloading these places a significant load on servers, and most are not going to contain URL metadata of use to the project. This is probably true of image files too....
It does not appear to be documented, and your crawler is wasting lots of bandwidth downloading lots of media files that do not contain any URL metadata.
Hello, i’m trying to subscribe to your patreon and everytime i do it the account gets disabled, i wanted to know if you have any contact method like telegram or...
This [person](https://github.com/ScottHelme) has been crawling Alexa's Top 1M domains for the past 7 years and has made the raw data public domain at the above website.