bug: number of domains found is not consistent

[kristovatlas]

Something I noticed when suffering from #9 is that the number of domain variants within a short period of time is inconsistent. This is probably some DNS quirk but might merit more research.

Here's output from a single invocation:

2017-01-16 19:21:16,065:INFO:Now checking [email protected] - example.com
2017-01-16 19:21:17,431:INFO:DNSTwist found 71 variant domains from example.com.
2017-01-16 19:21:17,433:INFO:Found 0 new domain variants from example.com
2017-01-16 19:21:17,433:INFO:Now checking [email protected] - example.com
2017-01-16 19:23:30,415:INFO:DNSTwist found 68 variant domains from example.com.
2017-01-16 19:23:30,416:INFO:Found 0 new domain variants from example.com
2017-01-16 19:23:30,416:INFO:Now checking [email protected] - example.com
2017-01-16 19:23:30,888:INFO:DNSTwist found 70 variant domains from example.com.
2017-01-16 19:23:30,890:INFO:Found 0 new domain variants from example.com
2017-01-16 19:23:30,890:INFO:Now checking [email protected] - example.com
2017-01-16 19:23:46,608:INFO:DNSTwist found 62 variant domains from example.com.
2017-01-16 19:23:46,610:INFO:Found 0 new domain variants from example.com
2017-01-16 19:23:46,610:INFO:Now checking [email protected] - example.com
2017-01-16 19:25:22,057:INFO:DNSTwist found 73 variant domains from example.com.
2017-01-16 19:25:22,059:INFO:Found 0 new domain variants from example.com
2017-01-16 19:25:22,059:INFO:Now checking [email protected] - example.com
2017-01-16 19:25:22,570:INFO:DNSTwist found 73 variant domains from example.com.
2017-01-16 19:25:22,571:INFO:Found 0 new domain variants from example.com
2017-01-16 19:25:22,572:INFO:Now checking [email protected] - example.com
2017-01-16 19:25:32,913:INFO:DNSTwist found 72 variant domains from example.com.
2017-01-16 19:25:32,915:INFO:Found 0 new domain variants from example.com
2017-01-16 19:25:32,915:INFO:Now checking [email protected] - example.com

[kristovatlas]

Note that for anyone who can tolerate waiting a few extra cron invocations, this doesn't matter much since it's stable enough over time to catch domains that get "missed" in a single invocation. Time-until-seen can be decreased by running cron job more often.

[0xd34db33f]

I've seen this before as well and just chalked it up to DNS rate limits (we're pretty much loading up a bunch of domains and firing it off which I've actually saturated my test DSL line before). If we introduce any internal rate limits I'm fine with that, we'll have to start modifying the dnstwist code though to handle it.