namecoin.org icon indicating copy to clipboard operation
namecoin.org copied to clipboard

3-millisecond and 100-millisecond benchmarks aren't reproducible

Open JeremyRand opened this issue 7 years ago • 1 comments

https://bit.namecoin.org/ states:

Standard DNS servers also take time to look up a website's information, which can take 100 milliseconds or more if you're unlucky. Since Dot-Bit keeps the phonebook on your own computer, looking up a website usually takes under 3 milliseconds.

The methodology isn't documented here, which makes it difficult to reproduce the results.

As I recall, the 3ms benchmark was obtained with NMControl in pre-caching mode. ncdns is very different than NMControl in many ways that impact performance (e.g. it's authoritative instead of recursive, which probably makes it slower; it's Go instead of Python, which probably makes it faster; and it doesn't have a pre-caching mode like NMControl does, which probably makes it slower). It would probably be productive to re-run these benchmarks, with high-quality documentation of how the benchmarks were achieved.

I also do not actually recall where the 100ms benchmark came from. It sounds entirely plausible, but we might want to actually find a citable source for this.

(Filing the bug in this repo since we're planning to move https://bit.namecoin.org to Jekyll in the future.)

JeremyRand avatar Jul 09 '18 14:07 JeremyRand

Also note that it's probably best to run these benchmarks without a hypervisor, since things like Xen are known for interfering with latency benchmarks. (I'm still a bit scarred from the issues Xen caused me in debugging my Firefox TLS patch....)

JeremyRand avatar Jul 09 '18 14:07 JeremyRand