(See bottom of post for an update running similar tests on OpenDNS.)

Methods: I searched Google for keywords that I believed fell somewhere between obscure and common and collected the first ten hostnames printed on the screen. I then used local installations of dig to query a collection of DNS servers for the hostnames’ A records and collected the response times. The different resolvers used were:

• A local BIND installation (127.0.0.1, cache empty) with Comcast Internet connectivity;
• A Comcast DNS server (68.87.69.150) via Comcast Internet connectivity;
• My employer’s internal caching DNS;
• Google (8.8.8.8) via my employer’s Internet connectivity (mostly Level 3);
• Google (8.8.8.8) via Comcast; and
• Google (8.8.8.8) via an Amazon EC2 instance in us-east-1a.

Anticipating a bimodal distribution of results, I assumed high latency responses were cache misses, while low latency responses were cache hits, and categorized results correspondingly.

Limitations: Chiefly, the small number of hostnames queried. Results from a larger group of domains would be more conclusive.

Results: Given in the format of Server/Connectivity: Cache Miss/Cache Hit

• Local BIND server/Comcast: 319ms/0ms
• Comcast/Comcast: 166ms/14ms
• Google/Comcast: no misses/73ms
• Employer/Level 3: 235ms/30ms
• Google/Level 3: 204ms/44ms
• Google/EC2: 190ms/4ms

I concluded that Google/Comcast had no misses by testing another set of obscure hostnames twice each, noting that the first query was slower (~120ms) and the second query was similar in latency to the results above (70ms). (My belief is that I inadvertently pre-populated the cache for Google/Comcast by my tests elsewhere.)

Discussion:

• It’s all about cache hits. Whichever resolver gives you the most cache hits will give you the best performance; cache misses are at least an order of magnitude slower than cache hits. In this extremely limited test, the cache-hits-champion appears to be Google. Excluding Google/Comcast, where I believe I pre-populated Google’s cache, Google had a 50% cache hit rate, while Comcast and my Employer only hit 20%.
• Location, location, location. Secondary to cache hits, the closer the resolver is to you, the better. Looking at the Comcast results, it’s hard to get closer than localhost, and, as seems logical, Comcast’s resolvers have lower cached latency than Google’s. Running a local caching resolver forwarding to Google may be a desirable configuration.
• Resolver behavior matters. Comcast is notorious for poor behavior. It’s reasonable to expect that Google will be mining your DNS query data. Running a slower but directly-controlled local, non-forwarding server may be preferable for privacy and security reasons.

Update: At the suggestion of @tscalzott, I researched OpenDNS’s performance with the same set of hostnames via the same connectivity on their DNS resolvers at 208.67.222.222. This time, however, I queried the A record for each hostname twice in rapid succession to ascertain how many of my queries were served from OpenDNS’s cache. Results are in the format:

DNS Server/Connectivity: Cache miss/Cache hit – cache hit rate

• OpenDNS/Comcast: 218ms/30ms – 20%
• OpenDNS/EC2: 144ms/2ms – 10%
• OpenDNS/Level 3: 230ms/4ms – 40%

Compared to Google, OpenDNS had similar latency for cache misses and lower latency for cache hits, but appears to possibly have a lower cache hit rate. It seems likely that the latency “winner” for each user’s individual situation will depend on where they are on the Internet relative to the nearest Google and OpenDNS installations. Google’s greater cache hit rate suggests it may offer better service, but testing a larger number of hostnames would be necessary before being able to state that with any certainty.

Disclosure: I use Google’s free Apps services to host personal email, and I use their public sites (Search, Reader, News, Analytics, etc.) extensively. I recently attended a Google Apps for the Enterprise dog-and-pony show where I received a number of small tchotchkes; my wife took the notebook, I kept the pen and binder and threw the rest away. My employer uses Postini. I tried OpenDNS briefly several months back, but did not use them long-term because of limitations in my own configuration.

I see echoes of Toyota teaching its Toyota Production System in Google’s recent release of information about their data centers. Relatively straightforward concepts – the challenge is in adapting your existing systems to them.