DNSSEC software bug causing nohats.ca down time – possible catch22

I sign my own domain nohats.ca using OpenDNSSEC. Since .ca is not yet signed, I added my key to the ISC DLV Registry. It is enabled by default by Fedora and RHEL if you install the bind/unbound name server for resolving.

Today, I removed and added a few zones to my opendnssec (an alpha version, 1.4.0a) based signer. These domains were unrelated to nohats.ca. But somehow I ended up with a “signed” nohats.ca zone that contained NSEC3 records but no RRSIG records, and one other domain that only contained 1 RRSIG record. As a result, anyone who was using DNSSEC could no longer resolve my domain nohats.ca. That included me. The nohats.ca domain is still in this weird state inside opendnssec, so I decided to remove the DLV record for that domain. Turns out I forgot the password to my login at dlv.isc.org. With my mouse hovering over the “request password reset” option, I realised that my MX records point to “nohats.ca”. If the dlv.isc.org site uses DNSSEC, they will fail to resolve the MX record to send me my password reset information to fix my DNSSEC setting. Catch 22!

Whether by design or by sheer luck, this did not seem to be the case and I received my password reset, and have removed the DLV record for nohats.ca.

I upgraded opendnssec to 1.4.0a2, but this has not resolved the issue of ods-signerd giving me a zone without RRSIGs. I’ll have to investigate this more tomorrow. Our tools really still have lots of room for improvement.

But it does bring up a delicate point. Registrars should ensure that there is a way you can remove a DS record using an account that somehow can do a password reset even if your only email goes into a failing DNSSEC domain. Perhaps a two factor using a text message to a phone? Or perhaps allow more then one email address for a reset, so that people could include use two different email addresses in two different domains. Though one should not point password resets to domains that are not secured by DNSSEC, because these are precisely the kind of messages people could abuse to hack access to your domain account. Another security catch 22.

So if you are a registrar, please think about this issue. Sooner or later one of your customers will be the position I found myself in…..

You can’t P2P the DNS and have it too

As DNSSEC evangelist, I am often told that DNSSEC cannot solve the fundamental problem of censorship and domain take down. And I agree. However, these people then follow up with mumblings about “P2P DNS” and how a decentralized DNS would solve this censorship and central control problem.

Unfortunately, per definition, there cannot be a decentralized name lookup service. Let’s take a step back and think again what the DNS is.

An address is only an address if we all agree

I live in Toronto. You might know where that city is located. If not, you can ask someone else you trust. Or you can ask a trusted company, like Google maps, about this “Toronto” thing, and it will tell you where this city is. Or you open 3 different atlas books by different vendors and you look up “Toronto” and compare the results.

The underlying assumption here is that there is only one LOCATION that is known by the NAME Toronto. We all know where “Toronto, Canada” is because we all agreed to name this particular location on the Earth with that globally unique identifier (mnemonic) that is easier to remember then “Let’s have coffee at Latitude 43.6444208 and Longitude -79.4025781”

And we see where this system can break down. Depending on who you ask, “China” refers to something slightly different. According to some, “Taiwan” does not exist. Do you remember a few years back when you suddenly realized that the city of Mumbai was actually what you knew until then as the city of Bombay? In a relatively brief period (a few years) everyone around you started using Mumbai instead of Bombay. This name change came in through some centralized authority and spread down the hierarchy all over the world. The name change within Mumbai undoubtedly went much faster then the update of the school atlas in Wiggins, Mississippi, USA. And just like DNS, this update of a city took some time, because people “knew”¬† (cached) that it was called Bombay without looking at the new atlas version. Every piece of information update has a “time to live” where you just cannot assume people are aware of the new name.

People have to agree to what an address location is or it won’t work. If I decide
to rename “100 Queen Street West, Toronto” into “Fnord Hall” and no one else knows
about the name change, we won’t be able to meet up for coffee at Fnord Hall because
you have no idea where it is or even how to look up what addressing scheme I used.
Similarly, if the people of Queen Street replace all the signs and decide to call
it King Street, there will be some confusion as to which of the two King Street’s
you might be referring to.

In other words, address agreements are not “peer to peer”. There is a centralized, or rather a hierarchical method for assigning, reassigning and distributing location name updates. And the only reason it works, is because there is censorship and central control.

The DNS is an addressing system. Any P2P DNS alternative still needs to have this property of everyone agreeing to give a certain name to a certain location. And with that comes authority to make changes and disagreements about certain changes (censorship)

The Name mapping problem

The Domain Name System is a lookup mechanism similar to the Geo Location Name System (AKA “atlas”) The DNS serves to match an easy to remember mnemonic (“nohats.ca”) to an impossible to remember IPV4 or IPv6 address. It ensures that no two entities can claim the same mnemonic. And it is also really really big. If we look just at the COM, NET and ORG we’re talking in the order of magnitude of 100 million entries. If you do not live in Canada, your atlas likely does not contain the exact location of 100 Queen Street West, Toronto, Canada. If it did, your house would be filled with just your atlas.

The DNS is hierarchical,and its trust model is hierarchical, assuming you deploy it with DNSSEC. It has censorship and central control. And that’s why it works.

So let’s say you have the perfect decentralized P2P DNS table. It can uniquely represent every “domain name”, using some bitcoin like system of random numbers and transaction logs copied all over the world in shorter times then current DNS TTL levels, supporting over 100M ever changing entries. This in itself would be an incredible, though unlikely, engineering achievement, but let’s assume you people are super smart, outpacing the collective 25 years of IETF DNS experiences, and you managed to built, distribute and continuously update a secure P2P DNS table.

So let’s say my web server is at 438946936129363483649513524349734902630640.namecoin.

Well that’s great. This is about as useful as saying that my web server is at 2001:888:2003:1004:c2ff:eeff:fe00:102. The only difference is that the first would be (assumed) cryptographically secure somehow. But no one is going to advertise their site as 438946936129363483649513524349734902630640.namecoin.

So now you have to design some mapping system, where this name maps to something easy like “nohats.namecoin”. Great, so now who runs this mapping service? Who is preventing someone else from taking” nohats.namecoin” and pointing it to their server? Who is going to release this name in 100 years when I’m long gone? When you start answering these questions, you quickly develop something of a Registry-Registrar-Registrant system, and you just re-implemented DNS except in a new TLD called “.namecoin” (or .onion, or whatever your name scheme will be)

To use this mapping system, you either need to have clients to do the lookups all by themselves (sort of like running a DNS resolver on your laptop) or you are going to chain proxies (sort of like recursive nameservers of your ISP). And guess what, governments will filter it forcing you to use their proxies, which will conveniently filter out some entries. At most, you can reach the level of DNSSEC where this tampering can be detected. But so far, you haven’t actually improved on the current DNS/DNSSEC system.

How can we ensure trust in the mapping service?How can you detect tampering? In the “trust no one” model, this needs to become an “n out of m” trust system. Not even Chuck Norris or Moxie Marlinspike¬† can vouch for 100M name mappings. Can you trust 100 Ambassadors? Or 90 out of 100? And what do you do when you DO find a lack of trust level in a name mapping? Is your browser just going to fallback to DNS or prompt you with “Add security exception”?

Systems like this have been devised for SSL certificates. CertPatrol, EFF’s trusted certificates, and Moxie’s Convergence all have this problem, and so will any P2P DNS approach. These systems fail because they don’t scale. If an update happens in these systems, these “trusted ambassadors” have to play catch up with reality. So every time your P2P DNS would change its mapping, it will take some time and false positives before things work again. What is really needed is that these updates are authenticated by the publisher, who is the only entity that really knows what is real and what is forged. And this is exactly what DNSSEC provides.

To summarize this part, any cryptographically secure unique identifier that is a replacement for DNS will introduce the exact same problem of needing a mnemonic mapping service that such a new solution is trying to obsolete.

Domain confiscations
Another motivation for P2P DNS is that no central authority can take a name identity away from the “real” owner. ICANN or Verisign can take away any com/net/org domain, and we have seen this happen. The solution is not to replace DNS with some decentralized system, because again that would be moving the problem around.

Who runs the new mapping service? Under what legal system will these mapping servers be running? Or alternatively, which guns are going to enforce that these mapping servers will be hosting, or not hosting, certain content? Or which banks will block payments to keep this alternative name mapping system up and running? Look at how torrenting distributed the crime of downloading movies illegally away from centralized websites to individual participants. People are still sued under the legal system of the physical location they reside in.

Unfortunately, all these new TLDs by ICANN will be under the same controls as com/net/org, so don’t get your hopes up there. The only known solution to defend against this, is to use the TLDs of various different countries at the same time. Remember when wikileaks.org was down and you could still access them via wikileaks.ch, wikileaks.de and wikileaks.is. That’s the only way I know of ensuring your “name” will be up in one of the various legal systems in the world.
Your chances of name survival increases if your do not use US based Registrars and DNS Operators for your non-US based domain registrations, or else you are still putting all your eggs in one legal system.

DNSSEC cannot be trusted because the USG holds the root key.

Finally, people tend to claim that the root key is Sauron’s ring, and whoever has it will get tainted by political or military requests to forge or remove data the DNS hierarchy. And they have the cryptographic key to do so because everyone is forced to trust this key as entry point into the DNS.

First of all, there is an Enigma Problem with that. As soon as someone posts a forged DNS record signed by the root key that is obviously not legitimate, people will quickly move away from such a root key. Very likely, the ccTLDs would get together and create an alternative root zone, split off their hundreds of root nameservers from the Verisign a.root-servers.net server, and then the US would never have this power again in the future. Such an event will happen exactly once.  Whoever would be the target of such forging would have to be a very valuable target indeed. If that is you, you have better things to do then reading this posting.

Second, DNSSEC did not change any of this. The legal system in place already allows the US government to take a domain away. And all the individual countries have similar provisions, or can enact such laws at any time. It is the legal system that will force the hand of the TLD operator. This situation is not different with or without DNSSEC. The
only difference is that with DNSSEC, there is a trail of cryptographic proof, so outsiders
actually can tell much better what happened and who was responsible for it.

More importantly, if you really don’t want to trust Verisign, or PIR, or IANA, you can sit
down with the people you DO trust and start one of these ambassador/convergence schemes for DNSSEC keys just like these systems do for SSL certificates. Collect and distribute all the ccTLD DNSSEC keys and publish it somewhere with appropriate cryptographic security and authenticity. Anyone can then regularly grab that information and prime their DNS servers with it, and all your ccTLDs will be safe from tampering by the US Government, ICANN, Versign, IANA or the root zone operators.

Back to reality: Registrar compromise does affect DNSSEC

A much more likely scenario is that of a compromised registrar, where an attacker gains access to your domain information including NS servers and DNSSEC public key material (DS records) and is able to change the entire domain to point to their own servers, update your DNSSEC trust anchor for theirs, and so validate their hostile take-over as being the cryptographically proven site you are running.

For your own domains, you really want your resolver to hard code that key, overriding any parental DNSSEC information. A registrar compromise will at least not compromise you further as you use passwords or other credentials against what you (cryptographically) think are your own proven servers, but which are really malicious. It might not help the world at large, but if you are guaranteed to find out, at least you can start the repair process. And you already have some kind of monitoring for this in place called “ssh known hosts”.

Image one day you’re having coffee at your local Star Bucks, and you cnnect to your
server and see this:

[paul@thinkpad ~]$ ssh nohats.ca
[paul@thinkpad ~]$ ssh root@nohats.ca
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Update the SSHFP RR in DNS with the new host key to get rid of this message.
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in /home/paul/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/paul/.ssh/known_hosts:560
RSA host key for nohats.ca has changed and you have requested strict checking.
Host key verification failed.

This particular errors shows up when your server was compromised or your routing was changed, while your DNSSEC was intact. If they had also changed your DNSSEC, you could not see the message about the SSHFP record being incorrect, but you would still see the RSA changed key warning, pointing out a strange new discrepancy between ssh reality and DNSSEC reality

You can decide to monitor a few domains you do not control but that you find important, like twitter or Facebook or Gmail. But again, this runs into the problem described above where you will suffer false positives when they legitimately change something.

DNSSEC actually helps you

With more and more devices running DNSSEC locally, it is actually becoming much harder to spoof or forge DNS. When governments dictate that ISPs need to filter or rewrite DNS requests, DNSSEC will detect this as malicious. This is why SOPA really failed. Not because politicians changed their mind about the legality or morality of hostile domain take over. SOPA failed because the DNS experts convinced the politicians it was totally useless because of DNSSEC.

DNSSEC even helps you against the NSA

Let’s assume Verisign had to give the NSA copies of the private key of the root zone, and the NSA can sign anything from the root down. They don’t have a copy of the .ca key, but since they have the root key, they can replace the public key for .ca (DS record) in the root zone with a new RSA key they just generated for this purpose. And then they repeat this process for “nohats.ca”.

To mislead my readers who think they are going to “nohats.ca”, the NSA intercepts your laptop’s DNS, and uses DNSSEC signed data to convince you my server is in Virginia. Doesn’t this mean DNSSEC would be useless?

Not really. Because this intercepting of your DNS is actually not an easy problem. First of all, they need to hide their tracks. If they change my domains in the “global DNS view”, then this is bound to be detected by me. So they have to do this in a very targeted matter that would only affect you, but not the masses. But with DNSSEC, you can ask any server in the world via any VPN tunnel or TLS connection for DNS data. In fact, currently my laptop will attempt to talk DNS over TLS to Fedora Project DNS servers when it detects that regular port 53 DNS is getting mangled by a hotspot. Or I could be using TCP connections via the TOR network to get my DNS lookups from a server in The Netherlands. Or I would be connected to the Red Hat VPN, and my DNS lookups originate from Rayleigh, North Carolina. So while my laptop is at a certain Star Bucks wifi location in Toronto, the NSA cannot just forge that local hotspot DNS. They will basically have to contaminate the global DNS view to ensure I am being properly misled. So anyone who runs a simple monitor script to check their DNS contents would immediately know their domain’s DNSSEC key was compromised. Or again, you could run some ambassador or n out of m system for this monitoring to detect such DNSSEC key compromise, though this system would also have to deal with false positives when real DNS updates happen.

Note that if the NSA would be that interested in your laptop traffic, they would be better of just redirecting your IP stream and leave the DNS alone. Though that method will finally come to an end soon with the new TLSA DNS record type, which I will explain in another post.


DNSSEC provides valuable additional security to the Domain Name System, even if you believe the root components such as Verisign cannot be trusted. DNSSEC does not protect you against the laws of nation states you disagree with, but then again, no technology does that with perhaps the exception of the cloaking device.

Any P2P DNS system that claims to better then DNSSEC, has not understood the actual problems. You just need to find and point out where in their system they are outsourcing their trust. Just like the DNS system.

ENS Update

For those telling me it can be done and that is what Ethereum Name Service (“ENS”) is, please see this Q&A of me at ICANN60:

Paul Wouters: Let’s say IETF gets the domains IETF in this naming system and we pay our fees for a couple of years. Everybody uses the site. And then at some point we forget to pay and the domain falls back into the pool and somebody else registers it and we don’t know where they are or who they are. Now i go to a court system. I get some legal opinion saying i own this trademark and now i want to get this domain back. Is there any way for me to get this domain back?

Leonard Tan: So right now, the ENS industry, you can change it because it requires four out of seven people. Most of them are ethereum developers. And it is a consensus of them to make any changes. So it is possible, but it is going to be a very difficult thing to do.

So, why have a distributed blockchain based system, when a court system (or Mafia, or a large wad of money( can convice or foce 4 out of 7 people to make a change? You might as well use non-blockchain security, maybe use Shamir’s Secret Sharing instead.

Of course, the question was a double edged sword. If he said it was not possible, he would also show a fundamental problem with ENS – you would then be able to lose your name for a number of reasons and never recover from it. Imagine Coca Cola or Nike losing their brand.

geome – a small utility to get your geo-location (to be used by OTR)

Lately, I’ve been doing FIPS related work for Red Hat, and I’ve spend way too much time rebooting kernels and debugging dracut. So when I recently saw fellow cypherpunks Ian Goldberg’s work on SkypeMorph, I was reminded of another project at the University of Waterloo, The Nearby Friend plugin for Off The Record.

For those who don’t know, Off-The-Record is an encryption protocol for Instant Messaging. It has some properties that are unlike regular encrypted chats. It offers repudiation, that is you can deny you said something, as no digital signatures are used to sign your messages with a private key only you would possess. A more recent addition to that protocol is an implementation of a PET paper Louis, Lester and Pierre: Three Protocols for Location Privacy. Using an OTR secured chat, you can exchange your location information in a special mathematical way so that you only reveal your location if the other person is within a certain range of you (which both parties set individually). Say you set your location to 10km. You can run this protocol with your 500 friends, but you are not sharing your actual location with them. But if any of your friends come within a 10km range of you, and they also have their setting to 10km or less, then the two of you will actually exchange your physical location. This gives you much more privacy then telling some central authority your location and then trusting that entity to only share it with people you approve of.

Of course, to do this, we need to actually know our location. These days, the most common method for that is to use a combinatin of GPS and wifi signals. And in fact, just the wifi signals are usually enough. A disadvantage of this method is that you are sending your wifi signal information to a central party, and that third party then tells (and knows!) your location, though they only know who you are based on a hopefully ever-changing IP address.

I had looked at the NearbyFriend pidgin plugin years ago, but it used a proprietary interface and database by SkyHook, so I stopped looking at it. I checked it again a few days ago, and the code has been sitting there unchanged for years. So I googled around for some alternatives, and found some code by Francis Markham at http://code.google.com/p/geolocate-cli/. It uses the Google Location API. Although Google does some evil these days, I do still believe they are still mostly good. And when questioning them for this information, you do not need to tell them who you are, so it is still somewhat anonymous. The nice thing about the Google API is that even if you have no wifi signals, it will tell you your location based on the IP address you queried them from.

It was written in python and setup to be quite modular, but with many of the modules not working and strange module names that are annoying to install on full systems (like misc.py and plugins.py). So instead of trying to package it up, I stripped it down to a minimum and put it all in a single python script.

The result is geome

This is geome running on my desktop (with no wifi)

[paul@bofh ~]$ geome
{"city": "Toronto", "street_number": "60", "country": "Canada", "region": "Ontario", "long": -79.381667, "street": "Queen St W", "postal_code": "M5H 2M3", "country_code": "CA", "lat": 43.6525, "accuracy": 166000.0}

For Nearby Friend, I just need the latitude and longitude, so I added a “-s” option to make that much easier

[paul@bofh ~]$ geome -s

(I’ll update the example with a wifi one, once I’m not actually at home :)

I already ripped out the Skyhook code from NearbyFriend, needed because I don’t even want the header files that contain the proprietary API inside any opensource software package. Over the next few days I hope to finish up the integration of geome with NearbyFriends and push a package into Fedora.