deFUDing “DNSSEC has failed”

It’s easy to forward or like an article of which the title says something you agree with, without actually reading the content. It seems “DNSSEC has failed” is one of those that showed up in my twitter feed. It’s so defunct of facts, it’s pretty clear most people have just read the title and retweeted it. Hopefully, they will at least read this post before not retweeting it.

It starts comparing the DNSSEC protocol with the SSH protocol. It claims ssh just works. It fails to take into consideration that the two are completely different trust models. With ssh, you tend to own both end points. So there is no issue of PKI or key distribution or trust. And even so, I am willing to bet the author never actually checks his SSH fingerprints when connecting to new servers, and just depends on the ssh leap of faith. Well, security is easy when you can assume you are not under attack. Most ironically, the easiest and only cross-trust-domain SSH public key distribution system out there is the DNSSEC protected SSHFP record. The only other one I know of is GSSAPI and other hooks into centrally-managed systems – eg all systems under one administrative control. Trust is pretty easy when you centrally distribute trust anchors.
Let’s also not talk about the debian ssh disaster, which was a pretty spectacular failure compromising thousands of ssh servers. Let me conclude the ssh comparison with inviting the author to login to bofh.nohats.ca using ssh in a fully trusted matter. He will end up 1) calling me to verify the ssh host key, 2) trusting the SSHFP key in DNSSEC or 3) leap-of-faithing it and hoping he was not under attack.

Next, the author brings up the tremendously easy to use and success of TLS. Never mind that we still need to run SSL 2.x for WinXP compatibility, most servers run TLS 1.0 which is severely broken, we just had the TLS 1.2 resumption/renegotiation disaster, about 4 CA compromises in 2013, and browsers running to patchwork solutions like hardcoded certificate pinning and Certificate Transparency registries that uses “n of m” style “it is probably the right certificate” solutions. There are about 600 (sub)CA’s that can sign any TLS certificate for any domain on the planet, ranging from Russia, to Iran to the U.S – all parties that are known to coerce parties and with a heavy industrial espionage component.

If TLS was successful, port 80 would be dead. Port 80 is the proof that the TLS trust model has failed. It involves money and so these certifications are only available to corporations. While DV certificates are now basically free, they are also considered worthless (protection against passive attack only) EV certificates are now dropping in price – which will make them finally available for more people but unfortunately because of that also lose their value. And all of this security hinges on the (mostly hidden) CABforum and browser vendor decisions on which CAs to include, and which CAs not to include.

While on most proprietary OSes, these preloaded CAs are at least managed by the vendor, the situation in Linux is a disaster. OpenSSL doesn’t support blacklists, NSS does. GnuTLS I don’t even know. Where are these CAs and blacklists? Do you know applications using openssl for their TLS are using the same CA bundle as applications using NSS or GnuTLS? Only very recently has Fedora and RHEL moved to a system where this is guaranteed. Even worse, most python wrappers using TLS don’t even check anything of your TLS connection. Presenting the wrong certificate will just cause that software to continue. And then look at the latest Apple TLS bug where anyone could bypass all TLS checks completely. How the author can even point to TLS as a security success is a complete mystery to me. It’s the worst computer security disaster we ever created! Look at the amount of complete failure TLS deployments at SSL Pulse – 30% allows weak ciphers, 6% has a broken certificate chain, less then 30% supports TLS 1.2, 6% vulnerable to the renegotiation/resumption attack, 70% vulnerable to the BEAST attack, 13% vulnerable to the compression attack, 56% allow RC4 and 50% does not support PFS. And that according to the author is the “success” that DNSSEC should take a lesson from?

Having listed the “success” stories of ssh and tls, he than looks at DNSSEC. His first factual statement regarding DNSSEC is “Most resolving software supports DNSSEC, but none have enabled it by default“. He is wrong. I have been the package (co)maintainer of bind, unbound and nsd for Fedora and EPEL for many years. On Fedora and EPEL/RHEL the default install enables DNSSEC for all three, and has done so for many years – even before the root was signed with DNSSEC!

Next he claims “To enable DNSSEC we have a ‘tutorial’ by Olaf Kolkman which spans a whopping sixty-nine pages“. As I said, the author is wrong. Simply “yum install bind nsd unbound” and the software comes with dnssec enabled, using the root key as trust anchor.

He then points to a DNSSEC training that takes “multiple days” and claiming that shows the complexity. The same vendor lists a two day DHCP course, a two and a five day “plain DNS & bind” course, and a two and five day IPv6 course. Does that mean those technologies are all failures too because it takes so many days to teach a course about it? According to the author, DHCP would be a failure and no one can deploy it because it “requires” a two day training? It’s just simplistic cherry-picked FUD.

His next argument is “Furthermore, there are almost no simple instructions for enabling DNSSEC, especially none that use KSKs and ZSKs, and include instructions on how to do key rollovers.“.

First, let me ask you how you do an ssh host key rollover? How do you make that seamless so it won’t trigger anyone who ever connected before from seeing a changed key? Where do they go to confirm the changed key is installed by the administrator and not the result of a hacked server or connection? Does this mean ssh has failed too? Again, amusingly enough the only way to rollover ssh keys is piggybacking on SSHFP in DNSSEC or GSSAPI trust models. But I don’t see the author claiming ssh has failed because it cannot do rollovers on its own. More cherry-picking.

The reality is, ssh and DNSSEC can do fine without rollovers. Only TLS requires rollovers because the trust model depends on the fact that you regularly pay the trustees managing that whole disaster. But if you do want or need to do a rollover, with ssh you’re fucked, with TLS you have to pay and with DNSSEC you have a seamless transition method. Let me repeat that – only DNSSEC actually implements a free and seamless key rollover mechanism! So of course, the author just states it is too hard to use. But again does not compare it against TLS or ssh at all. Cherry-picking all the way down. And by the way, there are also good reasons why you do not need to worry about DNSSEC rollovers at all.

As I said before, DNSSEC is already “enabled”, so let’s interpret the author’s quest for “simple instructions to enable DNSSEC” to mean “how do I sign my zone?”. Let us assume he is running bind right now with his zone in /var/named/1sand0s.nl. The author now has to issue the following complicated sequence of commands:


$ dnssec-keygen -f KSK -a RSASHA256 -b 2048 -n ZONE 1sand0s.nl
Generating key pair........+++ ..+++
K1sand0s.nl.+008+01699
$ dnssec-keygen -a RSASHA256 -b 1024 -n ZONE 1sand0s.nl
Generating key pair........................................++++++ ..................++++++
K1sand0s.nl.+008+24363
$ cat K1sand0s.nl.*.key >> 1sand0s.nl
$ dnssec-signzone -o 1sand0s.nl -k K1sand0s.nl.+008+01699 1sand0s.nl K1sand0s.nl.+008+24363
Verifying the zone using the following algorithms: RSASHA256.
Zone fully signed:
Algorithm: RSASHA256: KSKs: 1 active, 0 stand-by, 0 revoked
ZSKs: 1 active, 0 stand-by, 0 revoked
1sand0s.nl.signed

Edit named.conf and load the file 1sand0s.nl.signed instead of 1sand0s.nl and you’re done. And whenever you change your zone, re-run that dnssec-signer command.
(NOTE: and as pointed out to me, if you don’t have zone edits every week, do run that signer command in a cron job weekly to refresh the DNSSEC signature records)

You can put this into a git hook and maintain the whole thing without ever realising there is a signing step in between. Hardly rocket science. I think we can argue whether or not this is harder or easier than creating a CSR for Apache, get it signed, and than installed (and too many people will forget to actually get the intermediate CAs installed in apache, and their deployment is actually even broken!)

The author’s issue with opendnssec is that it is “yet another daemon” using a “database backend”. I guess he is also not running postfix because that thing has more daemons than I can remember the names of. Opendnssec’s default “database backend” is sqlite. If you hate those, there is a lot of software you’ll going to have to purge from your servers. Sqlite is a perfect backend for small databases that are just too complex for simple text files. For those of us who don’t mind two extra daemons and some sqlite files, to sign your domain from scratch with opendnssec:

# yum install opendnssec
# systemctl enable opendnssec.service
# systemctl start opendnssec.service
# ods-ksmutil zone add –zone 1sand0s.nl –input /var/named/1sand0s.nl –output /var/named/1sand0s.nl.signed
# ods-signer sign 1sand0s.nl

Change named to load the .signed zone instead of the unsigned zone and done. Shockingly complicated! Unsuitable for “common engineers” according to the author.

To make these instructions complete, the last step is to send your DS record to your Registrar which involves using their custom webgui to send them the following data:

yum install ldns
dig dnskey 1sand0s.nl > 1sand0s.nl.dnskey
ldns-key2ds -n 1sand0s.nl.dnskey

And if you want to do all of this within the same bind daemon and without sqlite, you can use bind’s builtin inline-signing management features instead.

Conclusion

Saying the tools are there for ssh (which doesn’t have ANY rollover support) and tls (which over half the world has misconfigured regardless of rollovers) but not for dnssec, is really short-sighted and wrong. What the author has done is taken his prejudice, and written some cherry-picked text justifying it. The only credit I can give him, is that he did not have to rely on the FUD statements by djb.

DNSSEC software bug causing nohats.ca down time – possible catch22

I sign my own domain nohats.ca using OpenDNSSEC. Since .ca is not yet signed, I added my key to the ISC DLV Registry. It is enabled by default by Fedora and RHEL if you install the bind/unbound name server for resolving.

Today, I removed and added a few zones to my opendnssec (an alpha version, 1.4.0a) based signer. These domains were unrelated to nohats.ca. But somehow I ended up with a “signed” nohats.ca zone that contained NSEC3 records but no RRSIG records, and one other domain that only contained 1 RRSIG record. As a result, anyone who was using DNSSEC could no longer resolve my domain nohats.ca. That included me. The nohats.ca domain is still in this weird state inside opendnssec, so I decided to remove the DLV record for that domain. Turns out I forgot the password to my login at dlv.isc.org. With my mouse hovering over the “request password reset” option, I realised that my MX records point to “nohats.ca”. If the dlv.isc.org site uses DNSSEC, they will fail to resolve the MX record to send me my password reset information to fix my DNSSEC setting. Catch 22!

Whether by design or by sheer luck, this did not seem to be the case and I received my password reset, and have removed the DLV record for nohats.ca.

I upgraded opendnssec to 1.4.0a2, but this has not resolved the issue of ods-signerd giving me a zone without RRSIGs. I’ll have to investigate this more tomorrow. Our tools really still have lots of room for improvement.

But it does bring up a delicate point. Registrars should ensure that there is a way you can remove a DS record using an account that somehow can do a password reset even if your only email goes into a failing DNSSEC domain. Perhaps a two factor using a text message to a phone? Or perhaps allow more then one email address for a reset, so that people could include use two different email addresses in two different domains. Though one should not point password resets to domains that are not secured by DNSSEC, because these are precisely the kind of messages people could abuse to hack access to your domain account. Another security catch 22.

So if you are a registrar, please think about this issue. Sooner or later one of your customers will be the position I found myself in…..