Why the Sun 2 has the message “Love your country, but never trust its government”

(UPDATE: Sevan Janiyan pointed out the Sun 2 was not a SPARC system)

Yesterday, Alec Muffet posted a few tweets regarding the Sun SPARC 2 bootloader’s DES routines.


Alec figured that message was never supposed to be seen and suggested it was a kind of silent protest of someone in Sun against the US Government. I replied, saying I was pretty sure such a message anywhere in the Sun bootprom code must have originated by John Gilmore. So I asked John, and he did not disappoint. This is what I wrote me back:

> That must have been you? :)

Yes. Vinod Khosla, first President of Sun, came to me at one point
and said to put something hidden, triggered in an unexpected way, into
the ROM Monitor, so that if somebody cloned the Sun Workstation
(violating our software’s copyright), we could do that unexpected thing
to the competitor’s demo workstation at a trade show and thereby prove
that they had cloned it.

The ROM Monitor command that printed “Love your country, but never
trust its government.” was “k2” followed by control-B and RETURN. (To
get to the ROM Monitor from the running Unix display, you had to hold
down the L1 key and hit “A” then release both; the monitor had special
code looking for that key sequence.) I had found that saying years
before on a hand-painted sign tacked up on a pole or tree in central
Pennsylvania, wrote it into one of my notebooks at the time, and
plucked it out as the hidden thing after Vinod asked. In the source
code it was obscured as a set of hex numbers, but in the binary it was
visible. (I didn’t bother to XOR it with something to make it more

The circumstance Vinod was concerned about never did come about;
nobody stole the Sun boot roms. Early 680x0s didn’t come with a
standard MMU, so everybody who wanted virtual memory had to invent
their own, and our roms were very tied to our custom static-ram based
zero-wait-state MMU, which was also patented. A few companies
licensed the board design from us, like Imagen for a laser printer
controller, but they had a license to use the ROMs.

The DES chip slot was intended to speed up DES for high security
networking applications, and we did get it working. I think Bill Joy
was the one who made it a part of the architecture, all the way
through the Sun-3, thinking that we’d use it as part of securing the
network file system. But the chip was expensive and
export-controlled, and our software never really used it (NFS ran in
plaintext and used the sender’s IP address for authentication!). So
it didn’t get stuffed on the production boards, and eventually they
stopped stuffing the support chips too. Tom Lyon wrote a nice device
driver for it, and a “des” user command that would use the chip for
file encryption if it existed (or fall back to software).
In one 68010 based model we also put in a slot for an Intel floating
point chip — I think the one that came with the 80286. We got it
working (I did the initial debugging) and fed it commands and
arguments via manual peek/poke in software. Doing that was somewhat
faster than the software floating point that we were otherwise using,
despite the overhead. But as I recall, Intel wouldn’t sell us the
chips in volume, because they didn’t want to make the 68000 more
competitive against their own x86 chips. So we had to wait til the
68020/68881 came out (Sun-3 era) before we had fast floating point.

It’s nice that Matt Fredette made a Sun-2 emulator. I may have
images of old SunOS release tapes that might work in it. And I have a
Sun “FE Handbook” for the field engineers, that has a lot of the
details about what chips go where, what the jumpers on each board
mean, what the memory map of each board looks like, and etc.

You can forward this info on to whoever…


Don’t stop using IPsec just yet

Today, Laura Poitras and Jake Appelbaum spoke at the 31C3 conference and in collaboration with Der Spiegel published an interesting article on VPN exploitation by the NSA.

The “TL;DR” summary of what follows below is: If you configure your IPsec based VPN properly, you are not affected. Always use Perfect Forward Secrecy (“pfs=yes” wich is the default in libreswan IPsec) and avoid PreSharedKeys (authby=secret which is not the default in libreswan IPsec). If you really need to use PSK, use a strong shared secret that cannot be brute forced. The NSA has their own version of IKEcrack running on millions of dollars worth of CPU’s. Also, the NSA sneaks into your router to steal your PSK’s so they can decrypt all your traffic.

Update 1: from media-35513.pdf (“TURMOIL/APEX/APEX High Level Description Document”):

CES generally requires the packets from both sides of an IKE exchange and knowledge of the associated pre-shared key (PSK) in order to have a chance of recovering a key for the corresponding cipher (ESP). A major goal of APEX is to access two sides of key exchanges of traffic of interest”

Update 2: To stop the NSA HappyDance, I just commited code to libreswan IPsec to warn about weak PSKs with a message notifying the sysadmin that as of July 1st, 2015, libreswan will refuse those weak PSKs. To determine the strength of the PSK, I used the Shannon Entropy value. If the PSK has a Shannon Entropy of less than 3.5, a warning will sound and in 6 months it will refuse to use that PSK.

Update 3: Just to make it clear – To break IKE PSKs, you first need to break the initial DiffieHellman exchange, which is usually MODP1024 or MODP1536 in the bad cases (and MODP2048+ in the good cases). The exception to this is IKEv1 Aggressive Mode, where the MAC computed with the PSK as secret key is sent
in the clear. As a result one can mount an offline dictionary attack.

IPsec is not IKE

First, let’s clarify things a little bit. Although everyone is talking about breaking IPsec, what they really mean is breaking the Internet Key Exchange (IKE) that is used to negotiate and create symmetric keys that are used with IPsec for encryption and decryption. IPsec itself can use various ciphers and algorithms. It can use a separate encryption (AES, CAMELLIA, SERPENT, TWOFISH, 3DES, etc) + integrity (SHA2, SHA1, AES_XCBC, MD5, etc) mode or it can use an AEAD cipher that combines these two into one (AES GCM, AES CCM, CAMELLIA GCM, etc)

But Jake said to stop using IPsec

None of what was said by Jake or published by Der Spiegel shows any attack on these ciphers or algorithms. Even the MD5 use within IPsec (HMAC-MD5) is not as weak as people usually believe it to be by confusing it with non-hmac use of MD5. The comments Jake made and the material in the published slides show attacks against weak IKE configurations, router compromises, and possibly against some IPsec hardware accelerators. Below are my comments on those slides that are interesting in the context of IKE and IPsec.


The focus of the VPN decryption team lies with IPsec and SSH. I think that fits the deployment. These two protocols are the most widely used encryption protocols to transfer bulk data. While the hacker community might focus on OpenVPN, that’s not where the real deployment is. The mention of PPTP is curious. From what I see, PPTP is basically dead as a VPN technology, although it is still widely used in Russia as an encapsulation technique for ISPs to connect endusers. It’s probably just listed there for intercept because it is so trivial to do, you might as well gather it.


This slide is harder to make sense of. I think that here the NSA is talking about IKE and IPsec combined, although they call it IPsec. Perhaps once they decrypt traffic, it goes into XKEYSCORE? They don’t seem to use XKEYSCORE as an input to crack the IKE PSKs? I’m amused by the name VULCAN-DEATHGRIP which might indicate that the HiFN (now Exar) Vulcan IPsec accelerator card might have a flaw in it that allows them to decrypt those IPsec VPNs. Also, I hope they properly mask their perl scripts for malicious decrypted traffic :P


This is probably the most interesting slide. It basically states that IPsec VPNs are compromised by router compromises that steal the IKE PSKs. This might be a good time to upgrade your router firmware to the latest version, and change to new strong PSKs (or switch it over to RSA 2048+ instead)
The AppID refers to the XKEYSCORE signature name. I’m glad to see that IKEv2 is finally catching on, even within the NSA :)


This slide is mentioning IPsec scripts, which might refer back to the previously mentioned genericIPSec_wrapper.pl script. This clearly indicates an IKE PSK brute force decrypting engine. If you are using weak PSKs, the NSA is decrypting your IKE traffic which will give them the keys to decrypt your IPsec traffic. Enabling PFS will help you a bit, but ultimtely you should really just switch to using RSA (or ECP) authentication and skip PSK’s alltogether. Note that the FIPS standard already disallows using PSK’s for IKE/IPsec. The NSA knows it is just too dangerous and weak.

Note to the NSA, please see RFC 4301 Section 1.1 which states:

The spelling “IPsec” is preferred and used throughout this and all related IPsec standards. All other capitalizations of IPsec (e.g., IPSEC, IPSec, ipsec) are deprecated.

On slide 31 it states: IPSec: 6640 (protocols) and 6648 (ports)

I don’t know what that refers to. Possible backhaul transport ports or part of an API port ?

On slide 33 it states: Need all the pieces (IKE and ESP for IPSec)

This really shows that the NSA is breaking IPsec by breaking IKE, which mostly seems to come down to weak PSKs and no PFS. And again, in case you still did not get it on slide 39:


“TAO got the configuration files which provided us the PSKs to enable passive exploitation”

So your IPsec VPN is getting owned because your VPN gateway got owned! Run your VPN on a dedicated machine, preferably opensource, and lock remote access via ssh down using strong keys and IP filtering.

And the last interesting nugget is on page 40:


“NSP had an implant which allows passive exploitation with just ESP”.

I read this to mean that the hardware or software of the system running IPsec was compromised, causing it to send valid protocol ESP packets, but creating those in such a way that these could be decrypted without knowing the ESP session keys (from IKE). Possibly by subverting the hardware number generator, or functions related to IV / ICV’s / nonces that would appear to be random but were not. Or less likely if the hardware/software was using padding bytes to reveal decryption keys.


If anything, I would say that IPsec and IKE have withstood the attacks of the NSA. The fact that the NSA needs implants and needs to use brutce force attacks against IKE PSKs shows that despite its kitchen sink nature, IKE and IPsec are doing quite well if you configure it properly and run it on a secure host. But misconfiguration of IKE is something we did a poor job of preventing. We can and should have more automatic IKE/IPsec deployed using strong default configurations.

And you should hear back from The Libreswan Team in the next two months regarding just that. Opportunistic Encryption: Round Two

deFUDing “DNSSEC has failed”

It’s easy to forward or like an article of which the title says something you agree with, without actually reading the content. It seems “DNSSEC has failed” is one of those that showed up in my twitter feed. It’s so defunct of facts, it’s pretty clear most people have just read the title and retweeted it. Hopefully, they will at least read this post before not retweeting it.

It starts comparing the DNSSEC protocol with the SSH protocol. It claims ssh just works. It fails to take into consideration that the two are completely different trust models. With ssh, you tend to own both end points. So there is no issue of PKI or key distribution or trust. And even so, I am willing to bet the author never actually checks his SSH fingerprints when connecting to new servers, and just depends on the ssh leap of faith. Well, security is easy when you can assume you are not under attack. Most ironically, the easiest and only cross-trust-domain SSH public key distribution system out there is the DNSSEC protected SSHFP record. The only other one I know of is GSSAPI and other hooks into centrally-managed systems – eg all systems under one administrative control. Trust is pretty easy when you centrally distribute trust anchors.
Let’s also not talk about the debian ssh disaster, which was a pretty spectacular failure compromising thousands of ssh servers. Let me conclude the ssh comparison with inviting the author to login to bofh.nohats.ca using ssh in a fully trusted matter. He will end up 1) calling me to verify the ssh host key, 2) trusting the SSHFP key in DNSSEC or 3) leap-of-faithing it and hoping he was not under attack.

Next, the author brings up the tremendously easy to use and success of TLS. Never mind that we still need to run SSL 2.x for WinXP compatibility, most servers run TLS 1.0 which is severely broken, we just had the TLS 1.2 resumption/renegotiation disaster, about 4 CA compromises in 2013, and browsers running to patchwork solutions like hardcoded certificate pinning and Certificate Transparency registries that uses “n of m” style “it is probably the right certificate” solutions. There are about 600 (sub)CA’s that can sign any TLS certificate for any domain on the planet, ranging from Russia, to Iran to the U.S – all parties that are known to coerce parties and with a heavy industrial espionage component.

If TLS was successful, port 80 would be dead. Port 80 is the proof that the TLS trust model has failed. It involves money and so these certifications are only available to corporations. While DV certificates are now basically free, they are also considered worthless (protection against passive attack only) EV certificates are now dropping in price – which will make them finally available for more people but unfortunately because of that also lose their value. And all of this security hinges on the (mostly hidden) CABforum and browser vendor decisions on which CAs to include, and which CAs not to include.

While on most proprietary OSes, these preloaded CAs are at least managed by the vendor, the situation in Linux is a disaster. OpenSSL doesn’t support blacklists, NSS does. GnuTLS I don’t even know. Where are these CAs and blacklists? Do you know applications using openssl for their TLS are using the same CA bundle as applications using NSS or GnuTLS? Only very recently has Fedora and RHEL moved to a system where this is guaranteed. Even worse, most python wrappers using TLS don’t even check anything of your TLS connection. Presenting the wrong certificate will just cause that software to continue. And then look at the latest Apple TLS bug where anyone could bypass all TLS checks completely. How the author can even point to TLS as a security success is a complete mystery to me. It’s the worst computer security disaster we ever created! Look at the amount of complete failure TLS deployments at SSL Pulse – 30% allows weak ciphers, 6% has a broken certificate chain, less then 30% supports TLS 1.2, 6% vulnerable to the renegotiation/resumption attack, 70% vulnerable to the BEAST attack, 13% vulnerable to the compression attack, 56% allow RC4 and 50% does not support PFS. And that according to the author is the “success” that DNSSEC should take a lesson from?

Having listed the “success” stories of ssh and tls, he than looks at DNSSEC. His first factual statement regarding DNSSEC is “Most resolving software supports DNSSEC, but none have enabled it by default“. He is wrong. I have been the package (co)maintainer of bind, unbound and nsd for Fedora and EPEL for many years. On Fedora and EPEL/RHEL the default install enables DNSSEC for all three, and has done so for many years – even before the root was signed with DNSSEC!

Next he claims “To enable DNSSEC we have a ‘tutorial’ by Olaf Kolkman which spans a whopping sixty-nine pages“. As I said, the author is wrong. Simply “yum install bind nsd unbound” and the software comes with dnssec enabled, using the root key as trust anchor.

He then points to a DNSSEC training that takes “multiple days” and claiming that shows the complexity. The same vendor lists a two day DHCP course, a two and a five day “plain DNS & bind” course, and a two and five day IPv6 course. Does that mean those technologies are all failures too because it takes so many days to teach a course about it? According to the author, DHCP would be a failure and no one can deploy it because it “requires” a two day training? It’s just simplistic cherry-picked FUD.

His next argument is “Furthermore, there are almost no simple instructions for enabling DNSSEC, especially none that use KSKs and ZSKs, and include instructions on how to do key rollovers.“.

First, let me ask you how you do an ssh host key rollover? How do you make that seamless so it won’t trigger anyone who ever connected before from seeing a changed key? Where do they go to confirm the changed key is installed by the administrator and not the result of a hacked server or connection? Does this mean ssh has failed too? Again, amusingly enough the only way to rollover ssh keys is piggybacking on SSHFP in DNSSEC or GSSAPI trust models. But I don’t see the author claiming ssh has failed because it cannot do rollovers on its own. More cherry-picking.

The reality is, ssh and DNSSEC can do fine without rollovers. Only TLS requires rollovers because the trust model depends on the fact that you regularly pay the trustees managing that whole disaster. But if you do want or need to do a rollover, with ssh you’re fucked, with TLS you have to pay and with DNSSEC you have a seamless transition method. Let me repeat that – only DNSSEC actually implements a free and seamless key rollover mechanism! So of course, the author just states it is too hard to use. But again does not compare it against TLS or ssh at all. Cherry-picking all the way down. And by the way, there are also good reasons why you do not need to worry about DNSSEC rollovers at all.

As I said before, DNSSEC is already “enabled”, so let’s interpret the author’s quest for “simple instructions to enable DNSSEC” to mean “how do I sign my zone?”. Let us assume he is running bind right now with his zone in /var/named/1sand0s.nl. The author now has to issue the following complicated sequence of commands:

$ dnssec-keygen -f KSK -a RSASHA256 -b 2048 -n ZONE 1sand0s.nl
Generating key pair........+++ ..+++
$ dnssec-keygen -a RSASHA256 -b 1024 -n ZONE 1sand0s.nl
Generating key pair........................................++++++ ..................++++++
$ cat K1sand0s.nl.*.key >> 1sand0s.nl
$ dnssec-signzone -o 1sand0s.nl -k K1sand0s.nl.+008+01699 1sand0s.nl K1sand0s.nl.+008+24363
Verifying the zone using the following algorithms: RSASHA256.
Zone fully signed:
Algorithm: RSASHA256: KSKs: 1 active, 0 stand-by, 0 revoked
ZSKs: 1 active, 0 stand-by, 0 revoked

Edit named.conf and load the file 1sand0s.nl.signed instead of 1sand0s.nl and you’re done. And whenever you change your zone, re-run that dnssec-signer command.
(NOTE: and as pointed out to me, if you don’t have zone edits every week, do run that signer command in a cron job weekly to refresh the DNSSEC signature records)

You can put this into a git hook and maintain the whole thing without ever realising there is a signing step in between. Hardly rocket science. I think we can argue whether or not this is harder or easier than creating a CSR for Apache, get it signed, and than installed (and too many people will forget to actually get the intermediate CAs installed in apache, and their deployment is actually even broken!)

The author’s issue with opendnssec is that it is “yet another daemon” using a “database backend”. I guess he is also not running postfix because that thing has more daemons than I can remember the names of. Opendnssec’s default “database backend” is sqlite. If you hate those, there is a lot of software you’ll going to have to purge from your servers. Sqlite is a perfect backend for small databases that are just too complex for simple text files. For those of us who don’t mind two extra daemons and some sqlite files, to sign your domain from scratch with opendnssec:

# yum install opendnssec
# systemctl enable opendnssec.service
# systemctl start opendnssec.service
# ods-ksmutil zone add –zone 1sand0s.nl –input /var/named/1sand0s.nl –output /var/named/1sand0s.nl.signed
# ods-signer sign 1sand0s.nl

Change named to load the .signed zone instead of the unsigned zone and done. Shockingly complicated! Unsuitable for “common engineers” according to the author.

To make these instructions complete, the last step is to send your DS record to your Registrar which involves using their custom webgui to send them the following data:

yum install ldns
dig dnskey 1sand0s.nl > 1sand0s.nl.dnskey
ldns-key2ds -n 1sand0s.nl.dnskey

And if you want to do all of this within the same bind daemon and without sqlite, you can use bind’s builtin inline-signing management features instead.


Saying the tools are there for ssh (which doesn’t have ANY rollover support) and tls (which over half the world has misconfigured regardless of rollovers) but not for dnssec, is really short-sighted and wrong. What the author has done is taken his prejudice, and written some cherry-picked text justifying it. The only credit I can give him, is that he did not have to rely on the FUD statements by djb.

Development of libreswan vs openswan


UPDATE: Statistics updated to include 2017

Yesterday, openswan released version 2.6.40 to address CVE-2013-6466. You might be confused by its changelog (not the non-updated CHANGES) crediting me for the vast majority of code changes. . Basically all changes are pulled from the libreswan repository and are backports to openswan. The exception is their version of a patch for CVE-2013-6466. Libreswan’s fix is not a band-aid but an updated state machine. The backported libreswan fix is what is going into the updated openswan packages for RHEL5 and RHEL6 and are available at libreswan.org/security/openswan/CVE-2013-6466. RHEL7 will contain libreswan.

Basically, yesterday’s openswan 2.6.40 release brings it up to the initial libreswan-3.0 release of two years ago, plus the two CVE issues. Except that it crashes KLIPS and backported a libreswan commit that broke all non-XAUTH IPsec connections.

History and implementation status of Opportunistic Encryption for IPsec

(as sent to the cryptography mailing list)


In light of the NSA achievements, a few people asked about the FreeS/WAN IPsec OE efforts and whatever happened to it.

The short answer is, we failed and got distracted. The long answer follows below. At the end I will talk about the current plans that have lingered in the last two years to revive this initiative. Below I will use the word “we” a lot. Its meaning changes based on the context as various communities touched, merged, intersected and drifted apart.

NOTE: On September 28, 2013 there is be a memorial service in Ann Arbour for Hugh Daniel, manager of the old IPsec FreeS/WAN Project. Various crypto people will attend, including a bunch of us from freeswan. Hugh would have loved nothing better than his memorial service being used as a focal point to talk about “new OE”, so that’s what we will do on Saturday and Sunday. If you are interested in attending, feel free to contact me.

OE in a nutshell

For those not familiar with IPsec OE as per FreeS/WAN implementation. When activated, a host would install a blocking policy for Every packet to an IP address would trigger the kernel to hold the packet and signal the IKE daemon to go find an IPsec policy for that destination. If found, the tunnel would be build, and an IPsec tunnel to the remote IP would be established, and packets would flow. If no policy was found, a “pass” hole was poked so packets would go out unencrypted. Public keys for IP addresses were looked up in the reverse DNS by the IKE daemon based on the destination address. To help with roaming clients (roadwarriors), initiators could store their public key in their FQDN, and convey their FQDN as ID when performing IKE so the remote peer could look up their public key in the forward DNS. This came at the price of two dynamic clients not being able to do OE to each other. (turns out they couldn’t anyway, because of NAT)

What were the reasons for failing to encrypt the internet with OE IPsec (in no particular order):

1) Fragmentation of IPsec kernel stacks

In part due to the early history of FreeS/WAN combined with the export restrictions at the time. Instead of spending more time on IKE and key management for large scale enduser IPsec, we ended up wasting a lot of time fixing the FreeS/WAN KLIPS IPsec stack module for each Linux release. Another IPsec stack, which we dubbed XFRM/NETKEY appeared around 2.6.9 and was backported to 2.4.x. It was terribly incomplete and severely broken. With KLIPS not being within the kernel tree, it was never taken into account. XFRM/NETKEY remained totally unsuitable for OE for a decade. XFRM/NETKEY now has almost all functionality needed – I found out today it shoudl finally have first+last packet caching for dynamic tunnels, which are essential for OE. Since the application’s first packet triggered the IKE mechanism, the application would start retransmitting before IKE was completed. Even when the tunnel finally came up, the application was usually still waiting on that TCP retransmit. David McCullough and I still spend a lot of time fixing up KLIPS to work with the current Linux kernel. Look at ipsec_kversion.h just to see what a nightmare it has been to support Linux 2.0 to 2.6 (libreswan removed support for anything lower then recent 2.4.x kernels)

Linux IPsec Crypto hardware acceleration in practise is only possible with KLIPS + OCF, as the mainstraim async crypto is lacking in hardware driver support. If you want to build OE into people’s router/modem/setup box, this is important, though admittingly less so as time has moved on and even embedded hardware and phones are multicore or have special crypto CPU instructions.

An effort to make the kernel the sole provider of crypto algorithms that everyone could use also failed, and the idea was abandoned when CPU crypto instructions appeared directly accessable from userland.

2) US citizens could not contribute code or patches to FreeS/WAN

This was John Gilmore’s policy to ensure the software remained free for US citizens. If no US citizen touched the code, it would be immune to any presidential National Security Letter. I believe this was actually the main reason for KLIPS not going in mainstream kernel, although personal egos of kernel people seemed to have played a role here as well. Freeswan people really tried had in 2000/2001 to hook KLIPS into the kernel just the way the kernel people wanted. (Ironically, the XFRM/NETKEY hook so bad, it even confuses tcpdump and with it every sysadmin trying to see whether or not their traffic is encrypted) I still don’t fully understand why it was never merged, as the code was GPL, and it should have just been merged in, even against John’s wishes. Someone would have stepped in as maintainer – after all the initial brunt of the work had been done and we had a functional IPsec stack.

In the summer of 2003, I talked to John and together we agreed it was time to fork. Openswan was born to clearly indicate US coders could contribute. However, at that point the (then crappy) FRM/NETKEY IPsec stack was there to prevent OE from working due to the missing first+last packet caching. The FreeS/WAN Project ended and Openswan continued. At first in good pace, but that later slowed down and OE was no longer its focal point. (Due to legal reasons, I cannot go into details regarding the openswan history)

3) Not using DNS without DNSSEC

There were various issues that caused DNSSEC to get massively delayed. We needed DNSSEC to secure our DNS based distributed public key platform. Although it would have worked fine to use DNS against passive attackers (NSA trawling), we believed it was principly wrong to trust cryptographic material that was untrusted and vulnerable against active attacks. So while the developers encouraged people to put keys in DNS even without security, no one else picked it up. It sucks to need to say ‘we told you so’. But we should have really not waited on DNSSEC.

4) Dealing with the DNS working groups at IETF

The DNS community is one of the most pedantic group of people I know. They are very smart, often right, and had been known to be extremely defense of their DNS turf. (Note that things have improved considerably and if you think this is still an issue, I’m happy to try and help)

IETF was divided about the convergence of the “security of the DNS” and the “DNS as PKI” despite that this had always been a goal of DNSSEC for a large group of people within the IETF. The FreeS/WAN people were driving DNSSEC not so much for DNS as for the key distribution. After all, you can detect DNS forging if you know your public keys.

When we had the KEY/SIG records ready to go, it was decreed that it could only be used for the DNS itself. Applications could not use this KEY record. To make that distinction more clear, on the next change in the draft protocol, KEY was obsoleted and DNSKEY introduced. So IPsec keys were relegated back to TXT, since at the time we had no Generic Record format (RFC 3597) support, so waiting for any new RRtype to get any deployment to become usuable would take years. Almost everyone was on bind4 and never upgraded left us with no other choice but the TXT. Even though we wrote the OE and IPSECKEY RFCs, OE’s only deployments were done using TXT records.

5) DNSSEC was delayed by a decade

DNSSEC deployment was slowly gaining traction, but I think we really needed the Kaminsky bug to get that extra push for DNSSEC outside the geeks of the IETF. The US government mandate for DNSSEC in .GOV helped as well. But by this time, OE was mostly forgotten.

djb repeatedly tried to peddle his own warez. While not at all realistic, it always gained a lot of hype and media attention and probably did cause delays of DNSSEC deployment.

Kaminsky himself was shooting down DNSSEC too. I personally heckled him at various Black Hat’s and ICANN conferences until we finally sat down for a couple of hours to talk about DNSSEC’s history and design goals. I’ll claim my 15 minutes of fame for having converted him. It helped having Kaminsky say that although he didn’t like the complexity, he couldn’t see anything better. DNSSEC was needed for everyone.

DNSSEC was gaining traction. Then we ran into a bunch of DNSSEC deployment issues. We had the delays due to NSEC vs NSEC3 with OPTIN, and then on top of that in 2008 when the first big ISP in Sweden turned on DNSSEC in their resolvers all that traction was blown away.

Most consumer routers ran DNS proxies that implemented DNS as “known bitstreams” instead of implemeting the actual DNS protocol. The DNSSEC OK bit caused thousands of routers to drop DNSSEC packets as “invalid DNS”. The only realistic solution: Turn it off and wait two years for those routers to get obsoleted by faster wifi standards and talk to those vendors so they would not repeat their mistake with their next generation of routers.

We now have the IPSECKEY record format (though RFC 4025 is not useful, see below) and RFC 3597 for the generic DNS record deployed on all DNS servers. And we’re on our way to have DNSSEC on every end node (see also draft-wouters-edns-tcp-chain-query-00 I just submitted to the IETF)

We have a mostly clean working UDP/TCP port 53 transport for DNSSEC on most networks (in part thanks to Google DNS). Although our hotspot handling is still a little rough, with dnssec-trigger the only tool to hack configurable DNSSEC support into the OS for our coffee shop visits when we need to rely on forged DNS.

6) When you’re NAT on the net, you’re NOT on the net.

Opportunistic Encryption relied on a clear peer to peer connection. But we managed to degrade the internet into servers and clients. NAT was the biggest problem, and with CGN around the corner, it’s not something that is going away despite IPv6 offering enough IPs for everyone. In fact, for our “new OE”, this is the biggest hurdle to overcome. When Alice cannot talk to Bob because she cannot reach him due to a (carrier grade) NAT, we are stuck wildly poking holes and hoping packets flow.

7) The reverse DNS tree is dead Jim

OE depended on the reverse tree as a security mechanism that someone who was claiming a public key for a specific IP range was actually the legitimate owner of that IP space. It was the security method for RFC-4025.

But unless you are running in a datacenter, you do not have access to the reverse DNS. It is useless as key distribtion method. On top of that, large IPv6 deployments don’t even care any more to run any authoritative DNS for their reverse.


The IETF tried to revive this OE with the Better Then Nothing Security (“BTNS”) working group. Contrary to the name, they also fell into the “perfect is the enemy of good” trap and most discussion seemed to go into “channel binding” to upgrade anonymous IPsec to some kind of authenticated IPsec – at least by the time I became aware of them. In other words, the most important problem of key distribution was left outside the scope and no one actually seemed to have implemented anything. Though I have to admit, I’m behind on reading the VPN auto-discovery drafts. It is just
very discouraging to still be reading problem statement drafts. More over, I don’t think we should setup IPsec tunnels based on packets hitting the kernel. We have better ways now that we can leverage DNSSEC.

9) We were all complacent

The only interest for IPsec was for corporate VPNs. During the above listed problem periods, OE people gave up. Some walked away from IETF. While everyone gained an always-on portable IP device,
their crypto capabilities were practically non-existent. My current iphone 5 can connect to a corporate VPN, but trying to make it _just_ send out encrypted packets is impossible. Some trickery can be used to cause almost any packet to setup the VPN, but while that’s going on it is still leaking like a sieve. VPN is seen by phone vendors as a method to gain some enterprise users, not as the tool to protect the consumer. The Apple VPN client is a 10+ year old patched version of racoon. The only vendor that took VPNs seriously was RIM and we punished them by not buying their products, because we had other priorities like FourSquare, Facebook and Twitter.

We can only hope that those PRISM players are now put under economic pressure by frightened consumers to fix this. But as long as VPNs and DNSSEC is slow and error-prone, it is better for them not to go there.

The New Opportunistic Encryption

I’ve been brainstorming with various people on how to put IPsec OE back on the table. I’ve discussed this with a bunch of people around me, including the late Hugh Daniel, John Gilmore and Hugh Redelmeier of freeswan.

The packet capturing policy is not a good method because we cannot make any decision on where to find a public key for an IP address. The reverse is unusable, and IP addresses change often. We used it because we had nothing better. But now we do. Since every (secure) platform now runs DNSSEC on the end node, we can use this as our decision making point. Imagine my phone running a DNSSEC resolver (say unbound) and an IKE daemon (say libreswan). The DNS server has access to the set of DNS name and matching IP address. It can lookup the key in the forward DNS zone, and hand over the public key, dns name and IP address to the IKE daemon!

1) User tells browser to go to www.cypherpunks.ca

2) browser does a lookup for the A/AAAA record of www.cypherpunks.ca

3) DNSSEC resolver performs the lookup/validation for the A/AAAA record of www.cypherpunks.ca and additionally looks up the IPSECKEY record of www.cypherpunks.ca.

4a) The resolver will wait with returning the A/AAAA record to the browser until it knows if the IPSECKEY record exists or not. If not, it releases the A/AAA answer to the application. Packets flow in the clear.

4b) The resolver finds an IPSECKEY record. It sends the pubic key, the FQDN and the IP address(es) to the IKE daemon and waits for a response. Meanwhile it does _not_ release the A/AAAA record to the application.

5) The IKE daemon sets up the IPsec tunnel. We haven’t reached agreement yet over how this should be done. There are two choices:

a) The client uses an “@anonymous” ID for itself along with sending its public key inline with IKE. The client is responsible for ensuring there is no MITM attack, as it knows the server’s public key (from DNSSEC). The responding server will just use any key it received inline if it was received for the “@anonymous” ID.

b) The initiator (aka client) uses its own FQDN-based ID. It has preconfigured its DNS so that an IPSECKEY record exists for its FQDN (protected by DNSSEC). The key is not send inline with IKE. Instead, when the responder (aka server) sees the non-anonymous ID, it will perform a DNSSEC secured lookup to obtain the IPSECKEY out of band. Both parties confirm there is no MITM.

The advantage of a) is that it leaks less user information and makes tracking users harder. The client can regularly generate another anonymous keypair. The disadvantage of a) is that it turns peers into clients and servers. And two clients cannot initiate OE to each other.

6) The tunnel is established and the IKE daemon notifies the local DNSSEC server that had instructed it to setup the IPsec tunnel.

7) The resolver releases the IP address to the application.

8) The applications starts sending packets and the IPsec policy encrypts them al.

I’m personally in favour of the @anonymous solution. But there is no reason why support for both could not be implemented.

What are some of the obstacles and work to do:

1) writing the unbound plugin

2) writing the support for @anonymous for the server-side. This includes raw keys for IKEv2 (draft-ietf-ipsecme-oob-pubkey)

3) With NAT, the client suggests an inner-IP. This could be abused or clash, We need to ‘contain’ each connection, possibly using generated ipv6 addresses 4) We cannot use the “gateway” field of RFC-4025, or people could trick a server into giving a client all communication to a certain IP address that does not belong to them

5) anonymous connections should generate throw-away keys to remain anonymous

6) implement draft-wouters-edns-tcp-chain or else latency/RTTs will prevent real-life deployment of DNSSEC validated IPSECKEYs on mobile devices.

7) This allows no upgrading from anonymous to mutually authenticated, but IKE policies can be added to the server/client that would match on different IDs (eg X.509) that work independantly of OE without introducing complicated channel binding promotion code. Other IKEv2 extensions could possible be applied to facilitate promotions.

I’m sure more implementation issues will show up once we get this going, but there are no real fundamental issues why we cannot deploy this in a couple of months of time. My plan is to get libreswan to support this version of OE. Additionally, once we use draft-wouters-edns-tcp-chain, it becomes cheap to do these lookups through the tor network. If the tor exit nodes then also feed each other with DNSSEC cache material, it should make tracing individual clients even harder.

(anyone willing to assist, especially with coding, do contact me)

Philips Hue alternative for “Lamp Stealer” using telnet

When you buy two Philips Hue light start kits, you have the problem that the lights are already paired with the bridge in each starter pack. When you search you will find a lot of people whining about how unfair this is and people talking about the “Lampstealer” OSX app that Philips released to fix it. I tried using the lamp stealer app but it would never find my bridge. I could also not use QuickHue which supposedly supported the lamp stealer function because it was compiled for OSX 10.8 and I still run 10.7.x. And compiling it from source with xcode didn’t work, likely due missing libraries and other mistakes I made since I’m not too familiar with Xcode.

I found out that the solution was really really simple, and requires no OSX, java or advanced rocket science. Place a bulb of the second starter kit into a socket within 30cm of the bridge from the first starterpack. Telnet to port 30000 of the bridge and type:


The light should blink a few times to acknowledge the hostile takeover. Now you can use your iphone hue app. Go into the app settings, select the bridge, then run “find new lights”, or us your own python code (I like phue.py). Change bulbs and repeat this process for the other two lights.

Skytech and Dawson College versus Ahmed the kid

In case you had not heard it, the information of about a quarter million Canadian students was compromised in a security incident last week. These kind of breaches are so common place now, I hardly take notice of them any more. But an article about this case appeared in the National Post, “Youth expelled from Montreal college after finding ‘sloppy coding’ that compromised security of 250,000 students personal data

It turns out 20 year old Ahmed Al-Khabaz found a flaw that was trivial. He reported it to the vendor. The data was not leaked online on pastbin – at least, not by Mr.Al-Khabaz. But others before Al-Khabaz could have come in, copied the data, and left.
According to the article,

After an initial meeting with Director of Information Services and Technology François Paradis on Oct. 24, where Mr. Paradis congratulated Mr. Al-Khabaz and colleague Ovidiu Mija for their work and promised that he and Skytech, the makers of Omnivox, would fix the problem immediately, things started to go downhill.

Two days later, Mr. Al-Khabaz decided to run a software program called Acunetix, designed to test for vulnerabilities in websites, to ensure that the issues he and Mija had identified had been corrected. A few minutes later, the phone rang in the home he shares with his parents.

It was Edouard Taz, the President of Skytech, the company who wrote the Omnivox software. Taz threatened to report Al-Khabaz to the RCMP promising six to twelve months jail time unless Al-Khabaz agreed to immediately meet up and sign an NDA. It goes down hill from here. Dawson College expelled the student for a “serious professional conduct issue”. Fourteen out of fifteen professors in the computer science department voted in favour to expel Al-Khabaz. And the result is that a smart young kid, who did the right thing, and made a little mistake out of curiosity, is facing serious long term effects of not being able to get a college degree. And with him having nothing left to lose, the NDA they forced him to sign became useless, and Skytech lost as well because we all now know how crappy the security of Omnivox is. Edouard Taz shot himself in the foot. It’s just too bad he had to take down Al-Khabaz with him.

And the vulnerability? We don’t know we can assume it was something as trivial as manually editing the URL in the address bar, or some simple SQL injection attack. Something that in any other industry would be called “gross negligence”, but for some reason IT companies get away with delivering cars with faulty breaks and calling the slippery road an “Advanced Persistent Threat”. And companies like Skytech, with the assistance of an ill-informed Computer Science department at Dawson College, take it out on a young kid who honestly tried to do the right thing. Security through bullying. The response of both Skytech and Dawson College was over the top, and counter-productive to everyone’s interest.

Compare this to the response of my old university, the Radboud Universiteit Nijmegen. A 19 year old kid spent all his money on a 1200 baud modem (half duplex) and uses it to login to a university LAT-TCP server to connect to something called “the internet”. He uses “ftp” and “sz” to download files, “nn” and “telnet” to read news and chat with people all over the world. He did much worse then Al-Khabaz, although he too was careful not to do any accidental damage. And like Al-Khabaz performing his scan of Skytech, this person knew he was a trespasser. An assistant professor known as “Sparky”, notices that a student who had not been active at the university, started logging in very regularly. He becomes suspicious that the account might have been compromised. So he sends a Unix Talk request to her (see man talk). The two briefly chat online. Sparky confirms the account is compromised. But instead of screaming and threatening like Mr. Taza, Sparky asks the hacker how he got into the account. The hacker tells him he used ypcat to get /etc/shadow, then ran a cracker. Sparky then tells the hacker he has two days to get his stuff from the account and leave, and not to come back using any other accounts. Three years later, our young hacker has actually enlisted with the university to study CS, and in his second year he becomes friends with Sparky – and even gets an account on the university’s historic PDP-11. He confesses the whole affair and both have a good laugh. The student later on starts an ISP, becomes a security consultant who speaks at Black Hat, and is part of a group that writes VPN software in use all over the world. And the young-hacker-turned-sysadmin also has to deal with hackers himself. One time he logs into an irc channels to tell a group of file sharing hackers that they overstayed their welcome and should not have filled up the disks, and that it was time to go – and probably stop stealing movies and ISP resources. The circle of not threatening kids who make mistakes continued.

I can’t imagine how my life would have turned out if Sparky had hunted me down and had threatened me with jail time or that he would have kicked me out of university when I was 19. I learned a lot from Sparky, and I’ve passed along the same treatment to others. I wish Mr. Al-Khabaz all the best. He deserves much better then the treatment he received from Skytech and Dawson College. Don’t we all make those mistakes when we are that young? The real mistake here, is that Omnivox is a product with severe security flaws, that could have resulted in identity theft of 250,000 Canadian Students. Perhaps Mr. Taz should focus his bullying at his own security department?

Can the NSEC3-OPTOUT record cover no RRTYPE’s?

At first glance, the answer seems obvious: No. The point of NSEC3 is to proof the (non)existence of data. With the OPT-OUT flag set, you skip all the non-secure data. So you would expect to have some data to cover for such an NSEC3 record. If you read RFC 5155 Section 3.2.1 it is not entirely clear, though I’m tempted to think these should not happen in the wild. Except we just found one, generated by bind’s dnssec-signzone:

aaa.ottawa.on.ca.  IN NS foo.
problementry.ottawa.on.ca.  IN NS foo.
problementry.ottawa.on.ca.  IN DS blob.
zzz.ottawa.on.ca.  IN NS foo.

The change was that “problementry.ottawa.on.ca” submitted a DS record which was added to the zone. Note also that  neither “on.ca.” or “ottawa.on.ca.” are zone cuts, although there are many delegations underneath XXXX.ottawa.on.ca. (but none with a DS record or orphan glue)

This caused bind to add two NSEC3 entries as part of the NSEC3 chain, an entry for HASH(on.ca) and an entry for HASH(ottawa.on.ca) (forgive the wordpress wrapping of lines)

+qf6k201pudkgikusoidrujdfikql15qc.ca.   3600 IN      NSEC3   1 1 5 – qnkecttopnji0h479fhpjmv18gsl1sdk ;{ flags: optout}

+qnkecttopnji0h479fhpjmv18gsl1sdk.ca.   3600    IN      NSEC3   1 1 5 – r85l4g712aibs65e47aj79e7odi202h9 ;{ flags: optout}

Note how these NSEC3 entries do not cover a single RRTYPE, because there is no zonecut and there are no RRTYPE’s for these entries.

When signing this zone with opendnssec, instead of bind, the entries are not added. Who is right? Before using my brains, I decided to use other people’s brains.

named-checkzone loaded both the bind and ods version with “OK”. So no help there. It’s a bug because it should declare at least one of the copies as containing errors in the NSEC3 hash chain

ldns-verify complained about ALL the non-zone-cuts in the zones but otherwise showed no difference. So that’s also a failure, on top of the failure of assuming dots are zone cuts.

validns found the bind zone OK, but complained about the ods zone with:

ca.signed.ods-signer-01.ldns:2898958: no corresponding NSEC3 found for on.ca.
ca.signed.ods-signer-01.ldns:2907968: no corresponding NSEC3 found for ottawa.on.ca.

So validns seems to agree with bind. Which software is right? The only thing I”m sure of now, is that I need more coffee and time to re-read RFC 5155, and specifically look at how things are supposed to work in this case.

I’m continuously surprised at how many TLD’s have rolled out DNSSEC, yet we’re still seeing differences between signer engines and validators all the time……


nsd packages for Fedora/EPEL build that address CVE-2012-2978

I had only gotten around to build nsd 3.2.11 a few days ago and run it through testing when I was told about the 3.2.12 security release. So there was only a small (but important) code change to address the vulnerability issue. It ran on my own name server for nohats.ca for 24h without issues, so I could quickly pull the pending 3.2.11 updates and build and release updates today for 3.2.12 when CVE-2012-2978 became public.

You can find the Fedora and EPEL packages here: https://bugzilla.redhat.com/show_bug.cgi?id=841268

(of course, this was kind of a test posting to see if I show up in http://planet.fedoraproject.org)

The Chinese G101 tablet and Fedora 17 (Part 1)

So I managed to get a tablet that’s actually a full PC. It has no name, but you can find it  when you google for “tablet g101 atom“.

The tablet installing Fedora 17 from USB DVD. An iphone 3G is placed in front for size (and shape) comparison

It is not an ARM based device like many Android tablets, but an Intel Atom N450, 64bit, 2GB RAM, 160GB HD, WLAN 54 Mbps, 10.1″ capacitive multitouch touchscreen 1024×600, 3 USB ports, 1 (optional, not there?) 3G SIM slot, 1x HDMI, front facing webcam, sound, mic.

The amazing thing, Fedora 17 installed straight onto it from USB-DVD. I did not need to do any command line tinkering to load any kind of kernel module, driver or xorg driver. It installed, booted and all the hardware worked, though I haven’t yet tested the HDMI output. Wireless worked fine, sound and video worked, even the webcam! And of course the touch screen worked, though I haven’t confirmed multitouch, as I don’t know of an app that supports it to test it.

So what didn’t work? The biggest issue is that the machine cold boots from sleep mode, instead of resuming. It seems to enter sleep mode fine, and the LED blinks in that sleepy way, but it will just cold boot after that. The “home” button seems to just map to a return key. And I haven’t found the accelerometers yet, so it won’t rotate the screen automatically like it did with the pre-installed win8 beta. I tried using lm_sensors and i2c_detect but those do not find anything. When you rotate the screen using xrandr, the touchscreen input does not rotate its location to match, and I haven’t found the right xinput command yet. The speaker volume is a little on the low side.

Gnome3 – made for tablets!

There were a few things that made using this device a frustrating job, mostly because of gnome3. Which is odd because gnome3 was meant to be tablet friendly. You can select the Universal Access symbol and get a virtual keyboard to login, kudos there since I have no keyboard unless i bring along a USB keyboard. But when the virtual keyboard vanishes, it goes into that bottom tray bar, which auto-hides. It is next to impossible to get the tray bar to appear again to select the keyboard. If someone knows how to NEVER hide the tray bar, let me know, google only knows about people who want to never see the tray bar at all). Next, selecting the “activities” is way too hard. Pressing it just fails too often. The same for selecting “windows” or “applications”. This is not due to the touch screen being of bad quality, as for instance scrolling using the scroll bar in firefox works fine.  Scrolling through the application list most certainly will start the application you didn’t want, because your finger motion either didn’t scroll up enough (so it bounces back without scrolling) or it scrolled too far (hitting the top and thereby jumping out of the select, and you have to start from scratch). It’s a terrible terrible user experience.

Disk encryption

Since this is a mobile device, I opted to use full disk encryption. But that was a mistake. When you boot the device, you get the nice login prompt for the passphrase for your root filesystem, but there is no virtual keyboard. I had to plugin a USB keyboard to boot the device. We can’t really load X here, as the rootfs is still encrypted. I’m afraid there is no other way then to add a new virtual keyboard option, somehow.


All in all, it’s a nice device. I can’t wait to put it in the hands of gnome3 developers and say “go fix it!”.

I’m not sure if you can actually buy it anywhere. I bought this one on craigslist. It seems most links I found on google lead to Chinese shops who want to sell you at least 10. The price would probably be somewhere between 200-400 dollar each, for a version with a little less disk and ram then I tested with.

So, Linux is almost ready for the tablet!