Private I: Trust and verify for network certificate roots

In a post on March 23, Google’s security team explained that it had discovered that someone was delivering digital certificates to users for Google domains that weren’t authorized by Google. A quick investigation discovered that a Chinese certificate authority (CA), CNNIC, had improperly given a reseller enough power to create verifiable certificates for any domain in the world.

With a verifiable certificate, any seemingly secured web connection can be intercepted by a party that can insert a tap into a network point between the browser and the server. It’s bad.

I’ll break down the details later in this article, but the critical fact is that this was apparently discovered and contained quickly. New mechanisms that have slowly been put in place to assure the integrity of secured Web sessions (and secured email and some other services) are—well, they’re actually working!

Mac and iOS users can’t yet take advantage of these with Safari, but it seems as if support is coming. If you’re lucky, you’ll never see an error that indicates a security connection has been redirected and hijacked. But if you do, this article will help.

Who’s in charge here?

If you want the full rundown on the CA system and how digital certificates work, you can consult my 2011 Macworld article “ Keep your Mac safe from Web security flaws.” After nearly four years, the basic infrastructure remains the same, but all the advice has changed. Some potential future improvements disappeared or stalled, while others have moved rapidly to deployment.

The tl;dr summary is that all secure web connections rely on a handshake between a browser and a server. The browser receives a digital certificate from the server, which contains its public encryption key, details of to whom it was assigned including a domain name or names, and a cryptographic signature that’s used to validate that none of the information in the certificate has been tampered with. The public key is used to protect an encryption key used for the current connection—a “session”—without requiring any other coordination between browser and server.

Several hundred entities around the world—companies, nonprofits, and some government agencies or companies closely affiliated with governments—act as certificate authorities, any one of which can sign a web server’s certificate on request (almost always for a fee). A CA provides another layer of trust and verification that the security document was provided by a party that controls the domain name that corresponds with the certificate.

CAs also need to be verified, and that involves baking some of their cryptographic data into operating systems, like OS X, Android, and Windows. Many browsers consult the OS list of CAs; some browsers do not and contain their own unique CA list. You can review Apple’s list of built-in CAs in OS X by launching Keychain Access (Applications > Utilities) and clicking System Roots in the Keychain list at left.

When a browser receives a certificate that can’t be verified, the connection fails and a user is warned. That’s one category of problem which you may have seen. It usually comes up accidentally, when a server is misconfigured or a certificate wasn’t created that includes all the domain names being handled by a given server.

But the other scenario, which I talked about in 2011, occurs when a legitimate, verifiable certificate is issued by a CA or one of its affiliates inappropriately. (Most CAs have reseller programs and allow third parties to sell certificates that are processed through a connection to the CA’s back-end systems.) Sometimes it’s an error, sometimes it’s a bad judgment, and sometimes it’s due to an attack on the CA or its affiliate.

In the case highlighted by Google Security, the root CA gave its reseller the keys to its kingdom: a private key that allowed the creation of a certificate that would work for any domain. This affiliate installed this into a data inspection device, often used in corporations and by governments to sniff secure data that passes across or between networks.

The legitimate use of this is in companies or agencies that disclose the behavior to employees because they have a need to scan for misuse of information or security leaks. A proper configuration involves configuring individual machines before they can use the network with a local certificate or proxy settings that allows this inspection.

What CA’s reseller did is now banned by all OS and browser makers as of a few years ago: installing a generic matches-everything certificate that can intercept all data, because that same certificate could be used anywhere in the world.

We don’t know precisely what triggered Google’s alert, and the company didn’t reply in time for this column to a query for more details. But based on its report, it was likely that users in the affected company or location who used Chrome received errors and reported those to Google. This will soon be a much more widespread option for more domains, and the warnings now appear in almost all modern browsers.

Pin the tail on the certificate

When a bad certificate is issued by any means, CAs should be able to issue a revocation—an automated “statement” of badness that’s sent out and consulted by any browser or other software before it accepts a certificate from any server.

In Keychain Access, in Preferences under the Certificates tab you can see (and set) the ways in which revocations are handled. Without getting into the weeds, the process isn’t considered reliable. Revocation servers aren’t always available, and if they aren’t, the lookup either locks your browser up or times out and accepts the certificate whether it’s revoked or not! (There’s a new way to manage revocations that’s gaining ground, but it’s not widespread enough to rely on yet.)

Instead, OS and browser makers release warnings and push micro-updates, often as automatic fixes, either to disable a particular certificate or a set of certificates, or to block an “intermediate” root certificate assigned from a CA to another party, like a reseller.

Two approaches have risen to the fore in providing sites and users with notification, though, one of which I mentioned in passing in 2011.

On the client side, “pinning” is a partial panacea for illegitimate certificates. Before pinning, any CA in the world, and any party they authorized, could issue a certificate that was valid for any domain in the world. Terrifying. It’s like letting a guy in an office in Brazil (or Kenya or Ukraine or Utah) make and sell keys to your apartment in Barcelona.

Pinning provides an explicit list of which CAs out of the hundreds that exist are entitled to issue a certificate for a domain. If a certificate appears that was signed by any other CA, bells and alarms go off. Google pioneered this and it’s now being expanded. Google pinned its domains inside of its Chrome browser starting in 2011, and let Chrome users enter local pins as well, useful for companies that installed Chrome in large numbers.

Mozilla (Firefox’s maker) added pinning in 2014 with version 32 for a set of domains, including its own and Twitter’s. It expanded those over subsequent releases to add Google and others.

That’s fine for these special cases, but shouldn’t this tool be available for all secure sites? I’ve used just a couple of CAs (though resellers) for the last few years for my web certificates, and it would be delightful to lock off any theoretical attacks against users fooled into thinking they’ve connected to one of my sites — much less a small credit union’s banking site or a major retailer.

A generic way to let any site publish via its web server which certificate authorities are valid has been in the works for a few years, and is beginning to be deployed. HTTP Public-Key-Pinning (HPKP) is the moniker. Firefox and Chrome support it currently, and it seems that it will be a necessary and required part of all major browsers in the future, though no deadlines are set or announcements made.

I can see right through this exploit

A second bit of help is coming from certificate transparency (CT), which Google is promoting and is still in the process of rolling out. With CT, every CA will have to publish information in a central log whenever (or even some number of hours before) a new certificate is issued. This allows Google and any other entity around the world to keep track of all legitimate certificates while also noting any that are issued by an authority without the authority to do so, based on pinning.

When CT is fully implemented in browsers and operating systems alongside pinning, a certificate that doesn’t appear in the corresponding CA’s certificate-issuing list or that fails a pinning test will give a user a chance to react. CT will also be used by companies like Google and independent security organizations to monitor actively for problematic security documents.

Pinning, and soon certificate transparency, absolutely do not solve all problems related to misuse of certificates. But on their own and together, they reduce the area of potential of harm by making it far harder for a sniffer to obtain a certificate and insert themselves into a secure connection without being immediately caught.

The alerts that browsers will provide will allow users quite legitimately to feel as if they are part of the effort to provide integrity to the Internet’s plumbing.

Correction: This reporter originally referenced a different secure web standard, HSTS, instead of HPKP, and thus overstated browser support. The article has been updated to correct that.