Four weak eMudhra keys broke any unpatched TLS connection

A platform team's Monday starts with a bug report from Friday

Picture a platform team running a few hundred services behind a shared egress proxy, with a certificate inventory that was last reconciled at the start of the quarter. On 17 October the original reporter, Wayne, signed a message with the private key of each of eight eMudhra intermediate CAs and reported it to the Root Program Teams. By the following Monday the team's monitoring has picked up the story, and the question on the call is not whether the keys are weak but which of their own systems still trust them.

The weakness is not a subtle one. Cloudflare describes eMudhra as having been trusted to pick a 115-digit number for a virtual combination lock and picking 4. The other seven keys were 56, 66, 115, 155, 156, 230 and 237. Because the affected intermediate private keys could be recovered almost instantly from the public certificates themselves, anyone could independently recover the same signing keys and create certificates for any domain that an unpatched client not enforcing Certificate Transparency would accept.

That is what separates this from the compromised certification authorities Cloudflare lists as precedent, DigiNotar, TÜRKTRUST and TrustWave. In those cases access to the keys was contained to some degree, so a limited set of attackers could issue fraudulent certificates. Here the recovery is a computation anyone can repeat, which means the population of potential forgers is not a list the team can reason about.

The clock is set by the root programs, not by the vendor

The eight certificates chain to roots trusted by the Mozilla, Microsoft, Apple and Chrome root programs and by many downstream trust stores. Those programs moved on their own schedules. The certificates have been blocked in Firefox and Chrome since 17 October, on Microsoft platforms since 20 October and on Apple products since 22 October. The issue was responsibly disclosed to downstream trust stores on 19 October, and Cloudflare says many have already released updates or are in the process of releasing them.

For the platform team, that timetable covers the browsers and the operating systems and stops there. Cloudflare's own follow-up page notes that many other clients do not consume revocation information and require software updates to ensure they no longer trust the compromised intermediates. It names applications and libraries that ship their own CA bundle, and apps built on Electron, such as Slack and Discord, which only pick up Electron's fix once they are updated to a version built on a patched release.

The page carries an update dated 28 October that sharpens the point for anyone on macOS: the developer tools Apple ships with the operating system, including git, curl and python3, still trust the compromised certificates, and macOS 27.2 undid user-applied fixes. A build agent running curl against an internal endpoint is exactly the kind of client that never sees a browser revocation list.

What Cloudflare saw when it logged the handshakes

Cloudflare started patching high-impact systems on 19 October. Customer origin-facing proxies using Pingora, responsible for most of its outbound connections, were patched in hours to reject the vulnerable eMudhra keys. An AI-assisted analysis of source code found the other systems that trust the weak keys, with a full analysis inside four hours, and lower-impact remediation was due by the end of the following week. All high priority Cloudflare systems were patched at the time of writing, with a small number of confirmed low impact tickets still being worked.

Logging added to Pingora finished rolling out on 21 October, and it is the most useful number in the write-up. A week later Cloudflare had seen about 22,000 attempted handshakes with the weak eMudhra keys. Two-thirds were fetches of test sites such as emudhra.we-have-a-problem.com. Almost all of the remaining third went to origins that appeared to have set up a certificate for themselves using the weak intermediate. Every handshake configured to verify the certificate was refused, because the proxies had already been patched.

About 160 handshakes stood out. Their certificates' Subject Alternative Names named other organisations' domains, unrelated to the Server Name Indication requested in the handshake, and some of those names were domains related to consumer IoT devices. The forged certificates were made after the keys leaked, and some were valid for over two years, longer than any public CA may issue. Cloudflare judged that they looked like servers' own default certificates rather than an attempt to intercept its connections, and they would have failed hostname verification anyway.

The timing is the part to take to a risk meeting. One of the forged certificates Cloudflare saw was dated 20 October at 18:19 UTC, less than an hour after Mozilla's bug report was made public. Cloudflare states there is no evidence of the weak keys being used to intercept its connections so far, and that this does not exclude carefully targeted interception. It also says it has not seen the usual indirect signals of system compromise.

Why a scanner will not settle it for you

The instinct on the Monday call is to point an external scanner at the estate and wait for a finding. That will not answer the question, because the exposure is not a service presenting a weak certificate. It is a client that would accept one if an attacker presented it, and a scanner looking from outside sees the server side of the handshake.

Cloudflare hit the same wall with its own visitors and had to work indirectly. It could not test whether a visitor still trusted the weak keys without breaking connections, so it used its challenge pages. It sent two probe connections on a fraction of challenge pages, one to ca1.cloudflarecaptcha.com and one to ca2.cloudflarecaptcha.com, initially installing an eMudhra-signed certificate on ca1 and a regularly signed certificate on ca2 as a control, then recording whether the probes succeeded. That is 50 million probes over the week, with the caveat Cloudflare states plainly: challenge page traffic is highly skewed, there are many unconfirmed bots in it, and only clients that execute JavaScript send probes.

The lesson for the platform team is that the answer lives in an inventory of clients, not in a scan result. The question to ask of each system is whether it validates certificates against the operating system trust store, which the root programs have already updated, or against a bundle it ships itself, which has to be replaced. The second list is the one that takes a week to assemble and the one that decides the risk.

What the vendor has and has not said

eMudhra announced it is exiting the public CA business and has claimed a mistake in a key generation script, but has not yet shared the script. Cloudflare also records that eMudhra signed a certificate revocation list with what might have been a correctly generated key and subsequently corrected it to be signed with the weak key. Cloudflare says much has to be cleared up, including the role of the auditors and cross signers.

Those are open questions, and they matter to procurement because they are the questions a supplier questionnaire will ask six months from now. They are not, however, the questions that decide this week's work. The keys are public, the root programs have acted, and the remaining exposure is entirely inside the estate.

The shortlist the team ends up with

By the end of the week the platform team is not choosing a product. It is choosing, for each class of client, whether the fix arrives through the operating system's trust store, through a vendor update, or through a bundle it has to rebuild and redeploy itself. The grid below is the one that goes into the ticket.

The distinction that carries the most weight is between clients that consume revocation information and clients that do not. Cloudflare's page is explicit that Mozilla, Microsoft, Apple and Chrome have remediated the issue for their users through tools for distributing revocation information such as OneCRL, CRLite and CRLSets, and that the CA has revoked the compromised intermediate certificates. Everything outside that set needs a software update, and the update has to actually reach the running process.

The checklist to hand over

Start with the inventory, not the scanner. List every system that terminates or initiates TLS and mark whether it uses the operating system trust store or a bundle it ships. The second group is the work.

Then work outward from the highest-impact client. Cloudflare's own order is the model: origin-facing proxies first, patched in hours, then a full analysis of everything else inside four hours, with high-impact systems patched immediately and lower-impact remediation scheduled. A team that starts with the build agents and the egress path will have covered most of the outbound risk before lunch.

Confirm the fix reached the process, not just the disk. An Electron app updated on the machine but still running the old build has not been fixed, and a macOS user-applied fix that macOS 27.2 undid has not been fixed either. Restart, rebuild images, and check the version that is actually executing.

Finally, keep the logging. Cloudflare's Pingora logging is what turned an unknown into about 22,000 attempted handshakes and 160 anomalies worth reading. A team that adds the equivalent for its own egress path will know within a day whether anyone is presenting a certificate it should not accept, rather than finding out from the next blog post.