Gold Certificate

"Dealing with real customers is a hard job," Katya declared from the safety of the employee breakroom. "Dealing with big companies is even harder!"

"I know what you mean," her coworker Rick replied, sipping his tiny paper cup of water. "Enterprise security requirements, arcane contract requirements, and then they're likely to have all that Oracle junk to integrate with ..."

"Huh? Well, that too, but I'm talking about Google."

"Google? What'd they do?" Rick raised an eyebrow, leaning against the wall by the cooler, as Katya began her story.

As the lead architect, Katya was responsible for keeping their customers happy—no matter what. The product was a Java application, a server that stood between legacy backends and mobile apps to push out notifications when things happened that the customer cared about. So when one of their biggest customers reported that 30% of the Google Cloud messages weren't being delivered to their devices in production, it was all hands on deck, with Katya at the helm.

"So I of course popped open the log right off," she said, her voice dropping lower for effect. "And what do you think I saw? CertPathValidatorExceptions."

"A bad SSL certificate?" Rick asked. "From Google? Can't be."

"You've done this before," Katya pouted, jokingly. "But it only happened sporadically. We even tried two concurrent calls, and got one failure, one success."

"How does that even work?" Rick wondered.

"I know, right? So we cURL'd it, verbose, and got the certificate chain," Katya said. "There was a wildcard cert, signed by an intermediate, signed by a root. I checked the root myself, it was definitely part of the global truststore. So I tried again and again until I got a second cert chain. But it was the same thing: cert, intermediate, trusted root."

"So what was the problem?" Rick asked.

"Get this: the newer cert's root CA was only added in Java 7 and 8, back in 2016. We were still bundling an older version of Java 7, before the update."

"Ouch," sympathized Rick. "So you pushed out an updated runtime to all the customers?"

"What? No way!" Katya said. "They'd have each had to do a full integration test cycle. No, we delivered a shell script that added the root CA to the bundled cacerts."

"Shouldn't they be worried about security updates?" wondered Rick

"Sure, but are they actually going to upgrade to Java 8 on our say-so? You wanna die on that hill?

"It just pissed me right off. Why didn't Google announce the change? How come they whipped through them all in two days—no canary testing or anything? I tell you, it's almost enough to make a girl quit and start an alpaca farm upstate."

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!