Tyler Zale's organization is a automation success story of configuration-as-code. Any infrastructure change is scripted, those scripts are tested, and deployments happen at the push of a button.

They'd been running so smoothly that Tyler was shocked when his latest automated pull request for changes to their HAProxy load balancer config triggered a stack of errors long enough to circle the moon and back.

The offending line in the test:

assert File(check_lbconfig).exists and File(check_lbconfig).size == 2884

check_lbconfig points to their load balancer config. Their test asserts that the file exists… and that it's exactly 2884 bytes long. Which, of course, raises its own question: if this worked for years, how on Earth was the file size never changing? I'll let Tyler explain:

To make matters worse, the file being checked is one of the test files, not the actual haproxy config being changed.

As it turns out, at least when it comes to the load balancer, they've never actually tested the live config script. In fact, Tyler is the one who broke their tests by actually changing the test config file and making his own assertions about what it should do.

It was a lot more changes before the tests actually became useful.