If we presume there is a Hell for IT folks, we can only assume the eternal torment involves configuring or interfacing with printers. Perhaps Anabel K is already in Hell, because that describes her job.

Anabel's company sells point-of-sale tools, including receipt printers. Their customers are not technical, so a third-party installer handles configuring those printers in the field. To make the process easy and repeatable, Anabel maintains an app which automates the configuration process for the third party.

The basic flow is like this:

The printer gets shipped to the installer. At their facility, someone from the installer opens the box, connects the printer to power, and then to the network. They then run the app Anabel maintains, which connects to the printer's on-board web server and POSTs a few form-data requests. Assuming everything works, the app reports success. The printer goes back in the box and is ready to get installed at the client site at some point in the near future.

The whole flow was relatively painless until the printer manufacturer made a firmware change. Instead of the username/password being admin/admin, it was now admin/serial-number. No one was interested in having the installer techs key in the long serial number, but digging around in the documentation, Anabel found a simple fix.

In addition to the on-board web-server, there was also a TCP port running. If you connected to the port and sent the correct command, it would reply with the serial number.

Anabel made the appropriate changes. Now, her app would try and authenticate as admin/admin, and if it failed, it'd open a TCP connection, query the serial number, and then try again. Anabel grabbed a small pile of printers from storage, a mix of old and new firmware, loaded them up with receipt paper, and ran the full test suite to make sure everything still worked.

Within minutes, they were all happily churning out test prints. Anabel released her changes to production, and off it went to the installer technicians.

A few weeks later, the techs call in through support, in an absolute panic. "The configuration app has stopped working. It doesn't work on any of the printers we received in the past few weeks."

There was a limited supply of the old version of printers, and dozens got shipped out every day. If this didn't get fixed ASAP, they would very quickly find themselves with a pile of printers the installers couldn't configure. Management got on conference calls, roped Anabel in on the middle of long email chains, and they all agreed: there must be something wrong with Anabel's changes.

It wasn't unreasonable to suspect, but Anabel had tested it thoroughly. Heck, she had a few of the new printers on her desk and couldn't replicate the failure. So she got on a call with a tech and started from square one. Is it plugged in. Is it plugged into the network. Are there any restrictions on the network, or on the machine running the app, that might prevent access to non-standard ports?

Over the next few days, while the stock of old printers kept dwindling, this escalated up to sending a router with a known configuration out to the technicians. It was just to ensure that there were no hidden firewalls or network policies preventing access to the TCP port. Even still, on its own dedicated network, nothing worked.

"Okay, let's double check the printer's network config," Anabel said on the call. "When it boots up, it should print out its network config- IP, subnet, gateway, DHCP config, all of that. What does that say?"

The tech replied, "Oh. We don't have paper in it. One sec." While rooting around in the box, they added, "We don't normally bother. It's just one more thing to unbox before putting it right back in the box."

The printer booted up, faithfully printed out its network config, which was what everyone expected. "Okay, I guess… try running the tool again?" Anabel suggested.

And it worked.

Anabel turned to one of the test printers she had been using, and pulled out the roll of receipt paper. She ran the configuration tool… and it failed.

The TCP service only worked when there was paper in the printer. Anabel reported it as a bug to the printer vendor, but if and when that gets fixed is anyone's guess. The techs didn't want to have to fully unbox the printers, including the paper, for every install, but that was an easy fix: with each shipment of printers Anabel's company just started shipping a few packs of receipt paper for the techs. They can just crack one open and use it to configure a bunch of printers before it runs out.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!