Everything was lined up for some seasonal specials at Chotchkie's (as we'll call it), a mid-size chain of family restaurants. Starting just in time for the weekend on Friday, there would be specials on Pizza Shooters, Shrimp Poppers, and Extreme Fajitas.

The special had been promoted, ads circulated, table cards and menus updated — everything was ready to go. That is, aside from setting up the discount in the PoS system on their antiquated Panasonic 7000-series cash registers. Because of the old architecture and a confusing interface, what should be a simple change for restaurant staff to update item pricing actually required the involvement of a developer; in this case, Mike L.

The circumstances had worked well for Mike. Chotchkie's had been running their PoS system for a long time — well past the point that support had ended. Chotchkie's Corporate had to figure something out, and came up with two options:

  1. Upgrade the PoS system for every Chotchkie's (at a cost of ~$28,000 per store, over three million for the whole chain), or
  2. Figure something else out.

They chose #2 and hired Mike, and things had been going smoothly for months.

Two days before the special started, Mike received a panicked call from the client, begging him to rush a change. "We've already begun advertising this," the client pleaded. "I know today's Wednesday and usually we have more time, but we really need this in by Friday morning!" Since they were in such a bind and Mike didn't have much of a choice, he ignored his conscience and agreed to a "rush job." This was his first mistake.

While preparing the changes, Mike realized that he had a narrow window to roll the change out to the 120 restaurants. In order to get the change in before Friday morning, he would have to bypass the usual client testing phase. Mike worked late on Thursday and dispatched the change to 120 Chotchkie's restaurants. This was his second mistake.

After a good night's sleep he returned to work on Friday. At 10:05 that morning, his phone started ringing. Constantly. The restaurants had been open for five minutes.

Everything at every Chotchkie's was ringing up for ten cents. Super Fries, Belly Busters, Onion Hoppers, Marshmallow Hoorays, even $50 gift cards were $.10 each.

Mike realized he had to do something drastic; something he hoped he'd never have to do. He shut down every register in all 120 Chotchkie's locations, leaving hundreds of employees with nothing to do, and turning away thousands of hungry customers.

This was not a popular decision, and only increased the number of calls he was receiving. Worse, clients had begun asking to remain on hold until the issue was fixed.

It didn't take him too long to find the source of the problem. Price changes were done with updates to four dBase III tables on each register, and five tables on one central computer for each restaurant. In an attempt to make his life easier, Mike wrote an app to streamline the process. It'd build these nine tables into a common table, then send the updates down to each register.

To deal with the quirkiness of the old Panasonic registers' database structures, Mike's app had to massage the data going to and from his own tables using some fairly painful string parsing and manipulation code. Though the price changes he needed to make were fairly straightforward, a missing leading zero on one of the items (.10 instead of 0.10) lead towards disasterous consequences: the register update loop got "stuck" on the ".10", causing all prices to be updated to $0.10.

Mike hacked like he'd never hacked before and had a fix ready before too long. Because the registers had already been opened for the day and the systems were designed to only accept automatic updates when the registers were closed, he couldn't do one big push. Four tedious hours of updating restaurants one-by-one and a $7,000 discount on the invoice later, he was out of the woods. It may sound bad, but he got off easy considering all the free food that was given to people that showed up right when the restaurants opened, the cost to have the staff sitting around doing nothing all morning, missing the lunch rush...

Needless to say, he's learned not to ignore his conscience anymore.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!