Renee’s company was a Silicon Valley managed services company. They needed the best data-center they could find to house their infrastructure, and cost was not an object. They loaded up a dump truck full of money and went shopping for the most expensive hosting they could find.

For Renee, this meant lots of data-center tours. She played the tame geek for her management, ostensibly there to inspect the IT services. Her bosses were just there to see how much their prospective vendors would butter them up to close the deal.

The front-runner was a company called “Isla Nublar”. Their first experience on the tour was a massive security gate, guarded by two armed men and a dog. They gave the entire car a quick search, including using mirrors to inspect the undercarriage. After being sniffed over by the dog and having their ID checked and re-checked, Renee and her bosses were allowed through.

The building itself came from the “an alien spaceship has landed” school of architecture. It was steel, glass, organic curves and confusing arcs. A security guard and a sales engineer named John greeted them at the door. “We’re so excited to have you here,” John said. “As you’ll see, we’ve spared no expense.”

He showed them into a glass-walled atrium, full of natural light, tropical plants, and a catered spread laid out on a table, complete with coffee, macarons, and a sushi chef making rolls to order. “Can I interest you in any refreshments? The coffee is Kona- we’ve spared no expense.”

John showed them down the hall to the NOC. Behind the glass wall, staff bustled about the room, mostly ignoring the Mission Control-style wall display, and focusing instead on their own screens. Everyone looked busy, but didn’t actually seem to be doing anything. It reminded Renee of animatronic figures in Disneyland’s old “Mission to Mars” ride, and they seemed about as useful.

“As you can see, we’ve spared no expense. Now, let me get you ready to see the data-center floor.”

“Getting ready” meant filling out piles of paperwork and NDAs, collecting everyone’s fingerprints, mugshots, and patting down everyone to ensure they weren’t smuggling a cellphone camera into the data-center.

The entry to the data-center floor was through a man-trap, the security version of an air-lock. One at a time, they passed through the outer door, and an armed security guard waited for it to seal before pushing the button that released the inner door. After everyone cycled through, John reminded them, “We’ve spared no expense.”

Over the noise from the racks, John shouted out the key features. The floor was raised, and each tile had a dedicated sensor to detect if it were moved. There were motion sensors, water sensors, and a few times, John mentioned “laser beams”. Every millimeter of the space was watched by cameras, and the room itself was a Faraday cage. The supports were earthquake proof and the walls were lined with Kevlar. They had triply redundant backup generators, and each rack had dual, independent power legs, in case anything went wrong with one of them. Each rack was secured with a grille that could only be opened with an admin’s ID badge.

“As you can see,” John began.

“You’ve spared no expense,” Renee finished. “I get it.”

Duly impressed, management signed the contracts the next day. That weekend, Renee and her team had to move their hardware in through the “secure loading dock”, which was just a regular loading dock with a sign that said, “Secure Area.” None of the extremely paranoid security they had seen coming in the front way was in sight here. No one complained, because it certainly made moving in easier.

By Saturday night, everything was up and running. The switchover went without a hitch, and the customers hadn’t noticed a thing.

By Sunday morning, disaster happened. Machines were down, network was down, and the customers were enraged. Renee was on the phone with the data-center support in an instant, and before five minutes had passed, John joined the call, trying to smooth things over. “I just want you to know, we’ve got no other outages. We’ve got the best electricians in the business looking at what appears to be a power outage in one of your racks. We’ve spared no expense.”

The first challenge for the electricians was simply getting permission to lift floor tiles without setting off every alarm in the building. They traced the power lines, but found no fault between the mains and the rack, which meant the problem must be in the rack itself.

“They’re trying to trace the fault in the rack, but they can’t access the power cabling,” John explained.

“What?”

“Well, we spared no expense. Our racks are custom built. The power lines run inside the frame of the cabinet. I’m getting Ray, one of the admins to join the call from the floor.”

“This is tricky stuff,” Ray said, “and I don’t know how quickly we can fix this.”

“We need our servers up now,” Renee said.

“Right, so here’s what I can do- I can move your servers to one of our empty cabinets. Sound good?”

“Whatever it takes.”

“We have spare cabinets,” John said, “because we spared-”

“Just get our servers back up.”

Ray stayed on the line while he worked. Renee heard him slide a server out of the cabinet and then say, “Hunh, that’s interesting.”

“That’s not something I want to hear when you’re moving my servers.”

“There’s a strange black switch down here. I’ve not seen a black switch in any of the other cabinets.”

Ray checked one of the other cabinets, and sure enough, it had a bright glowing red switch. “Hold onto your butts…” Ray said. He flicked the black switch in Renee’s failing cabinet. The light turned red, and the servers spun right back up.

The switch was inconveniently placed, near the floor, where someone passing by the cabinet could accidentally jostle it, which is likely what happened. Unfortunately, moving the switch wasn’t possible. With their bizarre internal power cabling, and the way the switch was soldered to the internal power strip, it would require cutting, drilling, and probably a bit of welding to move it.

The only practical fix was to use tape to secure the switch in the “on” position. “We’ll use the best tape money can buy,” John said, “and it won’t come off. We’ll spare no expense!”

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!