Bunker Buster was originally published on Febuary 19, 2007.

“Hope I’m not waking ya up, but we need a huge favor.” It was just past dawn on a Saturday morning, and Jack’s new boss was on the line. “We need you to replace some servers for the portal system.”

Earlier that week, Jack started working as a subcontractor to maintain and “fix up” a certain government agency’s web portal that was used by thousands of branch offices around the country. Jack’s boss, a tech-savvy businessman who had mostly setup the web portal by himself, was training users and demonstrating the product, and noticed they had a serious problem: their system was running ridiculously slow.

“Okay,” Jack answered still half-asleep, “where should I pick up the servers? Are they at the office?”

Up until this point, Jack had never been to the hosting facility nor had he seen the equipment they were using. It was seven-figure government contract, so he figured they had all the normal stuff: load balancer, redundant power, failover servers, and all that other stuff.

“Well,” the boss replied, “I was hoping you’d be able to head down to CompUSA and build us one.”

Jack wasn’t quite sure how to respond. Sure, he’s pieced together a few computers in his day, but how exactly does one build a server out of consumer-grade components? Before he could even reply, Jack’s boss said “I’ll pay you double.”

Not to be one to turn down money waved in his face, Jack agreed and headed down to CompUSA to build the best computer money could buy. And as it turned out, that was the Ultra Super Extreme 7000XL Deluxe Gamer setup minus the gaming video card. It had a four-disk RAID 0+1 array with screaming SATA drives, plenty of RAM, and lots of blinking neon lights, all which were visible through the cutout biohazard symbol on the side of the case.

After configuring the server, Jack called up his boss and told him that he was ready to take it to the hosting facility. All he needed was instructions on how to get there.

“The facility is about forty minutes from the office,” Jack’s boss replied, “it’s in the middle of a forest outside Conroe. I’ll call them up and have someone meet you at a gas station off of I45.”

The gas station meetup wasn’t arranged for dramatic effect or to protect the location of the facility. The hosting facility was truly in the middle of nowhere. It was only reachable by unpaved access roads, and none of those were on any maps.

Shortly after dusk, Jack arrived at the gas station and was greeted by a fellow wearing dark sunglasses and a black suit. Standing in front of black Chevy Suburban with tinted windows, he introduced himself simply as, “The President.” No, it couldn’t get any more cliché than that.

The President took a look at Jack’s car and said “we better take mine.” They loaded the server up and headed down an unlit county road to the hosting facility. A few miles down, The President turned down an unmarked access road and proceeded to drive for another few minutes through the dense forest. And then all of a sudden, they were in a small, open grass field with two small, concrete buildings and a three-car parking lot.

As they got out of the car, Jack noticed that one of the buildings had metal plates over the windows with small, rectangular holes cut out of the center. They were machine gun ports.

Jack and The President headed to the other building. Inside, it was no more than a small room (about the size of a one car garage) with an elevator at one end. The President walked towards the elevator and placed his palm on the scan pad. The unmistakable creak of a climbing elevator filled the room. Jack couldn’t resist, “what, no retinal scan?”

The President smirked. As they waited for the elevator, he told Jack about the facility. It was a decommissioned military bunker with a 40,000-square foot, two-story underground facility. They had air scrubbers, generators, food, water -- enough supplies to last completely cut off from the world for at least two months. They even had decontamination showers and holding cells. Yes, holding cells.

They descended down into the bowels of the facility and headed through a half-renovated hallway towards a small room that Jack’s boss was renting. It was mostly empty and had cables and miscellaneous parts strewn across the floor. “Let me know if you need anything,” The President said, leaving Jack and the new server alone in the room.

In the corner of the room was the portal system’s rack. It was about five feet tall and had a second-hand monitor sitting on a mounted 1U-hub. Below the hub sat a shelf filled with five or six consumer-grade routers all patched together. Below the routers sat a Compaq Presario desktop computer. It was jammed in and sat at an angle, as it couldn’t fit horizontally or vertically in the rack. Below the desktop were the two portal servers, one to run the database and one to run the web application.

The portal servers were a generic brand of Shuttle PCs: little white cube computers that have room for a CDROM, floppy drive, and maybe a single PCI card. They’re generally what you’d put in your living room as a media PC or what you’d buy for Grandma to check her email. They’re not generally what you’d expect to find in a datacenter running a large government portal.

It took Jack a little while to figure out exactly how to plug in the new server he built. From what he gathered, the hosting facility’s feed went directly into the hub, and each router was plugged in to one of the hub’s port. The servers were then plugged into the routers.

Whomever set up the servers must have had a hard time figuring out how to get the web server open to the “outside” while still being able to talk to the database server on the “inside.” So, he simply assigned both of them a routable IP address and exposed them to the “outside.” It didn’t help that all of the administrator passwords were kept as their default.

No matter, Jack was able to get it working and, after leaving the hosting facility, immediately called up his boss to explain the lax security situation. His boss had a rather odd explanation:

I know the security isn’t perfect, but this is an area we need to tread lightly on. The guy we’ve hired to maintain our servers has some mental issues and doesn’t respond well to criticism or working on Saturdays. I’d rather not upset him now, but I promise, we’ll get there soon.

Three months later, the servers are still completely unsecured and open to the Internet. One of them is currently participating as a zombie in a bot-net. And they’re still housed in one of the world’s most secure underground datacenters.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!