David was recently hired on to head the company’s development team. This was a brand-new position; previously, William, the company’s IT Manager managed the developers directly in addition to his other duties.

While getting his workstation set up, he was unable to install the FileZilla FTP client. It was completely blocked via domain policy. Finding this very strange, David talked to the IT Manager and hoped there was a legitimate reason.

A diagram of RAID10 design

“FTP is a big security risk,” William explained. “We got hacked through FTP once so I firewalled it by blocking FileZilla installation with Group Policy.” David began to suspect that things at the company were not quite right.

“Um, Group Policy isn’t a firewall,” David tried to explain. “And blocking an FTP client installer is unlikely to have an effect on network security.”

William clearly didn’t understand and doubled down on his incorrect ideas. “Trust me, it works,” he said. David soon gave up and simply installed a different FTP client–which worked just fine despite FTP being “firewalled.”

Later, while familiarizing himself with the business’s core applications, David decided to examine the company’s main database server. “It’s set up with RAID 10 for the best performance,” William explained. RAID 10, sometimes called RAID 1+0, mirrors all data into two arrays for fault tolerance, and then each array is striped onto additional disks to greatly improve read/write performance. It’s one of the fastest RAID configurations and a great fit for database servers.

But David ran a disk benchmark and was not convinced. “That disk I/O is way too low for RAID 10,” he explained. “Can you show me the configuration?”

“Of course,” William replied. “It has to be RAID 10. We ordered it that way.”

They opened the RAID manager for the system, and sure enough the system was not running RAID 10. “Um, this is RAID 5,” David explained. RAID 5 uses three or more disks for striping, but the capacity of one disk is lost for storing parity data which allows the array’s data to survive a single-disk failure. Due to the required parity calculations, RAID 5 writes are not nearly as fast as RAID 10, which is bad for an I/O bound application such as the company’s main database server.

“That can’t be,” William responded. “Oh, I know…Kieran must have changed it!” He called the company’s so-called systems administrator to the room, and David endured the uncomfortable exchange as William hurled wild accusations. “I know we ordered this system with RAID 10! Why did you change it?”

“I just unboxed it and racked it,” Kieran responded. “Whatever RAID level it has is what it was shipped with.”

“That can’t be true! You tampered with it!”

Kieran rolled his eyes and turned to leave. “I have real work to do elsewhere,” he rudely explained as he excused himself from the conversation. Later on, he sent an email to both David and William which contained the original Purchase Order for the server. The PO clearly showed that William had specifically requested RAID 5 for the server.

David found himself wondering if management was aware of William’s behavior and general incompetence, but tried to avoid making waves during training. But the final straw came when he set about installing several of the company’s internal applications and was unable to find any installers on his own. Once again he went to talk to William.

“Oh, we don’t install any of our applications. That’s too slow!” he explained. “Instead, look at the database server. It has a shared folder called ‘APPS’, and just map that as a network drive and run everything from there.”

Once again, David found himself doubting the IT Manager’s wisdom. “Doesn’t it hurt our database performance to have the entire company mapping a network drive to the same RAID array as the databases?” he asked.

“Oh no,” he replied. “It actually makes it faster. The apps can talk to the database much faster if they’re running from the same server!”

David was stunned by William’s poor understanding of how network-mapped file systems actually work. “That might be true if it was a terminal server…but the applications still run locally on the end user PCs. The files just happen to be stored elsewhere.”

“What do you know, you’re still the new guy! We’ve had this procedure in place for years now, and for good reason!”

Despite being a newbie, David decided to approach management about the ineptness he’d discovered, and explained that William was rude, incompetent, and had no business working in IT. To his surprise, management conducted a brief investigation and agreed with him! Shortly afterwards, William was let go.

The new IT manager was shocked at the company’s infrastructure and quickly made a range of improvements. This included installing a proper Storage Area Network which greatly improved the database server’s performance; a new storage pool for mapped drives which was on separate disks than databases; and a real hardware firewall which, among other things, actually blocked incoming FTP connections from the Internet.

David then forwarded his story on to us at The Daily WTF as a reminder that sometimes things actually do turn out okay in the vast WTF-land that is the IT industry.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!