|« Helpful Google!||A Long Running Process »|
At Cheryl C's company, most of the work was done, and all of the data was stored on Linux servers, but everyone used laptops to access the network and by extension, their Linux servers. Most of the time this worked out well. However, when something went wrong... support was handled by two different organizations: one in her company and one on The Other Side. Specifically, the company on the other side of the merger. Two behemoths had merged into one mammoth colossus of Jurassic proportions.
For the past 18 months, all sorts of efforts had been ongoing to divide the responsibilities to eliminate the redundancies. Managers met. Managers fought. Managers decided that it was redundant to have two PC support organizations, two network support organizations and two Linux support organizations, so they decreed that Cheryl's company would support the PC world and the other company would support the Linux world and the network. And never the twain would meet.
Rather than leave well enough alone, they continued to spew forth directives on how things would continue to change moving forward. Add rotating password requirements here. Use this software there. Follow these procedures elsewhere. All systems would run managed maintenance scripts to ensure that only blessed software was installed.
Cheryl had lugged her laptop through the frigid weather to get to work early. She put it in the dock, hit the power button, and began to sip her coffee while the machine booted. After a couple of sips, the machine appeared to freeze up. After cursing under her breath, she rebooted and waited again. After several more sips, the laptop appeared to freeze up at the same point.
She was about to call PC support when she heard someone else muttering about their laptop freezing up. Then someone else. Soon, it was clear that everyone in the office was affected.
She stood up and peered around for her fellow sufferers. After swapping stories, they realized that perhaps their laptops weren't the problem, but rather, the network. Upon calling Network Support, they're told the network is fine, and that they should call PC support. Upon calling PC support, they're told that everyone is having the same problem, but "Trust us, we'll get the network guys to fix it."
Cheryl knew from past experience that getting The Other Side to do anything was like trying to move Heaven and Earth; they simply punted all complaints back to the PC world because Linux was set up properly and the network had so many redundancies that it simply couldn't go down.
After about an hour, many other folks had showed up at Cheryl's company and encountered the same problem. Managers. Senior Managers. C?? executives. But still no word from PC support.
Managers met. Managers looked for somewhere to point a finger. A battle plan was formed. The Other Side was going to be held accountable for this massive outage. Calls were made to management of The Other Side and a meeting was arranged. At the meeting, more fingers pointed and more accusations were made.
Cheryl and her peers continued to wait. And wait. And wait some more.
Finally, she decided to go wait in the bar across the street. There, she encountered a PC-SA who had no clue why this was happening. Then he mentioned something in passing that made something click in her mind.
Managed maintenance software was recently installed and running on both Linux and PC systems to scan for software that was not blessed and should not be there, and delete it. Also, if something was missing, it would auto-install it for you.
After the initial pass of uninstalling everything folks had installed on their laptops (e.g.: SQL Developer in lieu of TOAD, Firefox in lieu of IE, etc), people got used to working with the blessed software. The problem was that in the PC world, you were supposed to use IE, TOAD and MS Office, and in the Linux world, you were supposed to use Firefox, SQL Developer and Open Office. Although it was annoying to have to switch back and forth between the different tools, people had acclimated to it.
But... recently, there was a decree that the Linux drives would be Samba-mounted on the laptops so that some folks could leverage MS-Office with data from the Linux boxes for certain types of documents. This was not to be done just for the folks who needed it. For uniformity and ease of support, everyone would get the same configuration. It had just gone live.
The managed maintenance software that was launched at PC-boot time would see the Samba-mount and start scanning for unapproved software. It would find Firefox, SQL Developer, Open Office, etc and immediately begin deleting them. At the same time, the Linux managed-maintenance scanning tool would detect the blessed packages disappearing and begin auto-reinstalling them.
Both sets of scanners didn't just go down a checklist once. Instead, they would go through the checklist, and if they detected something wrong, would fix it and restart at the top.
The laptops weren't hung; the boot processes simply never completed because the managed maintenance software was continuously battling it out with the Linux managed maintenance software to un/install software that was blessed in one world and prohibited in the other. Since the laptops weren't hitting their own disks, the only indications that they were doing anything were the network lights blinking away, and they were supposed to be on while booting.
Once it was diagnosed, verified and pushed up the hierarchy, a directive was immediately spewed forth to stop the managed maintenance software on both types of systems until formal policies of what was allowed to live where could be formulated. Then everyone was told to reboot ...just in time to go home.
|« Helpful Google!||A Long Running Process »|