• P (unregistered)

    Ahh, good old sitcom drama for the day.

  • (nodebb)

    That guy should've taped "NOT" right under the "DECOMMISSIONED" sign on the server.

  • (nodebb)

    In response to "Some of the people who work here are idiots" I would have looked hard and pointedly at him, saying, "Yes, they are."

  • Factory Improvement (unregistered) in reply to Steve_The_Cynic

    cue seinfeld bass theme

  • bvs23bkv33 (unregistered)

    three boolean values are "DECOMISSIONED", "DECOMISSIONED NOT" and "STOLEN"

  • Dave Hemming (unregistered)

    So they effectively performed a scream test on it.

  • (nodebb)

    So, how did someone manage to create a drive mapping that pointed to the physical box.. and somehow the mapping knew that the virtualized version wasn't the same box? Was Edward reckless enough to give the virtualized server a different name and then confident enough that he updated every single reference to it. This seems to be more Edward's wtf than the other guy's.

  • Craig (unregistered) in reply to Jaime

    I don't think it was a drive mapping to a physical box; that would require all sorts of static ARP wizardry to even set up. It would have been a drive mapping to a share on a hostname. The new virtualized host (presumably using the same hostname) didn't have that share, causing an error.

    That would have been all that was needed to have this rogue user go to investigate and power-on the decommissioned physical box. The user likely wouldn't have understood the difference between the various error messages ("can't connect to host" versus "share doesn't exist"). I bet that - through habit - their SOP for any kind of error was "make sure the physical box is turned on".

  • (nodebb)

    If the machine name, IP or Shares or Permissions were different then the CLONE was a failure.

  • Foo AKA Fooo (unregistered) in reply to Craig

    Of course, this and "just restart it" are SOP for trouble-shooting to any non-technical person. They've all been trained this way for decades. (In fact, it's even to me with some kinds of devices ...)

    Leaving the IP collision just a switch turn away is just reckless. They could've at least physically covered the switch, power supply, network port or anything or changed the software enough (while easy to undo if really needed) to avoid booting with the conflicting address or any number of other protective measures, especially after the first incident. Like, "unplug the server and hide it behind a cardboard box full of power cables", but why did it take them 2 months to get this brillant idea?

    "Half the servers in the datacenter have had stuff like that on them for years!" Indeed, that's the getting-used-to effect you breed when you leave things like that for too long. Though I wonder, if they so urgently needed the space, didn't they need to remove the decommissioned machines anyway?

    PS: "DC room #2, third gray desktop on the floor from the left [...] But, wouldn't you know it? There was a problem with "DC room #2, third gray desktop on the floor from the left." " Lamest Chekhov's gun I've ever seen, sorry.

  • Pabz (unregistered) in reply to Craig

    Indeed, the article states that Edward's cloning script had missed creating that share. To be honest though, I've just tried accessing a share that doesn't exist on a computer that does vs. accessing a share on a computer that doesn't exist and Windows gives me the exact same error message.

  • DontTouchThis (unregistered) in reply to Foo AKA Fooo

    If you remove a device then addresses like "third gray desktop on the floor from the left" become invalid.

  • DQ (unregistered)

    The real problem is that now the "fourth gray desktop on the floor from the left" became the third when that was hidden away. So now the entire floorplan is wrong...

  • Bill P. Godfrey (unregistered)

    Up once again with the ill behavior.

  • 🤷 (unregistered) in reply to DQ

    I see a few more WTF stories coming in. From Edward, who now can't find the machine he was looking for, because some idiot moved away a gray box and hid it behind some cardboard boxes.

  • sizer99 (google)

    | Over 100 people had unrestricted access to the datacenter.

    This is TRWTF.

  • (nodebb)

    They should migrate to the cloud. Now Edward would be fighting his way into an Amazon datacenter, trying to save DC room #2, third gray desktop on the floor from the left, and they would make a new Rambo movie out of it.

  • Prime Mover (unregistered)

    Should have gone like this:

    The beginning of the next month, there was no IP collision. However, there was a frantic call from one of the 24/7 datacenter monitoring personnel, claiming that one of the servers had been stolen. Edward ignored it, and continued to ignore it till the frantic call turned into a frantic visit.

    "Yes, we decommissioned it," said Edward. "It's gone."

    "But you should have got my telephone call!"

    "I routinely ignore calls from people who routinely ignore instructions via email."

  • Mover (unregistered)

    Why do my comments routinely get sent to moderation limbo when I post them as by "Prime Mover"?

  • Jim Did It (unregistered)

    I used to do data center migrations, where we moved client equipment from their environments to our locations. We spent over half our time (and budget) trying to properly identify and categorize the purpose of each physical box and deciding if it could be virtualized without moving it. It was always a challenge to identify what app or share or database was associated with them. In some cases we practiced power analysis - if nobody could tell us what the box was for we powered it off and waited for a phone call or e-mail.

    We did have to change our strategy to practice air gap analysis instead of power analysis - seems that some boxes were so old and had been on for so long that they sometimes wouldn't turn on after a shutdown, so we started just disconnecting the network cable instead.

  • ray10k (unregistered) in reply to Jaime

    Maybe it was something ridiculous like the share mapping to a USB pendrive plugged into an internal USB connector?

  • Neveranull (unregistered)

    “On the left” becomes “on the right” if you face the other way.

  • anonymous (unregistered)

    So then the manager hears about the "problem" with the mapped drive, decides that this counts as "an issue with the virtual environment", and insists that they have to go back to using physical "servers" for everything?

Leave a comment on “The Renegade Datacenter”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article