The Daily WTF: Curious Perversions in Information Technology

MoSlo · 2014-01-06 Reply Admin

"randomized server backup system"??? In what world would this EVER be a sane idea?

2014-01-06 Reply Admin

This is the moment SLAs were made for. But why didn't he immediately demand to have his site transferred to another server? He needs the capacity, and "sorry, you're boned" is an answer that usually leads to unpaid invoices, lawsuits, and lack of future business.

2014-01-06 Reply Admin

I don't think it's a big deal, but there's an error here:

Peter was cautiously optimistic before the busiest part of open class enrollment season at his employer

it should be like this:

Peter was cautiously optimistic before the busiest part of open class enrollment season at his employer :)

2014-01-06 Reply Admin

No SLA on a hosted system that was pivotal to the service delivery? Whoever came up with that decision needs to be sacked.

Paddles · 2014-01-06 Reply Admin

Only considered firing himself? As big a WTF as the load balancer and backup configuration is, as huge a WTF as not having any sort of SLA is, not even attempting to negotiate some sort of even marginally better outcome makes Peter TRWTF.

2014-01-06 Reply Admin

the article said:
[Peter] even contracted a third-party datacenter with “state-of-the-art load balancing hardware” to handle the web traffic stress on the system.

So it was Peter who contracted the datacenter. One would imagine that the question as to what load-balancing strategy is used would have been raised during the contracting phase. But ok, sometimes one professional just trusts another to do the right thing.

But what's completely bizarre ist the "2nd server used as backup server" thing.

If the mentioned servers are Peter's servers (i.e. NEIoWTF owns them), then a simple "you stop that backup immediately or you will hear from our legal department and the FBI" should do the trick.

If the servers are rented, then a SLA will exist. If that does allow the datacenter to install its own software and run it on a mission critical piece of hardware, then Peter should be fired immediately. Apart from the availability issue (a lot of people fail to recognize that even 5 nines still mean a permitted downtime of > 8h), there are severe privacy issues here, since the server is processing student data.

I don't know - a misconfigured load balancer balancing the traffic between just 1 machine would be ok and a nice wtf.

But the datacenter misappropriating a server as a backup server? Sounds too weird to be true.

skotl · 2014-01-06 Reply Admin

I cry shenanigans. So every server has 18TB spare disk space in case it is the chosen to be this week's random backup machine? Nahhh....

Leak · 2014-01-06 Reply Admin

skotl:
I cry shenanigans. So every server has 18TB spare disk space in case *it* is the chosen to be this week's random backup machine? Nahhh....

That's probably what network attached shortcake was made for...

2014-01-06 Reply Admin

Agreed, his response should have been, "You are not providing the service for which we contracted. The reasons for your failure are immaterial. You will provide that service now or you will see us in court."

2014-01-06 Reply Admin

Wait! The story continues... <<Peter called again to Roger in order to re-establish the service, and some other claims the dean told him to yell to Roger. After some arguing, Roger settled to take a look on what to do and be called back some time later. Roger and his staff got overwhelmed, so they flipped the switch on the phone system, activating the “We’re sorry, all lines are busy right now” prompt to stop the calls. “Now we must continue playing the Gears of Duty game to finish the match!”, Roger said to his guys as they have started up a game in that server and, of course, cannot be restarted in any way. All while the phones were still ignoring Peter’s calls. The sad part is that, accidentally, they were ignoring the President's daughter calls as well.>>

Steve The Cynic · 2014-01-06 Reply Admin

faoileag:
If the servers are rented, then a SLA will exist. If that does allow the datacenter to install its own software and run it on a mission critical piece of hardware, then Peter should be fired immediately. Apart from the availability issue (a lot of people fail to recognize that even 5 nines still mean a permitted downtime of > 8h), there are severe privacy issues here, since the server is processing student data.

http://en.wikipedia.org/wiki/High_availability says that three nines is 8.76 hours per year. Five nines is 5.26 minutes per year. A single 8-hour outage consumes your five-nines outage allowance for over 90 years.

2014-01-06 Reply Admin

Steve The Cynic:
faoileag:
that even 5 nines still mean a permitted downtime of > 8h
http://en.wikipedia.org/wiki/High_availability says that three nines is 8.76 hours per year. Five nines is 5.26 minutes per year. A single 8-hour outage consumes your five-nines outage allowance for over 90 years.

You are right, of course. And I agree completely with you. And with whoever said that you shouldn't do complex math when suffering from a cold ;-)

Quango · 2014-01-06 Reply Admin

Kudos for Peter in accepting that it was ultimately his fault: most people would blame the datacenter.

2014-01-06 Reply Admin

ZoomST:
Wait! The story continues...

Here's my try... "Ok, I've fed that guy from NEIoWTF some story about backups that should keep him off the phone for a couple of hours," Roger thought to himself. "Now to finding out out what really happened to their server."

It took him the best of five minutes to connect to the server, whith all the misspelled passwords and that. That server was a very busy server right now!

Finally he managed to connect, become root and do a ps. One process of which multiple instances seemed to be running caught his eye: jasmine. jasmine? He had never heard of it before. A quicl google search yielded nothing. Oh, it did yield a lot, but those results usually had something to do with people, not with processes.

Roger killed the processes and started a find. Hidden several layers deep under /etc/, "jasmine" turned out to be a bitcoin miner written in javascript! "The wonders of the world..." he thought.

But who had planted it? Examining the path to "jasmine" he found out that the final directory had been created by the user "forsetisdottir". WT...? He cast a stern look at the newly hired intern, who grew pale immediately. "On the count of three, you are out of here!!!". She was gone when he reached "Two!!".

The President's daughter herself! What a sad state the world had come to! He sighed and shook his head.

Still, the customer had believed the crap anout the backup. So why waste valuable cpu cycles? He fired up his own bitcoin miner, told everybody that nobody could be trusted anymore and marched off to have his lunch in the bar opposite his office.

2014-01-06 Reply Admin

Why would someone name a bitcoin miner written in javascript after a popular test framework for javascript? Or wait.. is THAT why it's free?

2014-01-06 Reply Admin

MoSlo:
"randomized server backup system"??? In what world would this EVER be a sane idea?

A hundred times this.

They might as well also have a randomised punch in the face system, which selects one customer per week and dispatches an employee to pummel their heads.

It would be exactly as convenient to the customer.

2014-01-06 Reply Admin

Nice XKCD reference thrown in there.

2014-01-06 Reply Admin

Wait, it said it was random, right? Which would mean only about half the students would fail to get in? Or I guess it's random per request, not per session?

Any case, it reminds of a huge WTF I'd forgotten about, because they did eventually fix it a couple years ago, it just took them forever - my favorite silly web game, Kingdom of Loathing, the guys who created it fully admit was created by a couple guys who thought they were creating it to fiddle around with PHP, they had no idea they were creating something that would turn into a fairly successful game. As such, a lot of the early stuff the game was built on was pretty wonky. Among which, they did have a load-balancer for some stuff, which automatically directed people to one of 8 frontend servers based entirely on which one had the fewest number of people logged in, which is a great idea in theory, but you can probably see where this is going...

Yep, periodically, one of the 8 servers would crash and boot everyone off. The load-balancer would look and say, hey! Nobody is on server 5 right now! Better start sending everyone there!

2014-01-06 Reply Admin

anon:
Why would someone name a bitcoin miner written in javascript after a popular test framework for javascript?

Because the author carried out only the most perfunctory research when trying to come up with a snappy name for a cycle-munching abomination :-)

da Doctah · 2014-01-06 Reply Admin

ZoomST:
Wait! The story continues... <<Peter called again to Roger in order to re-establish the service, and some other claims the dean told him to yell to Roger. After some arguing, Roger settled to take a look on what to do and be called back some time later. Roger and his staff got overwhelmed, so they flipped the switch on the phone system, activating the “We’re sorry, all lines are busy right now” prompt to stop the calls. “Now we must continue playing the Gears of Duty game to finish the match!”, Roger said to his guys as they have started up a game in that server and, of course, cannot be restarted in any way. All while the phones were still ignoring Peter’s calls. The sad part is that, accidentally, they were ignoring the President's daughter calls as well.>>

Somewhere in the darkness, a dog howled....

2014-01-06 Reply Admin

neminem:
Wait, it said it was random, right? Which would mean only about *half* the students would fail to get in? Or I guess it's random per request, not per session?

That's how I read it, and if we're talking about a "modern frontend" here there's only one sure thing. There will be multiple requests.

2014-01-06 Reply Admin

neminem:
Wait, it said it was random, right? Which would mean only about *half* the students would fail to get in? Or I guess it's random per request, not per session?

No, I think they mean random as in "We roll a couple of dice to determine the server we'll shanghai into becoming our backup server for the week".

What I wonder is: did NEIoWTF indeed only have two servers at the datacenter? So their database server is hosted elsewhere and the servers at the datacenter are set up solely as front-end servers for the enrollment process? Does this make sense?

2014-01-06 Reply Admin

Huzah!

This WTF is WTF-y. That's a first for months!

2014-01-06 Reply Admin

There where myriad connection errors.

There where also myriad spelling errors.

no laughing matter · 2014-01-06 Reply Admin

Elsewhere Peter said: "So half of the requests go to the busy backup server? Why don't we just change the settings to route all requests to the server who is not performing backup duties? It will be a bit slower, but still much better than what we have now!"

steenbergh · 2014-01-06 Reply Admin

faoileag:
You are right, of course. And I agree completely with you. And with whoever said that you shouldn't do complex math when suffering from a cold ;-)

It's "Don't drink and derive". A cold has nothing to do with math.

2014-01-06 Reply Admin

"ok, then - remove the loaded server from the round robin and start up 5 other servers in its place, oh and FYI - you are paying for them" should have been the right answer.

2014-01-06 Reply Admin

This story is BS.
In my university experience everyone registered for classes as soon as possible so they did not miss out on those required classes that fill up. That was the only think college students would not procrastinate on.

2014-01-06 Reply Admin

Met:
This story is BS. In my university experience everyone registered for classes as soon as possible so they did not miss out on those required classes that fill up. That was the only think college students would not procrastinate on.

This. At least, the site where you had to log in to register wouldn't work properly. So that part of the story isn't BS.

2014-01-06 Reply Admin

faoileag:
... newly hired intern, who grew pale immediately. ... The President's daughter herself!

I thought the President's daughter was ill. Wouldn't she already be pale? Plot hole!

2014-01-06 Reply Admin

For TRWTF - why didn't he do any of about 50 other things instead of just tell everyone to wait for the backup to complete? Tell the hosting company to kill the backup, or route requests to the good server, or spin up a new server with a hosting company that isn't insane, or....?

2014-01-06 Reply Admin

Insensitive Claude:
faoileag:
... newly hired intern, who grew pale immediately. ... The President's daughter herself!
I thought the President's daughter was ill. Wouldn't she already be pale? Plot hole!

Nope. She just turned "a whiter shade of pale".

2014-01-06 Reply Admin

Insensitive Claude:
I thought the President's daughter was ill. Wouldn't she already be pale? Plot hole!

Maybe this is the prequel, explaining how she got sick in the first place....

chubertdev · 2014-01-06 Reply Admin

faoileag:
Steve The Cynic:
faoileag:
that even 5 nines still mean a permitted downtime of > 8h
http://en.wikipedia.org/wiki/High_availability says that three nines is 8.76 hours per year. Five nines is 5.26 minutes per year. A single 8-hour outage consumes your five-nines outage allowance for over 90 years.
You are right, of course. And I agree completely with you. And with whoever said that you shouldn't do complex math when suffering from a cold ;-)

Off-by-one-nine errors are more common than you think.

2014-01-06 Reply Admin

Met:
This story is BS. In my university experience everyone registered for classes as soon as possible so they did not miss out on those required classes that fill up. That was the only think college students would not procrastinate on.

Indeed. In addition to the required classes thing, if you didn't get in early, you were also faced with only the unpopular sections being available - things like 7AM labs or back-to-back classes that were held on opposite ends of the campus.

DCRoss · 2014-01-06 Reply Admin

Peter took on the task of crafting an e-mail to the dean of Northeastern Institute of WTF with about 25 apologies and a request that the class enrollment period be extended a few days.

To: The Office of the Dean, NEIoWTF From: Peter Son of Peter, Holder Of The Holy Relics Of University IT. Subject: I'm sorry (1).

Martin,

I'm sorry that I blew up the registration system (2). At the time it seemed like a good idea to go with a hosting company run by a family member even if he did have a room temperature IQ and approximately three weeks of experience in the field. I'm sorry I did that (3). I'm sorry that we never discussed service level agreements beyond the level of "Don't worry, you'll be fine!" (4). I am also sorry that all negotiations were conducted in the VIP room at the Brass Rail (5), and I am still very sorry that we forgot to invite you seven times in a row (6). I am sorry that our bar tab for those meetings still exceeds all other costs involved in the implementation (7) and I am very sorry that I drank all of those shots of Jägermeister before signing the contract (8). I am also sorry that I drank all of that Goldschläger while signing it (9), and I am sorry that I could not think of any other drinks which contain umlauts in their names for after the signing (10), because putting umlauts in things just makes them more awesome. Heh. Ümläüt.

Sorry about that last part (11). It was not directed at you, but at the pounding in my head which still has not gone away. Sorry, head (12). Very sorry (13). Ow. Now I am feeling very sorry that I turned the lights back on (14). I'm sorry, lights (15).

And now I am sorry I put that last paragraph in there (16). It really doesn't belong. Sorry, extra paragraph, but it just wasn't meant to be (17).

Getting back to the point (sorry for going off topic (18)), I am sorry for not having even thought about having a backup plan (19) and for letting Roger get off so easily (20).

Did I use the phrase "get off"? Sorry about that (21). I was just reminded of something that Roger said during one of out planning meetings when Gertrude --

Sorry again (22). I'll stop now. Sorry (23).

Getting back to the point, would it be possible to just extend the class enrolment period by a few days instead of trying to fix the problem with our registration server? I'm sorry that it never even occurred to me that we could do that (24), and have been considering firing myself as an example to the others.

Very very very sorry (25),

Peter

herby · 2014-01-06 Reply Admin

MWF:
Met:
This story is BS. In my university experience everyone registered for classes as soon as possible so they did not miss out on those required classes that fill up. That was the only think college students would not procrastinate on.

Indeed. In addition to the required classes thing, if you didn't get in early, you were also faced with only the unpopular sections being available - things like 7AM labs or back-to-back classes that were held on opposite ends of the campus.

Or you couldn't get the prized required PE class "bowling" which was only offered once.

My sin was taking the "advanced badminton" class because it was later in the day (1 hour) than the intermediate one. It was still early in the day (around 8AM or so) and I didn't like it at all, but it was one of those "requirements" I needed for the degree.

At least they didn't have "homework", but it was only 1/2 unit anyway. More than likely an excuse to goof around if you were lucky.

(It was the early 70's, if you want to know. Things were different then)

2014-01-06 Reply Admin

Hannes:
Met:
This story is BS. In my university experience everyone registered for classes as soon as possible so they did not miss out on those required classes that fill up. That was the only think college students would not procrastinate on.

This. At least, the site where you had to log in to register wouldn't work properly. So that part of the story isn't BS.

Thirding this.

no laughing matter · 2014-01-06 Reply Admin

DCRoss:
... and I am sorry that I could not think of any other drinks which contain umlauts in their names for after the signing (10), because putting umlauts in things just makes them more awesome. Heh. Ümläüt.

Küstennebel Andalö Eisvögel Fiete's Möwenschiß. Yep, it means seagull shit! Wattenläuper - der Küstenkräuter (!) Klütenkööm(!) Tannenzäpfle - ok that last one is a beer, but the link (unlike all the others) goes to a site in english language.

Addendum (2014-01-06 13:47): Forgot the generic Likör, which leads us to stuff like Eierlikör.

And some Absinth labels: Kübler Grüne Fee

Addendum (2014-01-06 14:26): Alter Kurfürst (english site) Ammerländer (english site) Bärwurz (unfortunately not Bärwürz) Demänovka Fläminger Sanddornlikör Kümmel, a spirit made from caraway. Leibwächter Kräuterbitter (english site) Rhöntropfen Schinkenhäger - a Steinhäger (english site) Sechsämtertropfen Urkönig a black Häger (english site) Bündner Röteli Bärenjäger is a Bärenfang!

This one obviously is from people who really love to put Umlauts in things: Gräfs Likör-Partymini SpaßPäck Gürteltasche

Dismissed: Kuemmerling (so close!)

Zylon · 2014-01-06 Reply Admin

¯\(°_o)/¯ I DUNNO LOL:
There where myriad connection errors.
There where also myriad spelling errors.

Those are just the whorst.

chubertdev · 2014-01-06 Reply Admin

no laughing matter:
DCRoss:
... and I am sorry that I could not think of any other drinks which contain umlauts in their names for after the signing (10), because putting umlauts in things just makes them more awesome. Heh. Ümläüt.

:snip:

At first glance, that comment looked so much like spam that I thought Akismet had given you a cup of coffee after posting it.

2014-01-06 Reply Admin

Don:
No SLA on a hosted system that was pivotal to the service delivery? Whoever came up with that decision needs to be sacked.

Welcome to The Daily Bullshit Story - you must be new here. <spoiler> This is one of them which is so badly embellished that it's simply no longer true. (aside from no SLA, I can't believe this idea of randomized backups either.) </spoiler>

2014-01-06 Reply Admin

ufmace:
For TRWTF - why didn't he do any of about 50 other things instead of just tell everyone to wait for the backup to complete? Tell the hosting company to kill the backup, or route requests to the good server, or spin up a new server with a hosting company that isn't insane, or....?

1) tell them to Kill the backup - he did, and they told him to fuck off (ok, he should perhaps have argued the point....) 2) reroute requests - no doubt still would've been pushing shit uphill given Roger sounds like a Brain Giant. 3) new hosting company - you did understand that there was a bit of a time constraint (the shit hit the fan 3 days out from close of enrollment, and by the sound of it by the time he'd gotten Roger to tell him anything he'd lost another day)

2014-01-06 Reply Admin

DCRoss:
Peter took on the task of crafting an e-mail to the dean of Northeastern Institute of WTF with about 25 apologies and a request that the class enrollment period be extended a few days.

To: The Office of the Dean, NEIoWTF From: Peter Son of Peter, Holder Of The Holy Relics Of University IT. Subject: I'm sorry (1).

Martin,

I'm sorry that I blew up the registration system (2). At the time it seemed like a good idea to go with a hosting company run by a family member even if he did have a room temperature IQ and approximately three weeks of experience in the field. I'm sorry I did that (3). I'm sorry that we never discussed service level agreements beyond the level of "Don't worry, you'll be fine!" (4). I am also sorry that all negotiations were conducted in the VIP room at the Brass Rail (5), and I am still very sorry that we forgot to invite you seven times in a row (6). I am sorry that our bar tab for those meetings still exceeds all other costs involved in the implementation (7) and I am very sorry that I drank all of those shots of Jägermeister before signing the contract (8). I am also sorry that I drank all of that Goldschläger while signing it (9), and I am sorry that I could not think of any other drinks which contain umlauts in their names for after the signing (10), because putting umlauts in things just makes them more awesome. Heh. Ümläüt.

Sorry about that last part (11). It was not directed at you, but at the pounding in my head which still has not gone away. Sorry, head (12). Very sorry (13). Ow. Now I am feeling very sorry that I turned the lights back on (14). I'm sorry, lights (15).

And now I am sorry I put that last paragraph in there (16). It really doesn't belong. Sorry, extra paragraph, but it just wasn't meant to be (17).

Getting back to the point (sorry for going off topic (18)), I am sorry for not having even thought about having a backup plan (19) and for letting Roger get off so easily (20).

Did I use the phrase "get off"? Sorry about that (21). I was just reminded of something that Roger said during one of out planning meetings when Gertrude --

Sorry again (22). I'll stop now. Sorry (23).

Getting back to the point, would it be possible to just extend the class enrolment period by a few days instead of trying to fix the problem with our registration server? I'm sorry that it never even occurred to me that we could do that (24), and have been considering firing myself as an example to the others.

Very very very sorry (25),

Peter

Löwenbräu sprang to my mind

2014-01-06 Reply Admin

x2 - but well done!

2014-01-06 Reply Admin

Jörg:
x2 - but well done!

and I was sort of curious about the seagull shit, until I saw it was cherry flavoured something....

2014-01-06 Reply Admin

MoSlo:
"randomized server backup system"??? In what world would this EVER be a sane idea?

In the movie Idiocracy?

2014-01-06 Reply Admin

Aigarius:
"ok, then - remove the loaded server from the round robin and start up 5 other servers in its place, oh and FYI - you are paying for them" should have been the right answer.

That's kind of what I was thinking.

Considering any large university's resources, I'm not sure why they'd even turn that over to a third party data center.

dkf · 2014-01-06 Reply Admin

eric76:
Considering any large university's resources, I'm not sure why they'd even turn that over to a third party data center.

The departments probably regard their internal IT supplier as a hostile third party.

2014-01-06 Reply Admin

"It's really unfortunate that those backups are going to get corrupted when I reboot my server."

"That's okay, just let me adjust the priority on that backup task and I'll be done real quick."

"So, why is the load-balancing server still in round-robin, rather than directing all of the traffic to the server that you didn't fuck up?"

Balanced Registration

Leave a comment on “Balanced Registration”