The Daily WTF: Curious Perversions in Information Technology

2013-12-30 Reply Admin

It'd be good if there was some way to stop frost comments. They're soooo tedious.

2013-12-30 Reply Admin

Damn autocorrect... Frist, frist, frist...

mikeTheLiar · 2013-12-30 Reply Admin

Fritz:
Damn autocorrect... Frist, frist, frist...

Don't you mean Fritz, Fritz, Fritz?

2013-12-30 Reply Admin

Lucky the server wasn't set to UTC

martijntje · 2013-12-30 Reply Admin

And nobody investigated the cron mails they received daily, around lunchtime, saying something along the lines of:

kill: (21342) - No such process

I would consider that to be TRWTF.

2013-12-30 Reply Admin

mikeTheLiar:
Don't you mean Fritz, Fritz, Fritz?

Fritzo, Fritzo, Fritzo.

2013-12-30 Reply Admin

At least it was only

kill

and not

kill -9

or data might have been lost or corrupted.

2013-12-30 Reply Admin

Yeah but a kill 9 might have been logged letting them find the problem quicker

2013-12-30 Reply Admin

Somebody was knowledgeable enough to set up a cron job but did not know/care about that job being kill with a PID? sounds too far fetched

Matt Westwood · 2013-12-30 Reply Admin

Boss says: "Set up a CRON job to stop (such-and-such a program) on 22nd December."

Apprentice says: "How do I do that?"

Boss says: "Here's how to stop a program: first you do (whatever command it was to identify what the process ID is of the program in question, I can't remember), and then when you have its process ID, kill it." (Demonstrates by using "kill" to stop the program in question, which happens to have to ID 21342).

Apprentice says: "Okay, but how do I set up a CRON job?"

Boss says: "Use man for instructions, gotta run, got a meeting to go to."

dkf · 2013-12-30 Reply Admin

KatrinaS:
Yeah but a kill 9 might have been logged letting them find the problem quicker

Unless they use a non-standard kill that does that, no. That signal is the “Die. Now.” one, which you use only when all other options are exhausted. The process can't log what happened, because it is dead. (A supervisor process could log it, but could also have been the one that got hit…)

2013-12-30 Reply Admin

dkf:
KatrinaS:
Yeah but a kill 9 might have been logged letting them find the problem quicker
Unless they use a non-standard kill that does that, no. That signal is the “Die. Now.” one, which you use only when all other options are exhausted. The process can't log what happened, because it is dead. (A supervisor process could log it, but could also have been the one that got hit…)

The kill might not be logged, but the application itself could log an unclean startup (abnormal termination?) when it restarted. If the application gracefully shuts down in most critical error situations, this would be a red flag indicating that the application abnormally ended (possibly because someone killed it).

Zylon · 2013-12-30 Reply Admin

Ah yes, this was the WTF with the inexplicable food obsession.

2013-12-30 Reply Admin

OK, so that explains the process dying at 10:12pm, but why was it dying at 12:22?

2013-12-30 Reply Admin

Blake Swopes:
OK, so that explains the process dying at 10:12pm, but why was it dying at 12:22?

Let me summarise to you how this process works:

Steve:
I was called in twice because of this cron job entry:
12 22 * * * kill 21342
Turns out, sometimes task id 21342 was our mission-critical flagship application.

Remy Porter:
*snip*

chubertdev · 2013-12-30 Reply Admin

anonymous:
dkf:
KatrinaS:
Yeah but a kill 9 might have been logged letting them find the problem quicker
Unless they use a non-standard kill that does that, no. That signal is the “Die. Now.” one, which you use only when all other options are exhausted. The process can't log what happened, because it is dead. (A supervisor process could log it, but could also have been the one that got hit…)
The kill might not be logged, but the application itself could log an unclean startup (abnormal termination?) when it restarted. If the application gracefully shuts down in most critical error situations, this would be a red flag indicating that the application abnormally ended (possibly because someone killed it).

But that doesn't change the kill vs. kill 9 part.

operagost · 2013-12-30 Reply Admin

Matt Westwood:
Boss says: "Set up a CRON job to stop (such-and-such a program) on 22nd December."
Apprentice says: "How do I do that?"

Boss says: "Here's how to stop a program: first you do (whatever command it was to identify what the process ID is of the program in question, I can't remember), and then when you have its process ID, kill it." (Demonstrates by using "kill" to stop the program in question, which happens to have to ID 21342).

Apprentice says: "Okay, but how do I set up a CRON job?"

Boss says: "Use man for instructions, gotta run, got a meeting to go to."

That, or an experienced admin put it in there as a one-time fix for some issue, and forgot to remove it after it ran.

2013-12-30 Reply Admin

chubertdev:
anonymous:
dkf:
KatrinaS:
Yeah but a kill 9 might have been logged letting them find the problem quicker
Unless they use a non-standard kill that does that, no. That signal is the “Die. Now.” one, which you use only when all other options are exhausted. The process can't log what happened, because it is dead. (A supervisor process could log it, but could also have been the one that got hit…)
The kill might not be logged, but the application itself could log an unclean startup (abnormal termination?) when it restarted. If the application gracefully shuts down in most critical error situations, this would be a red flag indicating that the application abnormally ended (possibly because someone killed it).

But that doesn't change the kill vs. kill 9 part.

It does. By default, kill sends SIGTERM, which can be caught by the process, allowing it to exit gracefully. If you use kill -s 9 to send SIGKILL, it can't exit gracefully. SIGKILL cannot be caught, and the process is terminated immediately.

In the former case, the application might log a SIGTERM followed by a normal shutdown. In the latter case, the process would die before it could log anything, so it would log nothing until it was restarted, at which point it might log that it was restarting from an unclean shutdown.

TheCPUWizard · 2013-12-30 Reply Admin

So the correct fix is for the mission critical process(es) to check if their id is 21342. If so, gracefully exit and restart...Still leaves a tiny window...but should be a mjor improvement....

2013-12-30 Reply Admin

Fritz:
It'd be good if there was some way to stop frost comments. They're soooo tedious.

Turning up the heat is a great way to get rid of frost. But watch out for yellow frost. Frosty Piss can be a big problem when it melts.

chubertdev · 2013-12-30 Reply Admin

anonymous:
It does. By default, kill sends SIGTERM, which can be caught by the process, allowing it to exit gracefully. If you use kill -s 9 to send SIGKILL, it can't exit gracefully. SIGKILL cannot be caught, and the process is terminated immediately.
In the former case, the application might log a SIGTERM followed by a normal shutdown. In the latter case, the process would die before it could log anything, so it would log nothing until it was restarted, at which point it might log that it was restarting from an unclean shutdown.

Ahhh, thanks.

2013-12-30 Reply Admin

operagost:
Matt Westwood:
Boss says: "Set up a CRON job to stop (such-and-such a program) on 22nd December."
Apprentice says: "How do I do that?"

Boss says: "Here's how to stop a program: first you do (whatever command it was to identify what the process ID is of the program in question, I can't remember), and then when you have its process ID, kill it." (Demonstrates by using "kill" to stop the program in question, which happens to have to ID 21342).

Apprentice says: "Okay, but how do I set up a CRON job?"

Boss says: "Use man for instructions, gotta run, got a meeting to go to."
That, or an experienced admin put it in there as a one-time fix for some issue, and forgot to remove it after it ran.

An experienced admin who doesn't know the difference between "cron" and "at"? Well, maybe he also just wanted to check on the process and confused "ps" with "kill", who knows ...

2013-12-30 Reply Admin

anonymous:
dkf:
KatrinaS:
Yeah but a kill 9 might have been logged letting them find the problem quicker
Unless they use a non-standard kill that does that, no. That signal is the “Die. Now.” one, which you use only when all other options are exhausted. The process can't log what happened, because it is dead. (A supervisor process could log it, but could also have been the one that got hit…)
The kill might not be logged, but the application itself could log an unclean startup (abnormal termination?) when it restarted. If the application gracefully shuts down in most critical error situations, this would be a red flag indicating that the application abnormally ended (possibly because someone killed it).

It's not like they lacked indication that the application abnormally ended ...

2013-12-30 Reply Admin

foo AKA fooo:
anonymous:
dkf:
KatrinaS:
Yeah but a kill 9 might have been logged letting them find the problem quicker
Unless they use a non-standard kill that does that, no. That signal is the “Die. Now.” one, which you use only when all other options are exhausted. The process can't log what happened, because it is dead. (A supervisor process could log it, but could also have been the one that got hit…)
The kill might not be logged, but the application itself could log an unclean startup (abnormal termination?) when it restarted. If the application gracefully shuts down in most critical error situations, this would be a red flag indicating that the application abnormally ended (possibly because someone killed it).
It's not like they lacked indication that the application abnormally ended ...

It didn't abnormally end. It was told to shut down, and did so. The question of "who or what told it to shut down" is a distinctly different question than "why did it terminate without explanation".

2013-12-30 Reply Admin

martijntje:
And nobody investigated the cron mails they received daily, around lunchtime, saying something along the lines of:
kill: (21342) - No such process

Do you really think that they bothered to configure the application user's mail to go somewhere? Far more likely that it just built up forever in the local mailbox, never to be seen by human eyes.

2013-12-30 Reply Admin

anonymous:
Blake Swopes:
OK, so that explains the process dying at 10:12pm, but why was it dying at 12:22?
Let me summarise to you how this process works:
Steve:
I was called in twice because of this cron job entry:
12 22 * * * kill 21342
Turns out, sometimes task id 21342 was our mission-critical flagship application.
Remy Porter:
*snip*

22:12 != 12:22 so yeah why was it dying at 12:22?

2013-12-30 Reply Admin

Anonymouso:
anonymous:
Blake Swopes:
OK, so that explains the process dying at 10:12pm, but why was it dying at 12:22?
Let me summarise to you how this process works:
Steve:
I was called in twice because of this cron job entry:
12 22 * * * kill 21342
Turns out, sometimes task id 21342 was our mission-critical flagship application.
Remy Porter:
*snip*
22:12 != 12:22 so yeah why was it dying at 12:22?

Who said it was dying at 12:22? I'll help you: it was Remy

2013-12-30 Reply Admin

anonymous:
Anonymouso:
anonymous:
Blake Swopes:
OK, so that explains the process dying at 10:12pm, but why was it dying at 12:22?
Let me summarise to you how this process works:
Steve:
I was called in twice because of this cron job entry:
12 22 * * * kill 21342
Turns out, sometimes task id 21342 was our mission-critical flagship application.
Remy Porter:
*snip*
22:12 != 12:22 so yeah why was it dying at 12:22?
Who said it was dying at 12:22? I'll help you: it was Remy

You lost me. I don't know if you're being serious or not, so I'll sleep on it.

2013-12-30 Reply Admin

Anonymouso:
anonymous:
Anonymouso:
anonymous:
Blake Swopes:
OK, so that explains the process dying at 10:12pm, but why was it dying at 12:22?
Let me summarise to you how this process works:
Steve:
I was called in twice because of this cron job entry:
12 22 * * * kill 21342
Turns out, sometimes task id 21342 was our mission-critical flagship application.
Remy Porter:
*snip*
22:12 != 12:22 so yeah why was it dying at 12:22?
Who said it was dying at 12:22? I'll help you: it was Remy
You lost me. I don't know if you're being serious or not, so I'll sleep on it.

Guys, seriously, this is the internet. Sometimes, you just have to spell it out.

The actual article stated that the process always died during lunch, and specifically stated "12:22PM" in the last paragraph. However, the crontab line given in the article got the minute and hour reversed, which would should have caused the process kill to occur at 22:12, or 10:12PM.

tl;dr: anonymization failure.

2013-12-30 Reply Admin

22:12 is a strange time for a lunch

2013-12-31 Reply Admin

Maybe the WTF is that they aren't running a decent network service monitor?

I'd much rather have nagios tell me that a service has crashed then my boss, or my users.

2013-12-31 Reply Admin

I don't quite get how the process can have been running for days and then get killed by that cron job. I mean, unless this thing respawns itself each day or does something silly like that, it would have had that same process ID since it started, so why didn't it get killed the first time that cron job ran?

2013-12-31 Reply Admin

Presumably there's some sort of scheduled daily restart. Just go with it.

zelmak · 2013-12-31 Reply Admin

anon:
22:12 is a strange time for a lunch

They used UTC for their server clocks.

2013-12-31 Reply Admin

HiddenWindshield:
Anonymouso:
anonymous:
Anonymouso:
anonymous:
Blake Swopes:
OK, so that explains the process dying at 10:12pm, but why was it dying at 12:22?
Let me summarise to you how this process works:
Steve:
I was called in twice because of this cron job entry:
12 22 * * * kill 21342
Turns out, sometimes task id 21342 was our mission-critical flagship application.
Remy Porter:
*snip*
22:12 != 12:22 so yeah why was it dying at 12:22?
Who said it was dying at 12:22? I'll help you: it was Remy
You lost me. I don't know if you're being serious or not, so I'll sleep on it.
Guys, seriously, this is the internet. Sometimes, you just have to spell it out.

The actual article stated that the process always died during lunch, and specifically stated "12:22PM" in the last paragraph. However, the crontab line given in the article got the minute and hour reversed, which would should have caused the process kill to occur at 22:12, or 10:12PM.

tl;dr: anonymization failure.

My point (all along) was that MOST of the story was invented by Remy Porter and probably all the submitter gave him was the crontab line and a bare couple of facts. Hence my fictional quotes from Steve and Remy Porter. That's probably about how it went. So yes, anonymisation failure.

(Or, possibly, Steve made the anonymisation error by retyping the crontab line from memory, and misremembered the crontab format.)

2013-12-31 Reply Admin

hobbes:
I'd much rather have nagios tell me that a service has crashed then my boss, or my users.

Really? I'd be quite happy to hear that my boss and/or users had crashed.

2013-12-31 Reply Admin

ideo:
hobbes:
I'd much rather have nagios tell me that a service has crashed then my boss, or my users.
Really? I'd be quite happy to hear that my boss and/or users had crashed.

You'd hear that they did, right after you heard that the service crashed. First the service crashes, then the boss or users (I'm assuming it's a logical or, not an exclusive or).

2013-12-31 Reply Admin

The REAL WTF is the fact, they have no auto-restart service active!

Like "daemontools, monit, upstart..."

I thought it's standard toolset for every system admin and every mission-critical process is monitored!

2013-12-31 Reply Admin

This is the best joke I had on DailyWTF this year.

2013-12-31 Reply Admin

anonymous:
ideo:
hobbes:
I'd much rather have nagios tell me that a service has crashed then my boss, or my users.
Really? I'd be quite happy to hear that my boss and/or users had crashed.
You'd hear that they did, right after you heard that the service crashed. First the service crashes, then the boss or users (I'm assuming it's a logical or, not an exclusive or).

The opposite of "exclusive" is "inclusive", not "logical". Both exclusive and inclusive "or" are logical functions. Thanks for your attention.

2014-01-02 Reply Admin

Logic Nazi:
anonymous:
ideo:
hobbes:
I'd much rather have nagios tell me that a service has crashed then my boss, or my users.
Really? I'd be quite happy to hear that my boss and/or users had crashed.
You'd hear that they did, right after you heard that the service crashed. First the service crashes, then the boss or users (I'm assuming it's a logical or, not an exclusive or).
The opposite of "exclusive" is "inclusive", not "logical". Both exclusive and inclusive "or" are logical functions. Thanks for your attention.

Linguistics and logic differ in some ways, one being their use of the word "or". Let me summarise for you with this table.

LogicalLinguistic Inclusive"or""and/or" Exclusive"exclusive or""or"

As you can see, "or" occurs twice in this table (unless clarified, i.e. "exclusive or" or "and/or"). One "or" is logical, and the other "or" is linguistic. I meant the logical one.

2014-01-02 Reply Admin

Logic Nazi:
The opposite of "exclusive" is "inclusive", not "logical".

Also, shame on you for inventing an opposite where I used an implied "and". There is only and exactly one "or" that is logical [and] not exclusive.

2014-01-02 Reply Admin

Luc:
At least it was only
kill
and not
kill -9
or data might have been lost or corrupted.

That would have been an even bigger WTF. Anybody designing reliable software knows, that you need to be prepared for your program dying at any point in time and be able to recover on the next run without corrupting or losing data. Such an interruption could have happened for so many different reasons, and if you have designed for that, then kill -9 is actually quite graceful.

The existence of a signal handler does not automagically protect against data corruption. Many library functions are not signal safe, it is so easy to call the wrong function and cause memory corruption.

2014-01-03 Reply Admin

KatrinaS:
Yeah but a kill 9 might have been logged letting them find the problem quicker

Another fun technique is "kill -11".

2014-01-03 Reply Admin

anonymous:
Logic Nazi:
anonymous:
ideo:
hobbes:
I'd much rather have nagios tell me that a service has crashed then my boss, or my users.
Really? I'd be quite happy to hear that my boss and/or users had crashed.
You'd hear that they did, right after you heard that the service crashed. First the service crashes, then the boss or users (I'm assuming it's a logical or, not an exclusive or).
The opposite of "exclusive" is "inclusive", not "logical". Both exclusive and inclusive "or" are logical functions. Thanks for your attention.
Linguistics and logic differ in some ways, one being their use of the word "or". Let me summarise for you with this table.
LogicalLinguistic Inclusive"or""and/or" Exclusive"exclusive or""or"

As you can see, "or" occurs twice in this table (unless clarified, i.e. "exclusive or" or "and/or"). One "or" is logical, and the other "or" is linguistic. I meant the logical one.

Is that i.e. "excusive or" exclusive or "and/or", or "exclusive or" and/or "and/or"?

Your turn.

2014-01-06 Reply Admin

Clair I.T.:
anonymous:
Logic Nazi:
anonymous:
ideo:
hobbes:
I'd much rather have nagios tell me that a service has crashed then my boss, or my users.
Really? I'd be quite happy to hear that my boss and/or users had crashed.
You'd hear that they did, right after you heard that the service crashed. First the service crashes, then the boss or users (I'm assuming it's a logical or, not an exclusive or).
The opposite of "exclusive" is "inclusive", not "logical". Both exclusive and inclusive "or" are logical functions. Thanks for your attention.
Linguistics and logic differ in some ways, one being their use of the word "or". Let me summarise for you with this table.
LogicalLinguistic Inclusive"or""and/or" Exclusive"exclusive or""or"

As you can see, "or" occurs twice in this table (unless clarified, i.e. "exclusive or" or "and/or"). One "or" is logical, and the other "or" is linguistic. I meant the logical one.
Is that i.e. "excusive or" exclusive or "and/or", or "exclusive or" and/or "and/or"?

Your turn.

I anticipated this question and the answer is, it doesn't matter. Either is a true statement. I'd have been more specific otherwise.

2014-01-06 Reply Admin

anonymous:
Clair I.T.:
anonymous:
Logic Nazi:
anonymous:
ideo:
hobbes:
I'd much rather have nagios tell me that a service has crashed then my boss, or my users.
Really? I'd be quite happy to hear that my boss and/or users had crashed.
You'd hear that they did, right after you heard that the service crashed. First the service crashes, then the boss or users (I'm assuming it's a logical or, not an exclusive or).
The opposite of "exclusive" is "inclusive", not "logical". Both exclusive and inclusive "or" are logical functions. Thanks for your attention.
Linguistics and logic differ in some ways, one being their use of the word "or". Let me summarise for you with this table.
LogicalLinguistic Inclusive"or""and/or" Exclusive"exclusive or""or"

As you can see, "or" occurs twice in this table (unless clarified, i.e. "exclusive or" or "and/or"). One "or" is logical, and the other "or" is linguistic. I meant the logical one.
Is that i.e. "excusive or" exclusive or "and/or", or "exclusive or" and/or "and/or"?

Your turn.
I anticipated this question and the answer is, it doesn't matter. Either is a true statement. I'd have been more specific otherwise.

WTF? You answered me? Didn't you notice that you were supposed to ask me a question? I went to all that trouble to set this up (it's ors all the way down), told you it's your turn, and you refused? Sheeps, some people.

2014-01-07 Reply Admin

Clair I.T.:
anonymous:
Clair I.T.:
anonymous:
Logic Nazi:
anonymous:
ideo:
hobbes:
I'd much rather have nagios tell me that a service has crashed then my boss, or my users.
Really? I'd be quite happy to hear that my boss and/or users had crashed.
You'd hear that they did, right after you heard that the service crashed. First the service crashes, then the boss or users (I'm assuming it's a logical or, not an exclusive or).
The opposite of "exclusive" is "inclusive", not "logical". Both exclusive and inclusive "or" are logical functions. Thanks for your attention.
Linguistics and logic differ in some ways, one being their use of the word "or". Let me summarise for you with this table.
LogicalLinguistic Inclusive"or""and/or" Exclusive"exclusive or""or"

As you can see, "or" occurs twice in this table (unless clarified, i.e. "exclusive or" or "and/or"). One "or" is logical, and the other "or" is linguistic. I meant the logical one.
Is that i.e. "excusive or" exclusive or "and/or", or "exclusive or" and/or "and/or"?

Your turn.
I anticipated this question and the answer is, it doesn't matter. Either is a true statement. I'd have been more specific otherwise.
WTF? You answered me? Didn't you notice that you were supposed to ask me a question? I went to all that trouble to set this up (it's ors all the way down), told you it's your turn, and you refused? Sheeps, some people.

Read through the comment thread (which I've helpfully quoted) and you'll see that I (the anonymous) have consistently answered rather than asked questions.

2014-01-11 Reply Admin

TRWTF is that they fired someone and then let them work on the crontab...

Classic WTF: A Crony Joke

Leave a comment on “Classic WTF: A Crony Joke”