• (cs)

    Why didn't the sysadmin notice that here is a problem with the server? Or, anyone who is actually responsible for this?

    Why was this restart cronjob there anyway? And how was it affecting the performance of POSIX commands?

  • EuroGuy (unregistered)

    So there was a cron job that restarts the server every minute, the sysadmin was not aware of this and removes it without questioning why it was there in the first place?

    I say the sysadmin is the real WTF

  • Pista (unregistered)

    Yeah, this story is incomplete. It lacks TRWTF.

  • (cs) in reply to Pista
    Pista:
    Yeah, this story is incomplete. It lacks TRWTF.
    Here's TRWTF:
    And the klaxon would remain on until the crisis was resolved.
    So people are supposed to work on a concentration-requiring task while a klaxon is blaring away? The point being that if it is a *klaxon*, then it blares away at *ear-breaking* volume.
  • (cs)

    And what did "updating the firmware" have to do with Apache not restarting? Or was it restarting and failing? Completely unclear.

  • (cs) in reply to Steve The Cynic
    Steve The Cynic:
    Pista:
    Yeah, this story is incomplete. It lacks TRWTF.
    Here's TRWTF:
    And the klaxon would remain on until the crisis was resolved.
    So people are supposed to work on a concentration-requiring task while a klaxon is blaring away? The point being that if it is a *klaxon*, then it blares away at *ear-breaking* volume.

    Yeah... not sure I believe that bit - smells a bit embellishy to me.

  • (cs) in reply to derari
    derari:
    Why was this restart cronjob there anyway? And how was it affecting the performance of POSIX commands?
    First part: who knows? Bug? Debug script that got shipped? Something else?

    Second part: It smells of a combination of anonymistic license and an under-specced server that doesn't have enough spare cycles to run them at a sane speed while Apache is starting and starting and starting...

  • (cs)

    There are so many wtfs in the story, I wonder if it has been made up (inexpertly).

    "I think the firmware was updated recently", said Jacky, the system administrator. The system administrator whould know, not "think".

    And please: "firmware"? On a webserver???

    Then the question of the cron job. I can't think of any reason for a cron job sending apache down into a 10 second sleep every minute.

    On to "Executing simple posix commands took thirty seconds apiece": please, cycling apache every 50 seconds does not hamper the system so much as to render a shell unusable.

    This story is just strange.

  • (cs) in reply to skotl
    skotl:
    And what did "updating the firmware" have to do with Apache not restarting? Or was it restarting and failing? Completely unclear.
    I thought it was clear. The cron job (successfully) restarted Apache, and Apache logged that fact. Then a minute after restarting Apache, the cron job ran again and restarted Apache again.

    And again, and again, and ...

    The log output badly needs reformatting, though.

    Because if it was, it would be easier to see Apache catching SIGTERM (boring terminate signal) rather than "faulting". And a sysadmin who doesn't know the difference is either a raw noobeginner or not worthy of the name.

  • Sir Galahad the Pure (unregistered) in reply to faoileag
    faoileag:

    And please: "firmware"? On a webserver???

    Because webservers do not possess RAID controllers, motherboards or ethernet cards? All of these could have updateable firmware.
    faoileag:
    Then the question of the cron job. I can't think of any reason for a cron job sending apache down into a 10 second sleep every minute.
    Operator error? Maybe they wanted to get rid of things like leaking memory from PHP scripts, or something. Maybe they did it to fend off hackers, because mauve has the most RAM or something. Who knows. Not unlikely, though.
    faoileag:
    On to "Executing simple posix commands took thirty seconds apiece": please, cycling apache every 50 seconds does not hamper the system so much as to render a shell unusable.

    This story is just strange.

    Anyone caredd to read "this didn't make problems, until apache could'nt be stopped" at the end of TFA?

  • ANON (unregistered) in reply to Sir Galahad the Pure
    Sir Galahad the Pure:
    faoileag:

    And please: "firmware"? On a webserver???

    Because webservers do not possess RAID controllers, motherboards or ethernet cards? All of these could have updateable firmware.

    And these firmware updates let the command to stop apache fail? Sounds still strange to me.

  • (cs)

    Why has the "firmware on the RAID controller" to pick an example from the above post stopped Apache starting?

    And all the other WTFs on the WTF, as mentioned above.

    Bit of a mess, this one, again :(

    (and before people start bleating about "you get what you pay for" - I get that, honestly I do, I'm just baffled at the continued decline in quality of the front page stories as it used to be so much better)

  • (cs) in reply to Sir Galahad the Pure
    Sir Galahad the Pure:
    faoileag:
    And please: "firmware"? On a webserver???
    Because webservers do not possess RAID controllers, motherboards or ethernet cards? All of these could have updateable firmware.
    Yes. But such a firmware update would not touch the crontab of the operating system running on the machine employing the controller/motherboard/ethernet card.
    Sir Galahad the Pure:
    faoileag:
    Then the question of the cron job. I can't think of any reason for a cron job sending apache down into a 10 second sleep every minute.
    Operator error? Maybe they wanted to get rid of things like leaking memory from PHP scripts, or something.
    Yes, possible. I have seen it happen before. But in those cases, apache is cycled once a day. If you have to cycle apache every minute something will be so fundamentally wrong with your system that cycling apache probably won't help at all anyway.
    Sir Galahad the Pure:
    Anyone caredd to read "this didn't make problems, until apache could'nt be stopped" at the end of TFA?
    Actually, yes. But apache could be stopped. The logs show it. That sentence does not actually make much sense in the story, since it introduces a hitherto unknown problem we have not been told about.

    So, to repeat me: This story is just strange.

  • (cs) in reply to Steve The Cynic
    Steve The Cynic:
    The cron job (successfully) restarted Apache
    Actually, the cron job did not restart apache but executed a stop/start sequence (with a ten second sleep in between the two).

    A restart might have offered at least some explanation as to why the posix commands took so long, since if you use apachctl restart, the config files are syntax checked.

  • (cs)

    Ok... trying to explain that "firmware" thing. I just remembered...

    Think of a tank farm. Gauging on the tanks is done by sensors which report to a master device. The master device (typically small enough to be mounted on a top-hat rail) takes the data and - as a webserver - provides a web site that visualizes the data from the tanks.

    For the master device, it would not be unheard of if a firmware update would simply mean swapping the existing file system with an image containing the new firmware.

    In such a setting, klaxons blaring away would also be quite normal, since loosing the ability to monitor filling status in tanks can have serious consequences.

    Taking this into consideration the story could be real.

    The only wtf would then be supplying a firmware update with a cron job restarting apache every minute.

  • Charles F. (unregistered) in reply to Sir Galahad the Pure
    Sir Galahad the Pure:
    faoileag:

    And please: "firmware"? On a webserver???

    Because webservers do not possess RAID controllers, motherboards or ethernet cards? All of these could have updateable firmware.
    There is no ISO standard for what can be considered firmware.

    If you have a "network appliance" you might consider the whole system -- OS, Apache, configuration scripts -- to be "firmware." The appliance could offer shell access as a means to customize the install. This would also explain how an Apache start/stop would bog down the system; "appliance" boxen frequently only have enough computing power for whatever specific task they are intended for.

    But if they were using some kind of network appliance to front their web site, you'd think that detail might be mentioned.

  • Very Hardworking Dude (unregistered)
    in increasing frequency from every minute to 24 hours

    That's decreasing frequency.

  • Les (unregistered) in reply to faoileag
    faoileag:
    Yes, possible. I have seen it happen before. But in those cases, apache is cycled once a day.

    Exactly. I think TRWTF is cron. Someone wanted to re-start Apache once a day, and ended-up restarting it every minute instead.

    The other RWTF is letting people edit cron when they don't know what they're doing. That goes double when those same people think that software needs to be re-started every so often for no good reason.

  • Mr. Lamepun (unregistered) in reply to Charles F.

    Are you FIRMly committed to that opinion?

    Mr. Lamepun

    Charles F.:
    Sir Galahad the Pure:
    faoileag:

    And please: "firmware"? On a webserver???

    Because webservers do not possess RAID controllers, motherboards or ethernet cards? All of these could have updateable firmware.
    There is no ISO standard for what can be considered firmware.

    If you have a "network appliance" you might consider the whole system -- OS, Apache, configuration scripts -- to be "firmware." The appliance could offer shell access as a means to customize the install. This would also explain how an Apache start/stop would bog down the system; "appliance" boxen frequently only have enough computing power for whatever specific task they are intended for.

    But if they were using some kind of network appliance to front their web site, you'd think that detail might be mentioned.

  • (cs) in reply to faoileag
    faoileag:
    Ok... trying to explain that "firmware" thing. I just remembered...

    Think of a tank farm. Gauging on the tanks is done by sensors which report to a master device. The master device (typically small enough to be mounted on a top-hat rail) takes the data and - as a webserver - provides a web site that visualizes the data from the tanks.

    For the master device, it would not be unheard of if a firmware update would simply mean swapping the existing file system with an image containing the new firmware.

    In such a setting, klaxons blaring away would also be quite normal, since loosing the ability to monitor filling status in tanks can have serious consequences.

    Taking this into consideration the story could be real.

    The only wtf would then be supplying a firmware update with a cron job restarting apache every minute.

    All that would be fine, except that this sort of visualisation is not done with web servers on SCADA systems(*). And even if it was, the actual control of tank filling status is done by relatively dumb controllers, usually some sort of PID loop controller. These controllers are quite capable of managing the tank levels / temperature levels / etc. all by themselves. Sure, you want to have monitoring displays so the humans know what is going on, but the critical low-tank-level alarms (that sound the klaxons) are usually driven by the loop controllers. (The PID algorithms include loss-of-control detection, and these will be hooked up to real klaxons and flashing lights.)

    (*) (Caveat lector: my knowledge of how this stuff is done is a little rusty, but industrial process control is a relatively conservative field, and not subject to huge innovation of the "Internet time" variety.)

  • (cs) in reply to faoileag
    faoileag:
    Steve The Cynic:
    The cron job (successfully) restarted Apache
    Actually, the cron job did not restart apache but executed a stop/start sequence (with a ten second sleep in between the two).

    A restart might have offered at least some explanation as to why the posix commands took so long, since if you use apachctl restart, the config files are syntax checked.

    Hmm. A slightly fussy-sounding distinction, but of course the difference is important. Either way, I note that you aren't disputing the analysis that whether or not it was stop-started or restarted, as far as Apache itself was concerned, the operation was a success, and it wasn't "faulting".
  • (cs) in reply to faoileag
    faoileag:
    There are so many wtfs in the story, I wonder if it has been made up (inexpertly).

    "I think the firmware was updated recently", said Jacky, the system administrator. The system administrator whould know, not "think".

    And please: "firmware"? On a webserver???

    Then the question of the cron job. I can't think of any reason for a cron job sending apache down into a 10 second sleep every minute.

    On to "Executing simple posix commands took thirty seconds apiece": please, cycling apache every 50 seconds does not hamper the system so much as to render a shell unusable.

    This story is just strange.

    Hmmm, prehaps you have solved today's "Mystery of the Missing TDWTF"... it's the story itself!

  • (cs)

    I would like to know what, if any, parts of today's WTF weren't pure fiction.

  • my name is missing (unregistered) in reply to Zylon

    The comments.

  • Matteo (unregistered) in reply to faoileag

    I think you may be overthinking this; I read the "firmware update" explanation as a way to placate the boss, and assure him it wasn't the dreaded hackers.

    I'm still at a loss as to who is supposed to have inserted the cron job, though I'd bet, as noted above, that it was someone who wanted to restart Apache daily and effed it up.

  • (cs) in reply to Steve The Cynic
    Steve The Cynic:
    All that would be fine, except that this sort of visualisation is not done with web servers on SCADA systems(*). And even if it was, the actual control of tank filling status is done by relatively dumb controllers
    In the case I'm thinking of the gauge manufacturer supplied the visualization unit; this would be hooked up with the manufacturers sensor equipment. You would see alarms on the screen if, e.g., a tank reached 80% filling capacity. Control of the filling process would be handled separately and not from within that system.
  • (cs) in reply to Zylon
    Zylon:
    I would like to know what, if any, parts of today's WTF weren't pure fiction.
    Googling some words from the article, I found this:

    http://pastebin.com/pvki7hGh

    That's today's story, unembellished.

  • thedailyftb (the-daily-fuck-the-butt) (unregistered) in reply to @Deprecated
    @Deprecated:
    faoileag:
    This story is just strange.
    Hmmm, prehaps you have solved today's "Mystery of the Missing TDWTF"... it's the story itself!
    This is another story embellished (beyond any plausibility) by Erik Gern. Yes, Erik Gern, the "father" of Hanzo.
  • (cs)

    Obviously fake. A real Mr. Cullen would insist on rebooting the server IMMEDIATELY! OMG Hackers are getting us, quick, reboot NOW!

  • (cs) in reply to derari

    Most of the problems with the story are due to the very simple thing that this obviously is an embedded system running linux. Things like consumer routers and such.

    This explaines why restarting (e.g. the network interface or apache) slows down the thing so much (it's probably a low-clock ARM chipset) and why "firmware" is used, as well as why the sysadmin wasn't aware of the cronjobs running.

    Might be bad anonimization or it might be legitimately a webserver running on "embedded type" hardware, there are a lot of legitime uses.

    Reasons:

    • Running something like a simple grep in logs as root, that's bad practice in general but on these devices you usually can only login as root (to keep it simple)
    • "firmware" is usually used for the programming of these devices (because it is usually replaced uploading a single file which is flashed into the embedded memory)
    • Apache is in /opt/apache/. The prevalent package manager (optware) for these devices installs everything in /opt/
  • Anon (unregistered) in reply to skotl
    skotl:
    (and before people start bleating about "you get what you pay for" - I get that, honestly I do, I'm just baffled at the continued decline in quality of the front page stories as it used to be so much better)

    Probably has something to do with the fact that nowadays you risk Federal-PMITA-Prison if you expose any RWTFs, if not Guantanamo.

    Especially if someone might be able to use any of the information provided to hack something.

  • Content (And Not Star Wars) (unregistered) in reply to Anon
    Anon:
    Probably has something to do with the fact that nowadays you risk Federal-PMITA-Prison if you expose any RWTFs, if not Guantanamo.

    Especially if someone might be able to use any of the information provided to hack something.

    That doesn't explain the recent theming of articles with unrelated geek clichés, and elaborate purple prose.

  • (cs) in reply to thedailyftb (the-daily-fuck-the-butt)
    thedailyftb (the-daily-fuck-the-butt):
    @Deprecated:
    faoileag:
    This story is just strange.
    Hmmm, prehaps you have solved today's "Mystery of the Missing TDWTF"... it's the story itself!
    This is another story embellished (beyond any plausibility) by Erik Gern. Yes, Erik Gern, the "father" of Hanzo.
    Ah. That explains the mysterious case of the POSIX-commands taking 30s. This gem is not part of the original story (http://pastebin.com/pvki7hGh).
  • (cs) in reply to Steve The Cynic
    Steve The Cynic:
    So people are supposed to work on a concentration-requiring task while a klaxon is blaring away? The point being that if it is a *klaxon*, then it blares away at *ear-breaking* volume.
    That's the first thing they teach you at PHB school: people work better when you're constantly screaming at them.
  • (cs) in reply to faoileag
    faoileag:
    Zylon:
    I would like to know what, if any, parts of today's WTF weren't pure fiction.
    Googling some words from the article, I found this:

    http://pastebin.com/pvki7hGh

    That's today's story, unembellished.

    TRWTF is how much embellishment happened from that draft of 2 years ago

  • gnasher729 (unregistered) in reply to faoileag

    Excellent find. Seems that a line in the cronjob was commented out by using a hash character - but a hash character doesn't comment out lines in cron job.

  • anonymous (unregistered) in reply to my name is missing
    my name is missing:
    The comments.
    This comment is false.
  • Paul Neumann (unregistered) in reply to ratchet freak
    ratchet freak:
    TRWTF is how much embellishment happened from that draft of 2 years ago
    So TDWTF lag time is still ½ that of Butterfly Labs...
  • (cs) in reply to skotl
    skotl:
    (and before people start bleating about "you get what you pay for" - I get that, honestly I do, I'm just baffled at the continued decline in quality of the front page stories as it used to be so much better)

    Well, it's sort of like the End of the Internet ( http://hmpg.net/ ) -- we've used up all the good WTFs there ever were, and people aren't creating new ones fast enough.

  • jumentum (unregistered) in reply to faoileag
    faoileag:
    Zylon:
    I would like to know what, if any, parts of today's WTF weren't pure fiction.
    Googling some words from the article, I found this:

    http://pastebin.com/pvki7hGh

    That's today's story, unembellished.

    +1 for Featured Comment!

  • Try To Make It Idiot-Proof (unregistered) in reply to cellocgw
    cellocgw:
    Well, it's sort of like the End of the Internet ( http://hmpg.net/ ) -- we've used up all the good WTFs there ever were, and people aren't creating new ones fast enough.

    So, it turns out, that they can't invent a bigger idiot? Then why isn't my faith in humanity restored?

  • (cs) in reply to faoileag
    faoileag:
    Zylon:
    I would like to know what, if any, parts of today's WTF weren't pure fiction.
    Googling some words from the article, I found this:

    http://pastebin.com/pvki7hGh

    That's today's story, unembellished.

    We need a "suggest featured comment" button. Much, much more than that stupid "reply" button.

  • Coding Practices MUST Be Followed (unregistered)

    And finally we get the truth about The Daily WTF - the stories are either made up or altered so much that they may as well have been, and today the intern writing the story didn't do a very good job of it.

    captcha: abigo - the value of thedailywtf.com is a big zero.

  • ¯\(°_o)/¯ I DUNNO LOL (unregistered) in reply to Steve The Cynic
    Steve The Cynic:
    The log output badly needs reformatting, though.

    Because if it was, it would be easier to see Apache catching SIGTERM (boring terminate signal) rather than "faulting". And a sysadmin who doesn't know the difference is either a raw noobeginner or not worthy of the name.

    It would also be easier to see that the restarts happened at :01 or :02 of every minute. I didn't even need to read that cron was involved to know what was going on.

    So TRWTF is using a terminal program that doesn't display newlines properly?

    [root@localhost ~]# grep '\[notice\]' /opt/apache/logs/error_log | less [Tue Mar 29 06:42:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:42:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:43:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:43:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:44:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:44:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:45:02 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:45:12 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:46:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:46:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:47:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:47:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:48:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:48:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:49:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:49:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:50:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:50:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations
  • (cs) in reply to ¯\(°_o)/¯ I DUNNO LOL
    ¯\(°_o)/¯ I DUNNO LOL:
    Steve The Cynic:
    The log output badly needs reformatting, though.

    Because if it was, it would be easier to see Apache catching SIGTERM (boring terminate signal) rather than "faulting". And a sysadmin who doesn't know the difference is either a raw noobeginner or not worthy of the name.

    It would also be easier to see that the restarts happened at :01 or :02 of every minute. I didn't even need to read that cron was involved to know what was going on.

    So TRWTF is using a terminal program that doesn't display newlines properly?

    [root@localhost ~]# grep '\[notice\]' /opt/apache/logs/error_log | less [Tue Mar 29 06:42:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:42:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:43:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:43:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:44:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:44:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:45:02 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:45:12 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:46:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:46:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:47:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:47:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:48:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:48:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:49:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:49:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:50:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:50:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations
    Wait - noone made the obligatory comment so far?

    OK, here is the obligatory comment: TRWTF is PHP!

  • asdfg (unregistered) in reply to faoileag
    And please: "firmware"? On a webserver???
    It means that Jackie put the script there and needed to give some technobabel to keep further questions from being asked.
  • o11c (unregistered)

    Wow, no one has commented on the critical misformatting of the script itself?

    This article is terribly written ... the wtf itself is fine though.

  • (cs)

    Why do I get the feeling that the editor of this article has already been featured in an older article on this site?

    http://thedailywtf.com/Articles/5_years_C-pound_experience.aspx

  • foo (unregistered) in reply to skotl
    skotl:
    (and before people start bleating about "you get what you pay for" - I get that, honestly I do, I'm just baffled at the continued decline in quality of the front page stories as it used to be so much better)
    Don't forget inflation!

    Sure, you paid $0 in 5 years ago and got good quality. But these were 2008 dollars! With your lousy 2013 $0 you can't expect that much.

  • The Sauce (unregistered)

    The original story was both substantially more interesting and considerably more plausible. Altered story makes the submitter look like an idiot.

    Clearly the site authors are partaking in negative energy for their productions.

    As an aside, I don't think I'll submit anything of my own to this site unless this trend changes. I'd prefer not to be rewarded for my trouble and contributions by being unintentionally painted as a moron as a result of well-intentioned, but failed, attempts at improving humor.

Leave a comment on “83.3% Uptime”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article