The Daily WTF: Curious Perversions in Information Technology

2010-02-12 Reply Admin

Mail server today have limits on retry, space used, bounces to bounces and so on, all to prevent this from happening. On the other hand, managing a server is 24 hour job. There is no way this incident would have escaped monitoring in a proper environment.

If this didn't happen 25 years ago there is no excuse for running sendmail on an unmonitored server, with crap storage and without bounce and retry limits. And certainly this isn't the fault of the program manager.

2010-02-12 Reply Admin

reminds me of something that happened once at a rather large ISP I was working at. mind you, this was before my time there, so all I got was hearsay.

someone managed to send a 'goodbye! I quit!' mail to the entire company, all 60.000 employees got it. quite a few were away, and had set the away notification. for some reason, the away notifications started replying to away notifications.

even though the replied only once to every different user, that's still thousands of away notifications responding to thousands of away notifications. it came crashing down, hard.

XAPBob · 2010-02-12 Reply Admin

Anonymous:
Just because someone else fubar'd the config doesn't mean it isn't the OP's job to ensure it's running properly. It sounds like the company in the OP is fairly small, but still how can you blindly support something without grokking the config?

Easily

I have multiple servers (some of which I have never seen) which I'm meant to maintain.

Two are gentoo servers:

One of which had hardware failure, so is now virtualised (disk copy as was). I've gradually migrated services off it, it now only runs the intranet, with a hit rate of less than one hit per day, so it can go die.
The other is vital, but I have a decent backup system of the one critical service, and replacing the server with my virtual "backup" version will be a matter of an hours work.

I also have two custom linux based servers, They are the only two of their distro in the world, we have no documentation for them. Again most of the functionality has been migrated, file sharing and maintaining intra office connections are all they have left (oh and backup VPN connections)

2010-02-12 Reply Admin

I recall back in 98 I broke my university mail server by setting up my account to automatically forward a copy of all emails to a hotmail account - external email access wasn't available at the time so it was a common workaround.

Unfortunately I sent some fragment of code to myself and a friend, and either hotmail or our uni system decided to interpret the code fragment instead of treating it as body text and got very unhappy. Somewhere along the line the email bounced back, the bounce got forwarded again, and the new forward got rebounced. Over the weekend the accumulation each time of another three lines of headers had made each email ridiculously large and brought down the mail system until they disabled my account and that of my friend.

I logged in on the Monday to see my internet credit go from $20 to -$485 in about half a second due to the 2c/international megabyte charges. Took em two weeks to refund me the money because they thought I'd set it up deliberately.

2010-02-12 Reply Admin

This reminds me a lot of the SOBIG virus outburst, who would sent itself with its 2MB payload to an infected's buddy while spoofing another buddy's address.

My inbox exploded from the multiple "delivery failure" messages that contained the entire message, including the 2MB payload...

2010-02-12 Reply Admin

nonpartisan:
WilliamF:
Bcc is BLIND carbon copy which sends to the addresses without storing them in the message.
Whoosh!!

Yeah, I heard it too. In fact, I think that joke actually broke the sound barrier as it was passing over his head!

2010-02-12 Reply Admin

Wheaties:
Um, so what's the status now? . . . Ok, how about now? . . . Now? . . Now? . Now? Now?! Now!? Now!?! Now!!

I'm not a programmer, but I had a situation not unlike this. Wednesday morning we had a meeting and decided on the things we'd look at for an issue, and report back at Friday morning's meeting. No problem.

Actually there was a problem: my bosses (who are usually pretty sharp and good to work for) decided to keep dropping in to offer new ideas, drag me into two hour meetings on unrelated projects that could have waited, etc. That week, I was scheduled to be off Friday, and my girlfriend and I had made plans for that Friday and bought tickets to a movie well in advance (Star Trek actually).

Thursday morning, I warned my supervisor that I wasn't going to be able to get the work done by Friday morning. She asked when, and I told her it could be the following Tuesday, if I can get the time to work on it (hint, hint). I knew exactly how important this was; apparently she didn't.

Friday rolls around, and I get called into work to work on this problem (and of course nothing gets done that day because they're STILL badgering me). We had to work Saturday, too. The bosses that delayed me from working on the problem were there, too, also looking into the problem from other angles, which I do respect. One had canceled a weekend trip to be there.

As evidence that the work definitely could have been done during business hours, I like to point out that it only took us 8 hours Saturday to do it all...and that was with the server being down for the first 4 hours. Between Wednesday morning and Thursday afternoon, I had about 15 hours of work time in which it could have been done, and a lot of people, including me, would have been much happier.

I got a lecture because I was averse to working that weekend. I calmly explained that the work COULD have easily been done Wednesday and Thursday had I been allowed to actually sit and do some work instead of being constantly dragged into meetings. As far as I know, my bosses got no reprimand for mis-managing their employees, though to me their foul-up was far worse than mine.

It'd be different if the problem had just come up Thursday and we had to figure out what was going on, but the way my bosses handled it leaves a bitter taste in my mouth even now. My manager said I was right, but the lecture continued. Whatever.

2010-02-12 Reply Admin

TRWTF is sendmail.

Amen. And who thought a build system for configuration was a good idea?

lolwtf · 2010-02-12 Reply Admin

Zylon:
The (well okay, not "the", but certainly "a") real WTF is email servers that can't handle such a trivial task as failing gracefully when they run low on disk space.

Yeah, what's with that? Everyone has a story of a mail server critically failing because of a chain reaction set off by one person. A system should not be so easy to bring down.

2010-02-12 Reply Admin

"You want to know when I will finish? You say you can't do ANY work till it's fixed? Good! You're drafted. Your job is stand at that door and prevent anyone from interrupting me."

2010-02-12 Reply Admin

As any good manager knows, if a project is behind schedule, the way to get it back on schedule is to hold a daily progress meeting to discuss why the project is behind schedule.

After all, if the staff can't get the project done working 8 hours a day, it stands to reason that if we require them to spend 1 hour a day in a meeting discussing what they got done the previous day and another 1 hour per day preparing their report for the next day's meeting, leaving only 6 hours per day for actual work, they will clearly get the job done much faster.

2010-02-12 Reply Admin

Okay, this is one of my pet peeves. For no particularly good reason it really grates on me:

"cc" does not stand for "carbon copy". It stands for "copies".

There is a convention in English, admittedly rarely used, that to make a one-letter abbreviation plural you double the letter. Thus the abbreviation for "page" is "p" and the abbreviation for "pages" is "pp". The abbreviation for "line" is "l" and the abbreviation for "lines" is "ll". The abbreviation for "copy" is "c" and the abbreviation for copies is "cc". See, for example, http://www.umc.pitt.edu/styleguide/a.html.

On the other hand, I guess language means what the majority of readers and writers think it means, and so many people now think that "cc" stands for "carbon copy", that maybe that's what it stands for now.

The other grammar issue that unduly annoys me: When an issue is no longer relevant, it is not a "mute point": it is a "moot point". A "mute point" would be one that a person cannot say out loud. </grammar_nazism>

2010-02-12 Reply Admin

Anonymous:
Just because someone else fubar'd the config doesn't mean it isn't the OP's job to ensure it's running properly. It sounds like the company in the OP is fairly small, but still how can you blindly support something without grokking the config?

Sure. And any time a programmer starts work on a new project, the first thing he should do is read all the existing code and insure that he understands every line of it. After all, how can you modify a system if you don't understand the existing code? Then, in his SECOND hour on the project ...

frits · 2010-02-12 Reply Admin

Jay:
Okay, this is one of my pet peeves. For no particularly good reason it really grates on me:
"cc" does not stand for "carbon copy". It stands for "copies".

There is a convention in English, admittedly rarely used, that to make a one-letter abbreviation plural you double the letter. Thus the abbreviation for "page" is "p" and the abbreviation for "pages" is "pp". The abbreviation for "line" is "l" and the abbreviation for "lines" is "ll". The abbreviation for "copy" is "c" and the abbreviation for copies is "cc". See, for example, http://www.umc.pitt.edu/styleguide/a.html.

On the other hand, I guess language means what the majority of readers and writers think it means, and so many people now think that "cc" stands for "carbon copy", that maybe that's what it stands for now.

The other grammar issue that unduly annoys me: When an issue is no longer relevant, it is not a "mute point": it is a "moot point". A "mute point" would be one that a person cannot say out loud. </grammar_nazism>

And here I was thinking it meant "courtesy copy" the whole time.

2010-02-12 Reply Admin

Jay:
Okay, this is one of my pet peeves. For no particularly good reason it really grates on me:
"cc" does not stand for "carbon copy". It stands for "copies".

There is a convention in English, admittedly rarely used, that to make a one-letter abbreviation plural you double the letter. Thus the abbreviation for "page" is "p" and the abbreviation for "pages" is "pp". The abbreviation for "line" is "l" and the abbreviation for "lines" is "ll". The abbreviation for "copy" is "c" and the abbreviation for copies is "cc". See, for example, http://www.umc.pitt.edu/styleguide/a.html.

On the other hand, I guess language means what the majority of readers and writers think it means, and so many people now think that "cc" stands for "carbon copy", that maybe that's what it stands for now.

The other grammar issue that unduly annoys me: When an issue is no longer relevant, it is not a "mute point": it is a "moot point". A "mute point" would be one that a person cannot say out loud. </grammar_nazism>

No, you're 99% wrong on this. (I'll give you the 1% for the perhaps-current interpretation to mean "copies".)

Perhaps cc now means "copies", but its origin is when people would use carbon paper to make a duplicate of a letter in a typewriter. The only reason it means "copies" these days is because "carbon copy" doesn't make sense any longer since we don't use carbon paper. Nevertheless, its origin is in "carbon copy" from days past.

http://www.phrases.org.uk/meanings/carbon-copy.html http://www.kevinlaurence.net/essays/cc.php http://dictionary.reference.com/browse/carbon%20copy

I could do more, but I've got more important things to deal with.

You are 100% correct, however, with "moot point" vs "mute point", and that grates on me as well.

2010-02-12 Reply Admin

The queue directory contained such a preposterous number of files that wildcards such as ? and * could not expand. That meant there was no immediate way to list only the Sendmail "q*"

find . -name 'q*' -exec ls -l {} ;

2010-02-12 Reply Admin

Jay:
And any time a programmer starts work on a new project, the first thing he should do is read all the existing code and ensure that he understands every line of it.

FTFY. Somehow, I can't imagine getting a payout on memory/understanding failure…

2010-02-12 Reply Admin

DysgraphicProgrammer:
"You want to know when I will finish? You say you can't do ANY work till it's fixed? Good! You're drafted. Your job is stand at that door and prevent anyone from interrupting me."

Nice.

2010-02-13 Reply Admin

What gets me really is those people who constantly ask for status updates when you need to take some time alone to actually fix the issue. I've been through that situation lots of times it's exasperating.

Sigh.

2010-02-13 Reply Admin

I just read this comment after I posted mine. Huh. I hate getting dragged into meetings during a critical system issue. Plus the fact that my bosses start asking for updates every second while I'm trying to investigate the cause of the problem. And telling me I should do this, that first before looking into issue.

If there's an issue, my first thought is to get it fixed. Let's get into red tape stuff afterwards.

2010-02-13 Reply Admin

TroZ:
Qyn:
This reminds me of the "Reply All" fiasco at my workplace last year. One of the company-wide email blasts went out for some such or the other. One of the management recipients had a question, and replied to the email. Of course, being management, technology is a mystery, so instead of sending a direct reply, they sent a "reply all" to over 20,000 people. <snip> Needless to say, several heads rolled, and the "reply all" button was stricken from the email client.

That is why company wide emails should be BCCed to all instead of TO all.

Nononono. They should be FCC'd directly to recipients' mailboxes.

2010-02-15 Reply Admin

Jay:
"cc" does not stand for "carbon copy". It stands for "copies".
There is a convention in English, admittedly rarely used, that to make a one-letter abbreviation plural you double the letter. Thus the abbreviation for "page" is "p" and the abbreviation for "pages" is "pp". The abbreviation for "line" is "l" and the abbreviation for "lines" is "ll". The abbreviation for "copy" is "c" and the abbreviation for copies is "cc". See, for example, http://www.umc.pitt.edu/styleguide/a.html.

On the other hand, I guess language means what the majority of readers and writers think it means, and so many people now think that "cc" stands for "carbon copy", that maybe that's what it stands for now.

The other grammar issue that unduly annoys me: When an issue is no longer relevant, it is not a "mute point": it is a "moot point". A "mute point" would be one that a person cannot say out loud. </grammar_nazism>

This is total rubbish, of course it's "carbon copy", the expression has been used for decades (http://en.wikipedia.org/wiki/Carbon_copy). As for a "mute" point I have never heard anyone make that mistake before. "Moot" is a commonly used word in the UK so maybe it's just you yanks that don't quite get it.

Screw you Akismet. I said SCREW YOU AKISMET. LAST TRY, AKISMET, BEFORE I START BUSTING HEADS! Fine, you win, I give up... Oh come on now... This is taking the piss! STOP REFUSING MY COMMENT YOU SHITTY SPAM FILTER! Is this for real? 8 refusals? FINE, I'll take out the 1 lousy URL from my fucking post, fuck you Akismet and all the shitty developers that make you.

2010-02-15 Reply Admin

This isnt a WTF.... this is the reality of sendmail.

2010-02-15 Reply Admin

Anonymous:
This is total rubbish, of course it's "carbon copy", the expression has been used for decades (http://en.wikipedia.org/wiki/Carbon_copy). As for a "mute" point I have never heard anyone make that mistake before. "Moot" is a commonly used word in the UK so maybe it's just you yanks that don't quite get it.

Yeah, I already hammered him on the "carbon copy" issue. But I (yes, a Yank) hear many people referring to a topic being a "mute point" and that bugs the hell out of me too. The other day someone asked for people to identify what was wrong with a sentence that had "should of" (instead of "should've"), which is another issue I see often.

People don't read these days -- they take what they hear and they parrot it. "Should've" sounds like "should of" and so they write it that way without thinking of the underlying contraction. "Moot point" sounds to some like "mute point" and they write it that way because they don't know the word "moot" but they see the word "mute" on their most important piece of electronics.

People just don't seem to give a damn about this fundamental issue: the loss of the well-written word. They dismiss it as being "unimportant." "Hey, you knew what I meant!" It shows a fundamental disregard to detail, a disrespect for the audience, and a personal arrogance that what they have to say is so important they cannot take the time to get it right. It's a symptom, small though it may be, of a larger issue in society.

I don't claim to be perfect. There may be grammatical errors in this post as well. However, I've spent several minutes and several previews going through this to ensure that the most obvious and basic mistakes do not exist. If someone can identify where I've failed, then I deserve to go back to a remedial English class.

2010-02-15 Reply Admin

DES:
Won't work. Try it yourself. Even ls -f probably wouldn't have helped if this was more than ~5 years ago.

Between 1998 and 2002 I worked in a Linux shop, supporting a crufty overextended software package that regularly produced directories with 50,000 to 100,000 files in them. The "ls | grep" trick and its cousin, "ls | grep | xargs", were standard tricks I employed regularly to sift through overburdened directories to find what I was looking for.

I do not know why you think that would not have worked.

2010-02-15 Reply Admin

I know, Exchange will eat it's self without large attachments. Sendmail at least will keep chugging along. It's all about not allowing management set server rules.

His management were idiots, simple as that.

2010-02-16 Reply Admin

nonpartisan:
Jay:
Okay, this is one of my pet peeves. For no particularly good reason it really grates on me:
"cc" does not stand for "carbon copy". It stands for "copies".

There is a convention in English, admittedly rarely used, that to make a one-letter abbreviation plural you double the letter. Thus the abbreviation for "page" is "p" and the abbreviation for "pages" is "pp". The abbreviation for "line" is "l" and the abbreviation for "lines" is "ll". The abbreviation for "copy" is "c" and the abbreviation for copies is "cc". See, for example, http://www.umc.pitt.edu/styleguide/a.html.

On the other hand, I guess language means what the majority of readers and writers think it means, and so many people now think that "cc" stands for "carbon copy", that maybe that's what it stands for now.

The other grammar issue that unduly annoys me: When an issue is no longer relevant, it is not a "mute point": it is a "moot point". A "mute point" would be one that a person cannot say out loud. </grammar_nazism>

No, you're 99% wrong on this. (I'll give you the 1% for the perhaps-current interpretation to mean "copies".)

Perhaps cc now means "copies", but its origin is when people would use carbon paper to make a duplicate of a letter in a typewriter. The only reason it means "copies" these days is because "carbon copy" doesn't make sense any longer since we don't use carbon paper. Nevertheless, its origin is in "carbon copy" from days past.

http://www.phrases.org.uk/meanings/carbon-copy.html http://www.kevinlaurence.net/essays/cc.php http://dictionary.reference.com/browse/carbon%20copy

I could do more, but I've got more important things to deal with.

You are 100% correct, however, with "moot point" vs "mute point", and that grates on me as well.

Actually doubling the one letter abbreviation to make the plural is common in MANY languages. To abbreviate USA in Spanish one says: ee uu.

No it doesn't mean carbon copy, as the abbreviation was in use before "carbon" paper was in common use (hand copies!)

2010-02-16 Reply Admin

Herby:
Actually doubling the one letter abbreviation to make the plural is common in MANY languages. To abbreviate USA in Spanish one says: ee uu.
No it doesn't mean carbon copy, as the abbreviation was in use before "carbon" paper was in common use (hand copies!)

I wasn't arguing against the technique. You are correct that it is common to pluralize an abbreviation in this way. You are not correct, however, that it doesn't mean carbon copy. "pp" means pages; "ff" to mean following pages ("f" meaning the next page). But in today's language, its primary meaning is "carbon copy."

2010-02-17 Reply Admin

Once upon a time the company network admin called me to help with an email loop. Three mail servers were configured to treat the next in the ring as the "smart" forwarding server. Mail message went around the ring until it reached the hop count limit, then bounced back with an error. The error message went around until it reached the hop count, and so on. Problem solved by unplugging the network cable at the right time to pin the message in place on the local server to be removed.

2010-02-18 Reply Admin

Coward:
Mail server today have limits on retry, space used, bounces to bounces and so on, all to prevent this from happening.

It's a one-liner in sendmail config:

define(confPRIVACY_FLAGS',nobodyreturn')dnl

2010-02-24 Reply Admin

Gump:
I call 100% BS

I call 50% dyslexia. Why and how would Exchange prevent the delivery of the "invalid account" message generated by The Todd's server?

2010-02-24 Reply Admin

Ahh, but you forget, users will grow their files to fill all available space.

It doesn't matter if they got by with a 1G quota for years, as soon as it's lifted to 2 gigs, they will fill it. Even if they don't know the limit, they will fill it.

Users are like cockroaches. It's not their fault, it's just who they are.

2010-02-24 Reply Admin

Gee, you're right, Mason. Like most users, I have a 6th sense for how much space my mail server has, and I change the volume of mail I send each day accordingly.

2010-03-10 Reply Admin

No - I've gotten phone calls because a similar "out of office" was killing exchange...

2010-07-17 Reply Admin

Zylon:
The (well okay, not "the", but certainly "a") real WTF is email servers that can't handle such a trivial task as failing gracefully when they run low on disk space.

because management explicitly TOLD the IT how it should and shouldn't handle it... read the thing again. IT: "If a message fails to send, the notification about the failure should not include the entire message (with its large attachments that caused the failures)" Management: "That is not acceptable, it must include the entire message, including attachments".

this is a classic case of MIM, Morons In Management who tell the actual experts how to do their job.

2023-08-07

Comment held for moderation.

2024-06-13 Reply Admin

The Great Cascade

Leave a comment on “The Great Cascade”