- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
So why not just buy a larger hard drive?
Admin
Well, there's your problem right there...
Admin
Wow that's sad... The common motto of the IT department I used to work in when things like this happened was always "It's easier to beg forgiveness than to ask permission"; the server would have been scaled back anyway... Usually turns out that if you set stuff like this up where it works RIGHT, people thank you when it doesn't crash, and don't even notice that their ENTIRE email isn't coming back to them.
My point exactly :D
Addendum (2010-02-11 10:46): I guess I shouldn't say that "people thank you"... people just pretty much ignore it if it doesn't crash, they only notice when it DOESN'T work. However, it looks good on your review to have high uptime on the systems you are administrating.
Admin
One thing all email systems should do is when sending an error email, never ever, under any circumstances send it with the attachments included. Ever!
Admin
This reminds me of the "Reply All" fiasco at my workplace last year. One of the company-wide email blasts went out for some such or the other. One of the management recipients had a question, and replied to the email. Of course, being management, technology is a mystery, so instead of sending a direct reply, they sent a "reply all" to over 20,000 people. Needless to say, there were many confused souls that promptly replied that they should probably not be receiving this reply... and wouldn't you know it THEY all hit "reply all" too. It became the joke of the day to send out a reply company-wide and wait for someone to reply that their mailbox was filling faster than a plate at a Vegas buffet. Under the crushing weight of hundreds of thousands of emails, the servers crashed, and email was out for several hours. Needless to say, several heads rolled, and the "reply all" button was stricken from the email client.
Admin
We had the same thing at my work - the update rolled out and Reply All is grayed out in Outlook. Thank goodness for Ctrl + Shift + R :-)
Admin
Perhaps this is a stupid comment. by why not just strip the attachment out of any returned e-mail
Admin
Things like this happened with distressing frequency when I was in high school. I don't remember what the idiotic client we had for the school email system was called, but the default when you clicked "reply" was "reply to all," not "reply to sender." You had to go out of your way to hunt down the "reply to sender" command if you didn't want every one to see your reply. Needless to say, hillarity ensued, repeatedly.
Admin
Obvios but not stupid. Well obvious to anyone except those in the story.
Admin
This is TRWTF.
Admin
Not too long ago I ended up in an email spiral of death dropping my companies entire email system.
I wasn't able to directly connect my phone to our Exchange server - so I set a rule that forwarded all my emails to my server at home - which I could then connect too. At one point someone up in management sent a meeting request - and they weren't happy to get a notice that it had been forwarded "outside" the company.
I turned outlook rules off and deleted the email account from my server.
Fast forward a few months - I leave for vacation. I set my "out of office" on and leave for a week. A few days later my manager calls me and says that my email account is crashing the entire system - and could I please take a look.
Long story short - when I turned "out of office" on - it turned rules back on. I hadn't deleted the rule - so it reactivated. Whenever someone sent me an email - it automatically forwarded to my servers - which didn't have that account so it sent an "invalid account" message back to my email address - which immediately tried to forward it back to my servers - rinse and repeat several million times.
In the end - I built a small macro to loop through my OWA account and delete all emails and move to the next page. I watched that macro run for 11 hours before finally deleting all of the bogus emails.
Admin
Pardon my saying so, but this is bulls*. A normal Sendmail setup will not behave as described in the article. Either Chris or his predecessor made a complete hash of configuring sendmail, or whoever sent this in made it all up.
Admin
TRWTF is sendmail.
Admin
Was waiting for someone to point that out. Is it so bloody difficult to understand a sentence with a missing word and not whine about it?
Admin
The (well okay, not "the", but certainly "a") real WTF is email servers that can't handle such a trivial task as failing gracefully when they run low on disk space.
Admin
Um, so what's the status now? . . . Ok, how about now? . . . Now? . . Now? . Now? Now?! Now!? Now!?! Now!!
Admin
Admin
He could have listed the q* files with find:
$ find /path -name 'q*'
find doesn't have globbing size restrictions, which are solely an argc/argv limit.
Admin
TRWTF is that the company wide blasts weren't sent via an alias that only a handful of people could send to.
Admin
Admin
Admin
Admin
Admin
The queue directory contained such a preposterous number of files that wildcards such as ? and * could not expand. That meant there was no immediate way to list only the Sendmail "q*" files
Really? I think we can be a bit more creative than that.
$ ls /var/spool/mail | grep ^q $ find /var/spool/mail -name 'q*' $ perl -e 'print join "\n", </var/spool/mail/q*>' $ python -c 'import os; print [x for x in os.listdir("/var/spool/mail") \ if x.startswith("q")]'etc, etc.
Admin
You did not "end up in", but caused that spiral of death ;).
Admin
Kudos to Zylon for having a featured and banned post on the same article.
From Mark - Zylon is my favorite comment troll after all!
Admin
Yeah, well, then TRWTF is why Chris didn't investigate or fix these fundamental errors in his sendmail config.
Admin
Just because someone else fubar'd the config doesn't mean it isn't the OP's job to ensure it's running properly. It sounds like the company in the OP is fairly small, but still how can you blindly support something without grokking the config?
Admin
So it's badly configured, but it's not impossible.
Admin
That is why company wide emails should be BCCed to all instead of TO all.
Admin
RETURN TO SENDER
ERROR: It appears the administrators comment database is full.
Error log will be sent to administrator ...
Admin
I once got in trouble at university for bringing down the entire university's email system (and the separate engineering dept's email system) because my engineering email has set to forward to my main university email acct; I sent an email with an attachment to an engineering acct, it bounced because it was over-quota, the bounce (with attachment) bounced back to me, with attachment, which simultaneously bounced back to the recipient AND forwarded to my other email, which also bounced the full attachment, until there was a feedback loop that brought down two email systems.
I only discovered this the following monday when I couldn't log into my account (Which had been suspended); The admin threatened to make this an academic standing issue but I think he backed down when I pointed out that not bouncing the attachment would have prevented this whole thing.
Admin
I snipped the comments from a recent reply to all fiasco, I find it addictive reading.
STOP ANSWERING TO THIS SPAM FOR GOD'S SAKE!!!!
PLS STOP SENDING IT ...
This is a 5 MB email it will crash our email system.
Sorry for contributing.
Scott
sssssssssss (PLEASE STOP REPLYING)
sssssssssss - sorry, I tried to ignore this but it's not acceptable: each Email has 4-6 MB --> are you all crazy to push "REPLY TO ALL"-button - STOP THIS !!!!!!!!
Sharing with everybody is ultimate communication. One of our goals ?
can we all just stop .. with all these stupid replies... you are all just adding to the problem!!!!!
Zoe,
Please don't 'reply to all' telling everyone not to 'reply to all'...
Is everyone clear on that?
GET ME OFF YOUR LIST
ME TO
Hi - Whats that ???? Can you please stop sending things like that !!!!!! THANKS
Admin
As a sysadmin, I can relate to the "status-seekers" problem. When we have a major problem it will take at least 2 hours before I can even look at the server after going through a cycle of phone calls, office visits, and 15-minute status-update meetings with my boss.
These meetings generally consist of a cycle of my boss asking what the problem is, me saying I don't know cuz I haven't even had a chance to look yet, and my boss repeating the "what's the problem" question with slightly different wording, and me giving the same answer. Rinse and repeat, over and over. Like he thinks if he can just find the magic wording for the same general question then my answer will suddenly be "oh, the problem is blah blah blah, and to fix it I just have to enter this one command and I've really just been waiting for you so we can have a lengthy discussion about entering that command."
We've literally had a server down for over half a day with a fix that only took about 20 minutes because of this sh*t.
Admin
Admin
I gave a letter to the postman, he put it his sack. Bright in early next morning, he brought my letter back.
She wrote upon it: Return to sender, address unknown. No such number, no such zone. We had a quarrel, a lover's spat I write I'm sorry but my letter keeps coming back.
So then I dropped it in the mailbox And sent it special D. Bright in early next morning it came right back to me.
Admin
I call 100% BS
This is either total BS or your company has drastically mis-configured Exchange. Exchange, by default, does not behave this way. You get 4 tries across something like 48 or 72 hours, and the message dies.
Also, Outlook does not behave how you describe. Out of Office Assistant the Outlook Rules are 2 completely different things and have their own rule sets. Rules set under Tools > Rules do not fire or get turned on if off, when you turn on OoOA under Tools > Out of Office Assistant. The OoOA has a complete different set of rules that are only fired when the OoOA is turned on.
Admin
Correct, it will include the entire message in the NDN, IIRC. Not all MTAs do that; I'm pretty sure Postfix doesn't. There's probably a config option to truncate the message that bounced; it might even be the default in recent versions, but this was years ago.
What it will not do, however, is generate NDNs for transient errors. This means that each message will either be delivered once or bounce once. All in all, there should be exactly one copy of each message per recipient on the server: one in each recipient's mailbox for recipients that didn't bounce, one in the queue for each recipient that the server hasn't yet given up on, and one in the sender's mailbox for each recipient it did give up on. With 16 recipients and a 76 MB file, that adds up to 1.2 GB - well short of 80 GB.
Bounce loops should not occur either. The first bounce goes to the sender, the second to the admin (postmaster), and the third is discarded.
Admin
Well played. I was going to take this comment seriously, but you thankfully added the word "trivial" to let us know that you were being sarcastic.
Admin
TRWTF is PHBs who think that everyone prefers a shiny powerpoint slide deck, or even an amply fontified Word doc, over just a plain text message that says "we roll out the new products on tuesday, as listed below...".
This has caused email bloat to no end. An email message does not need to be 10MB! Geez.
Admin
My co-workers tend to get in the way more than my boss does -- while trying to think the problem through, they'll make multiple suggestions on how to fix it but I need to pause and take a minute to think the cause through in order to be sure that the suggestion being proposed (a) is the correct solution, (b) it won't cause any additional excessive downtime, and (c) that I implement the solution in an organized, logical manner. This process normally doesn't take more than a minute or two and ensures that I have clarity (and in fairness to my colleagues, their suggestions are usually right on), but I still need to do it in order to make myself comfortable that I'm not going to make anything worse.
Admin
Admin
"Bounce loops should not occur either. The first bounce goes to the sender, the second to the admin (postmaster), and the third is discarded."
Should is a key word there. I've seen systems set up by clueless sysadmins that would change stuff like that because they are afraid they would lose something. In fact I've seen it quite often. They would set something like send all bounces to the sender and the admin and never discard anything. They take an approach of enable everything, save everything, and notify me about everything with no understanding of what they are doing.
Admin
This reminds me of a studio-wide email from a sysadmin when I worked at a certain large US game company. The message was to tell everyone to go through their email folders to clean out stuff because the server was getting full. The thing was, he attached a company logo that took up almost 1 MB because it was some sort of uncompressed Windows metafile image.
Admin
by default, ls sorts file names before printing them, so at the very least, you have to ls -f
I don't know how long ago this was, but Unix / Linux traditionally handles large directories very poorly. Look up "qmail big todo" for an example of the problem, and "dirhash" for an example of the solution. Very briefly, directory lookups are linear, so the cost of listing the contents of a directory is at best quadratic: you need to scan the directory once to get the names, then once for each entry when you stat it.
Admin
Two words: Nagios monitoring
Admin
Admin
There's a whole list of WTFs here:
Continuing to accept & process mail in a low diskspace situation, as you suggest.
Having a tight quota on the mailbox for the double-bounce recipient, assuming it's monitored and maintained.
Continuing to attempt to deliver bounces when the bounce and double-bounce recipients are unavailable.
Not aborting delivery and deleting the failed messages upon issuing a bounce. (How could a few bounces generate too many files to list? If message X fails, you send message X' and delete message X, you don't generate bounce X' and try to send it in parallel with X.)
But probably the real WTF here ...
Admin
This is why, at my company, we forbid the use of any email clients. You can telnet to port 25, keyboard your own SMTP commands, and say whatever you have that's so bloody important, or you can flake off.
Admin
We had a similar problem occur. The solution was to restrict emailing of certain Exchange distribution lists to users in specific AD groups.
IT also educated the senders to set the reply-to field by going to the message options in Outlook (when composing an email) and checking "have replies sent to" and filling in the recipient. We've had no more Reply All problems since then (whew!).