The Daily WTF: Curious Perversions in Information Technology

2009-01-29 Reply Admin

Once, I thought I saw the light at the end of the tunnel. It turned out to be a guy in a miner's cap with more work for me.

danixdefcon5 · 2009-01-29 Reply Admin

dkf:
To make matters even worse, the agency’s budget changed with political winds, leaving the “less important” support functions (such as IT) with minimal resources. Massimo certainly didn’t expect to find top-of-the-line workstations, but he was astonished to learn what they considered top-of-the-line: a seven-year old Pentium II-350 with 64MB of RAM. Actually, compared with the rest of their technology, those were top-of-the-line.
I call shenanigans! That's far too modern for Large Government Agencies…

They just described the server I actually replaced at a State Government site.

Basically, most of the PCs were not even configured to use it as a DC, and its "big job" consisted in storing an Access database containing biometric data for the timesheet app.

The new server (which was donated by another branch) was running Windows Server 2003, and became the new DC, and received the Access "database". However, the old P2 "server" running NT4 server is still somewhere out there...

2009-01-29 Reply Admin

Zylon:
I have no idea WTF that business with the asterisk was supposed to mean. You put an asterisk at the end of word, that means there's a footnote. Where's the bloody footnote?

The footnote has been delayed due to budget cuts.

2009-01-30 Reply Admin

foo fighter:
S&W's _The Elements of Style_ sez:
"Typographical usage dictates that the comma be inside the marks, though logically it often seems not to belong there.

'The Fish,' 'Poetry,' and 'The Monkeys' are in Marianne Moore's Selected Poems."

Hardly a ringing endorsement of the inside the quotes argument.

The phrase "typographical usage" being significant here. The "commas inside quotes" thing dates back to the days of lead type and the difficulty of properly kerning commas and apostrophes: typesetters used ," instead of ", because the result looked nicer.

Since then, needless to say, lead type has gone the way of flint knives.

C4I_Officer · 2009-01-30 Reply Admin

192MB? My graphics card has more ram than that! My old phone had 64MB. My old semi-retired linux box has 512MB. My new (1yr old) machine has 8GB.
Even this pathetic machine I have at work has 1GB.

Good for you. Unfortunately I have to salvage PC-100 and PC-133 SDRAM sticks from whatever comes in, and those are typically 64-128 MB in size. So I could give more RAM to a single machine, but usually the choice is between giving 384 MB RAM to a single "high end" machine, or setting up three working "word processor" PCs with 128 MB RAM (still more than adequate with XP and non-essential services turned off).

Getting new PC-133 sticks has a very low value/price ratio, and the machines that are "new" enough to be using DDR-1 (again, max 256 MB) are few and far between. DDR-2 machines are usually private machines brought in for software installation. In general, it's all about keeping a stock of working components and complete PCs, there's no telling how many broken incoming machines or sudden requests for a "working PC" you might get on a given day/hour.

2009-01-30 Reply Admin

Exactly. I used to work in a small software team in a mostly non-software company, and we always took our desktop systems out from under corporate IT and ran our own server because it was the only way to meet our unique needs.

Yes, I had to change some things this morning for one of those "unique needs" that are bodgied together. Instead of asking for some competant help there is now some "business critical" homemade database app knocked up in some descendant of VB which can only run if the user has Administrator priveleges - but at least the guy that knocked it up had the foresight to use a lockfile to prevent two people at a time from using this fragile app and write over each other's data. The big WTF is this thing was the deciding factor to migrate some machines from Linux to XP. That means extra costs for office software, antivirus and exceed (expensive X windows for MS Windows). They need exceed because the software they need to run as the primary part of their job has never been ported to MS Windows - so they need to log onto a server to do the work they used to do onh their desktops.

2009-01-30 Reply Admin

Franz Kafka:
m0ffx:
C4I_Officer:
...army corps HQ...Windows XP, cleaning viruses, salvaging data from aging HDs

This worries me. We wonder why our militaries are having such a tough time in Iraq and Afghanistan - maybe this is why.

I doubt it. Our problems in the sandbox are largely due to an idjit CinC fighting 2 wars instead of 1. IT like that sucks, but it doesn't compare to a nasty error like invading Iraq.

Dude, you guys have fought 2 wars at the same time almost 7 decades ago. Iraq is on one end of the Persian Gulf and Afghanistan/Pakistan is on the other end, but back then your wars were on opposite sides of the frakking planet. What happen? All your base are belong to USA? Someone set up USA the bomb?

Although back then there were hardly any computers, surely there must have been some bureaucracy too.

2009-01-30 Reply Admin

Massimo:
...they sent me to shred some light on the Big Mail Relay Project...

The ability to shred light. Now that's what I call a cool super-power!

dkf · 2009-01-30 Reply Admin

IHasYerCheezburger:
Dude, you guys *have* fought 2 wars at the same time almost 7 decades ago. Iraq is on one end of the Persian Gulf and Afghanistan/Pakistan is on the other end, but back then your wars were on opposite sides of the frakking planet. What happen? All your base are belong to USA? Someone set up USA the bomb?
Although back then there were hardly any computers, surely there must have been some bureaucracy too.

The issue is that to fight those wars, the economy had to be substantially restructured. No attempt to do that was done for the wars in Afghanistan and Iraq.

2009-01-30 Reply Admin

I was going to comment, really; however, I have too many other higher priority comment projects this year. We'll see if maybe we can get it in next year.

2009-01-30 Reply Admin

P4 here! (It's a discard from work they let me take home)

Running Ubuntu Server 6.04 with 500GB of software RAID1 storage.

2009-01-30 Reply Admin

<quote>Pretty sure I have a better setup in my home network, after all, my server works off a Pentium III!</quote>

No, either your server is a Pentium III or, at a stretch, your server runs on a Pentium III. And you are probably pretty sure that your home network is better set up.

Please people, learn to write or you'll always be assumed to be thick. Doesn't matter how much Klingon you know or that you've 20 years embedded C experience if you can't make yourself understood.

It's mostly laziness.

halcyon1234 · 2009-01-30 Reply Admin

common sense:
to shake you awake ... you know, like your mum has to every morning.

Yeah, I have to be shaken awake by your mum every morning, too.

2009-01-30 Reply Admin

Most of the “critical condition” severs

Sewers?

2009-01-30 Reply Admin

C4I_Officer:
That doesn't surprise me, as it's actually the same condition I have to work in the army everyday: one of my company's tasks is supporting a whole army corps HQ and many of its units, which usually means re-installing Windows XP, cleaning viruses, salvaging data from aging HDs and sometimes de-soldering bad caps from mobos.

There's a pretty good WTF right there -- running the famously virus-prone Windows XP on military computers. You might as well tar up all all your military secrets and email the file to China's consulate directly, and get it over with.... ;^)

2009-01-30 Reply Admin

How much do I love Italy... only in a place like this all of this mess could have happened.

DaveK · 2009-01-30 Reply Admin

You know, we can keep this up as long as you like. But in the end, I will win and you will lose. This is utterly inevitable, and it all derives from one simple flaw in your strategy:

You are defending the guy who did the WTF and the strategy he adopted that caused the WTF. You are on the wrong side.

Still, let us proceed. As I recall, I was arguing that his ill-judged meddling was the cause of the problem and the result of an unprofessional approach, and you were arguing that it was entirely professional and the only thing wrong with it was that he didn't do enough of it, which you proposed to remedy with more-of-the-same. Pray continue.

Procedural:
DaveK:
Procedural:
DaveK:
Procedural:
Want to install SMTP Services ? If the SMTP port is free (and Exchange is not working), install an API and Registry Spy and figure out how SMTP Services determines that Exchange is present, and force the result to be false by removing the diagnostic causes.
That's exactly the kind of faulty thinking that caused the damage in the first place. It's a fairly high risk kill-or-cure strategy - don't ever tell yourself "This OS install is so badly messed up now that nothing I could do wrong could possibly make it worse", or it will immediately prove just how wrong you can be...

Sorry pal, I think our job involves clarifying what happens in that dark box with the pixies and the fairies and all the magical wizardry. Not give up and hope that it keeps working. Why would people hire us for that ?

It's not working. That's why someone is hired to fix it.

So fix it.

That's exactly the kind of faulty thinking that caused the damage in the first place.
Where did I say "don't fix it"?

Nowhere.

There is a right way and a wrong way to fix it.

The wrong way is to randomly lay about it with an API monitor and attempt to infer what's going on inside a very complex and proprietary undocumented black-box process, hacking out whatever random bits of registry or filing system happen to catch your eye and look like they might be relevant.

The right way is to go RTFM. There are numerous pages on MSDN that give you full step by step instructions on how to grub out the leftover bits of a bollixed exchange server installation. That is the simplest, easiest, correctest, quickest, most non-empirical way to fix it.

The faulty logic was in your "Either do it this way or don't do it at all" false dilemma. I wasn't saying "Don't fix it". I was saying "Don't be a cowboy".

We are, after all, professionals.

Your willingness to accept without questions the KB articles from MSDN is a very dangerous idea,

That's exactly the kind of faulty thinking that caused the damage in the first place.

Where did I say "accept without question"?

Nowhere.

You are really suggesting that because the docs might be wrong, the correct thing to do is not even look at them? Your ideas are interesting to me, but only by way of morbid fascination. Please do not subscribe me to your newsletter. There may or may not be inaccuracies in the docs, but they are still absolutely a source of useful and relevant information from which you can learn.

The faulty logic was in your "Accept without question or disregard entirely" false dilemma. All information in life comes with an implied E&OE warning on it. It is our job as devs to apply intelligent discrimination in order to filter and verify that information through research and testing.

We are, after all, professionals.

Procedural:
especially when it comes to NT 4.0-era installations (besides I challenge you to find a good article from that era, MSDN was atrocious back then).

Here are a few that are directly relevant to the discussion.

How to manually remove Exchange Server 5.5 completely http://support.microsoft.com/default.aspx?scid=kb;en-us;259158
Remove All Leaves Exchange Keys In Registry http://support.microsoft.com/default.aspx?scid=kb;en-us;177766

XADM: Best Practices for Removing an Exchange 5.5 Site from the Organization http://support.microsoft.com/default.aspx?scid=kb;en-us;324340

XADM: Remove All Does Not Remove Setup Key in Registry http://support.microsoft.com/default.aspx?scid=kb;en-us;178975

XADM: Remove All Does Not Remove Everything http://support.microsoft.com/default.aspx?scid=kb;en-us;147373

Those should be enough to root out the remnants and get the SMTP service going.

Procedural:
As a dev you should be more than just a recipe implementer. Being professional requires a questioning mind.

That's exactly the kind of faulty reasoning that caused the damage in the first place.

Where did I say "be a recipe implementer"? Where did I say "don't ask questions?"

Nowhere.

The information gained from reading the docs is just that: information. It is a tool of our trade, one of the basic pieces of equipment we need with which to do our work. Ensuring that you have a fully-equipped toolkit before you begin is just common sense.

The faulty logic was in your "Either read about stuff or find it out empirically" false dilemma. Both are valid and useful sources of information. Ignoring either is pointless, but it's best to start with the background research. It is our job as devs to do our homework before beginning a job.

We are, after all, professionals.

Procedural:
See, my professionalism is much bigger than yours.

That's exactly the kind of faulty reasoning that caused the damage in the first place.

Procedural:
Besides, I don't see what the big risk is in doing those trials in VMWare.

Nice sign-off! There are so many things wrong with this statement that I've decided to offer you a menu. Take your pick from any and/or all of:

A) Yeh, clever. That's gonna work real good on a "Pentium II with 256 MB of RAM running Windows NT", isn't it?

B) So, while you're spending hours imaging the original server to a VM, and you haven't even started on the real job yet. You are a professional - at yak shaving. Meanwhile, I've read the docs, tried the procedure they suggest, found it works, and am getting on with installing the SMTP server, and it's only taken me twenty minutes.

C) So what? It's entirely irrelevant to the discussion that you could do it in a VM, because it's orthogonal to the question of how you actually go about fixing the failure. Regardless of whether you try to fix it on the broken server or in a virtualized instance, you can still do it either my way or your way.

I am, after all, a professional.

C4I_Officer · 2009-01-30 Reply Admin

Jeremy Friesner:
There's a pretty good WTF right there -- running the famously virus-prone Windows XP on military computers. You might as well tar up all all your military secrets and email the file to China's consulate directly, and get it over with.... ;^)

Well, if anything, Windows and Office are the only choice for dummy users that barely have ECDL-level skills, and it would be far-fetched to try and find a substitute for pure office use. In other words there are thousands of civil employees and support staff with pure office duties, who are very happy and already familiar with Windows and Word, can't change that overnight.

As wonderful as it would be installing Linux and OpenOffice for everyone and gradually phasing out windows, there would still be major issues with older DOC and XLS documents that won't open/display properly with OpenOffice and existing workgroups/domains that would need to be ported to SAMBA. Not to mention that adding Linux workstations (not domain controllers or print/file servers) to an existing Windows domain is anything but trivial.

I would also shudder@ having to perform a complete HD format and software re-installation at every hard disk crash, either due to old age or user mishandling e.g. turning the PC off. In windows you can get away with that and usually solve everything with a CHKDSK and at most a repair installation -while keeping accounts, settings and programs intact. With Linux...tough cookie.

2009-01-30 Reply Admin

DaveK:
You know, we can keep this up as long as you like. But in the end, I will win and you will lose. This is utterly inevitable, and it all derives from one simple flaw in your strategy:
You are defending the guy who did the WTF and the strategy he adopted that caused the WTF. You are on the wrong side.

Still, let us proceed. As I recall, I was arguing that his ill-judged meddling was the cause of the problem and the result of an unprofessional approach, and you were arguing that it was entirely professional and the only thing wrong with it was that he didn't do enough of it, which you proposed to remedy with more-of-the-same. Pray continue.

Procedural:
DaveK:
Procedural:
DaveK:
Procedural:
Want to install SMTP Services ? If the SMTP port is free (and Exchange is not working), install an API and Registry Spy and figure out how SMTP Services determines that Exchange is present, and force the result to be false by removing the diagnostic causes.
That's exactly the kind of faulty thinking that caused the damage in the first place. It's a fairly high risk kill-or-cure strategy - don't ever tell yourself "This OS install is so badly messed up now that nothing I could do wrong could possibly make it worse", or it will immediately prove just how wrong you can be...

Sorry pal, I think our job involves clarifying what happens in that dark box with the pixies and the fairies and all the magical wizardry. Not give up and hope that it keeps working. Why would people hire us for that ?

It's not working. That's why someone is hired to fix it.

So fix it.

That's exactly the kind of faulty thinking that caused the damage in the first place.
Where did I say "don't fix it"?

Nowhere.

There is a right way and a wrong way to fix it.

The wrong way is to randomly lay about it with an API monitor and attempt to infer what's going on inside a very complex and proprietary undocumented black-box process, hacking out whatever random bits of registry or filing system happen to catch your eye and look like they might be relevant.

The right way is to go RTFM. There are numerous pages on MSDN that give you full step by step instructions on how to grub out the leftover bits of a bollixed exchange server installation. That is the simplest, easiest, correctest, quickest, most non-empirical way to fix it.

The faulty logic was in your "Either do it this way or don't do it at all" false dilemma. I wasn't saying "Don't fix it". I was saying "Don't be a cowboy".

We are, after all, professionals.

Your willingness to accept without questions the KB articles from MSDN is a very dangerous idea,

That's exactly the kind of faulty thinking that caused the damage in the first place.
Where did I say "accept without question"?

Nowhere.

You are really suggesting that because the docs might be wrong, the correct thing to do is not even look at them? Your ideas are interesting to me, but only by way of morbid fascination. Please do not subscribe me to your newsletter. There may or may not be inaccuracies in the docs, but they are still absolutely a source of useful and relevant information from which you can learn.

The faulty logic was in your "Accept without question or disregard entirely" false dilemma. All information in life comes with an implied E&OE warning on it. It is our job as devs to apply intelligent discrimination in order to filter and verify that information through research and testing.

We are, after all, professionals.

Procedural:
especially when it comes to NT 4.0-era installations (besides I challenge you to find a good article from that era, MSDN was atrocious back then).
Here are a few that are directly relevant to the discussion.
How to manually remove Exchange Server 5.5 completely http://support.microsoft.com/default.aspx?scid=kb;en-us;259158
Remove All Leaves Exchange Keys In Registry http://support.microsoft.com/default.aspx?scid=kb;en-us;177766

XADM: Best Practices for Removing an Exchange 5.5 Site from the Organization http://support.microsoft.com/default.aspx?scid=kb;en-us;324340

XADM: Remove All Does Not Remove Setup Key in Registry http://support.microsoft.com/default.aspx?scid=kb;en-us;178975

XADM: Remove All Does Not Remove Everything http://support.microsoft.com/default.aspx?scid=kb;en-us;147373

Those should be enough to root out the remnants and get the SMTP service going.
Procedural:
As a dev you should be more than just a recipe implementer. Being professional requires a questioning mind.
That's exactly the kind of faulty reasoning that caused the damage in the first place.
Where did I say "be a recipe implementer"? Where did I say "don't ask questions?"

Nowhere.

The information gained from reading the docs is just that: information. It is a tool of our trade, one of the basic pieces of equipment we need with which to do our work. Ensuring that you have a fully-equipped toolkit before you begin is just common sense.

The faulty logic was in your "Either read about stuff or find it out empirically" false dilemma. Both are valid and useful sources of information. Ignoring either is pointless, but it's best to start with the background research. It is our job as devs to do our homework before beginning a job.

We are, after all, professionals.

Procedural:
See, my professionalism is much bigger than yours.
That's exactly the kind of faulty reasoning that caused the damage in the first place.
Procedural:
Besides, I don't see what the big risk is in doing those trials in VMWare.
Nice sign-off! There are so many things wrong with this statement that I've decided to offer you a menu. Take your pick from any and/or all of:
A) Yeh, clever. That's gonna work real good on a "Pentium II with 256 MB of RAM running Windows NT", isn't it?

B) So, while you're spending hours imaging the original server to a VM, and you haven't even started on the real job yet. You are a professional - at yak shaving. Meanwhile, I've read the docs, tried the procedure they suggest, found it works, and am getting on with installing the SMTP server, and it's only taken me twenty minutes.

C) So what? It's entirely irrelevant to the discussion that you could do it in a VM, because it's orthogonal to the question of how you actually go about fixing the failure. Regardless of whether you try to fix it on the broken server or in a virtualized instance, you can still do it either my way or your way.

I am, after all, a professional.

See, this is exactly the kind of flawed thinking that created this mess in the first place.

As I also like making unsubstantiated conclusions and rely on repetition to give them credence, let me add the following as well:

Elvis IS Alive JFK Was Killed By The Mob

and, of course:

Elvis IS Alive JFK Was Killed By The Mob

For all this talk, you propose to do all of this fine, inquisitive, professional work on the production environment, and herald this as a pure example of professionalism, just so you can save yourself from imaging an old server and doing your experimental work on it ? How will you find out whether that KB article failed you ? Common examples from the under-endowed would include: 1) Hide next to the support desk with fingers crossed hoping that the call volume won't suddenly jump (and that the stalked employee won't press charges against that weird stalky unshowered guy); 2) Making live changes, rebooting the server, and "get a feel for it" (you know, look at the GUI, checking that services are "running", etc.); 3) Writing a beautiful memo laying the blame on Microsoft so that they'll keep paying you.

Any way you want to look at it, you will have to interfere with the system to validate your actions before staging and deployment. Your assumption that this can't be done because you can't run a VMWare image on a PII is faulty for many reasons: 1) Yes, it will run; 2) Why is this your only option ? You don't have a laptop or some other device with which you can test things ? You actually want to run the VM on the same machine that you are testing ?

See, this is exactly the kind of flawed thinking that created this mess in the first place.

(much, much bigger)

KenW · 2009-01-30 Reply Admin

Jeremy Friesner:
There's a pretty good WTF right there -- running the famously virus-prone Windows XP on military computers. You might as well tar up all all your military secrets and email the file to China's consulate directly, and get it over with.... ;^)

That's a pretty good WTF right there... Senselessly and mindlessly bashing Windows when you obviously know nothing about the subject. Just proves you're computer illiterate and trying to pretend otherwise.

FYI: XP run as a non-administrative user is about as secure from viruses as a Linux box running as non-root. (Not that there are many viruses written for Linux - most Linux desktop users are too well educated to run as root, and virus writers won't waste their time targeting such a minor user base as Linux on the desktop anyway.)

jo42 · 2009-01-30 Reply Admin

Massimo:
...for this agency...

This "agency" wouldn't happen to be The Whitehouse, would it?

White House Tech More Tired Than Wired http://blog.wired.com/gadgets/2009/01/wired-or-tired.html

Massimo · 2009-01-30 Reply Admin

Lex:
How much do I love Italy... only in a place like this all of this mess could have happened.

Oh yes, Italian WTFs are truly the vast majority we talk about here...

Massimo · 2009-01-30 Reply Admin

jo42:
Massimo:
...for this agency...
This "agency" wouldn't happen to be The Whitehouse, would it?

We don't have one here ;-)

pink_fairy · 2009-01-30 Reply Admin

Massimo:
Lex:
How much do I love Italy... only in a place like this all of this mess could have happened.

Oh yes, Italian WTFs are truly the vast majority we talk about here...

Good Lord, man, it's not like there aren't plenty of them; you might have one or two yourself. Generally, they come with either the label "Berlusconi" (or "Andretti" for Win95 cognoscenti) or "Napoli."

Nice pizzas -- oops, that would be part of Lost Savoy, (c) 1866, wouldn't it -- but let's be honest. We deal in real WTFs round these parts. Not penny-ante megalomaniac fascist loons or gun-toting drug-baron gibbons.

(Posted after a discussion with my rather depressed and oldest friend, who's 1/4 Roman, 1/4 Norman Sicilian, and 1/2 everything else.)

m0ffx · 2009-01-30 Reply Admin

dkf:
IHasYerCheezburger:
Dude, you guys *have* fought 2 wars at the same time almost 7 decades ago. Iraq is on one end of the Persian Gulf and Afghanistan/Pakistan is on the other end, but back then your wars were on opposite sides of the frakking planet. What happen? All your base are belong to USA? Someone set up USA the bomb?
Although back then there were hardly any computers, surely there must have been some bureaucracy too.
The issue is that to fight those wars, the economy had to be substantially restructured. No attempt to do that was done for the wars in Afghanistan and Iraq.

Personally I suspect a major part of it is that in WWII pretty much all belligerents were not bothered about killing enemy civilians, and not too concerned about the deaths of a few hundred of their own soldiers.

Such a total war of course had to be abandoned with the advent of nuclear weapons. If we'd fought Iraq like we'd fought WWII, but with today's weapons, we would have 'won' - but we'd have glassed pretty much the whole Middle East.

2009-01-30 Reply Admin

I doubt they'l be upgrading in 2009...financial crisis and all!

2009-01-30 Reply Admin

That installation logic of having Exchange installed so you need to remove it but you can't because its not really there is what I hate about Windows servers.

I would love an option to just install period. Each component knows how to configure itself, perhaps remove old bits of itself - whatever - just do it.

I know that such an approach might make matters worse but in some cases I just want a fresh install of Exchange or IIS and the data be damned....

2009-01-30 Reply Admin

Cyberwizzard:
That installation logic of having Exchange installed so you need to remove it but you can't because its not really there is what I hate about Windows servers.
I would love an option to just install period. Each component knows how to configure itself, perhaps remove old bits of itself - whatever - just do it.

I know that such an approach might make matters worse but in some cases I just want a fresh install of Exchange or IIS and the data be damned....

Spoken as someone who's never seen that message come from a *nix box. It's not the OS, it's overly arrogant software developers leaving crud all over the system, hooking much deeper than they need to. Just look at Oracle. (And sometimes, OS developers not creating APIs so those hooks don't have to be used. Vicious circle.)

2009-01-31 Reply Admin

m0ffx:
C4I_Officer:
...army corps HQ...Windows XP, cleaning viruses, salvaging data from aging HDs

This worries me. We wonder why our militaries are having such a tough time in Iraq and Afghanistan - maybe this is why.

There's nothing wrong with old hardware - but it should be old RELIABLE hardware. And there's not too much wrong with Windows XP per se, but if it's getting viruses, then the users and admins are doing something wrong.

In the army, "Fatal Error" really could mean that. They should sort out their IT.

You should see "Windows for Warships" if you want scary!

2009-01-31 Reply Admin

Cyberwizzard:
That installation logic of having Exchange installed so you need to remove it but you can't because its not really there is what I hate about Windows servers.
I would love an option to just install period. Each component knows how to configure itself, perhaps remove old bits of itself - whatever - just do it.

I know that such an approach might make matters worse but in some cases I just want a fresh install of Exchange or IIS and the data be damned....

Then you're in luck! A lot of things have happened since NT4. With Windows, you can use Windows Installer packages. With linux, you can use package managers like RPM or APT. You can even "virtualize" servers now. Crazy times we live in, eh?

Severity One · 2009-02-02 Reply Admin

my name is missing:
Yet the government (assuming US here) has $$$$ to send Billion$ and Billion$ to banks.

Given that he's called Massimo, I'd hazard a guess that we're talking about Italy. Mind that in Italy, bureaucracy has been raised to an art.

A very, very dark art.

Peter

Kuba · 2009-02-02 Reply Admin

C4I_Officer:
I would also shudder@ having to perform a complete HD format and software re-installation at every hard disk crash, either due to old age or user mishandling e.g. turning the PC off. In windows you can get away with that and usually solve everything with a CHKDSK and at most a repair installation -while keeping accounts, settings and programs intact. With Linux...tough cookie.

What a load of FUD.

Any sanely configured Linux deployment should survive hard power-off on desktops without any problems. Writable data should be mounted with full journaling (both data and metadata) -- that way it's pretty foulproof, and there's no chkdsk ever to be done.

If it's deployed properly, then /etc and /usr should be on a partition mounted read-only and perhaps rsync'd (or unison'd) from a central location on startup. Or at least, if the infrastructure is too crappy to support such network load, it should incrementally check signatures at startup (say 2-3% of partition on each boot-up) so that any bitrot can be detected and manually fixed by rsyncing the image from your blessed media.

You then use say unionfs to put a read-write mount on top of the read only image. As for "keeping accounts", everything is mostly in /home, /etc/passwd, /etc/shadow and /etc/group. Settings usually sit in /etc, some do in /var but that's easy to find out. rpm allows you to list all configuration files, and it's easy to use it to list all non-package-managed files in "system" locations too. This is much harder to do on a Windows systems simply because Microsoft has no established (as in a decade+ old) way of listing and checking integrity of system files.

A repair installation on an rpm-based linux is pretty trivial if your distro comes with a live CD: you simply copy over all non-configuration files, as well as restore permissions and reset the package manager's database (/var/lib/rpm). This is something you'd do on your one-off home machine, though, If the deployment is big enough, all of this should be automated anyway -- you should have a rescue disk which will restore everything automagically.

For a large deployment on crappy hardware with minimal "supporting infrastructure" (a bunch of 10mbit hubs :), I'd suggest also doing distributed backup, where a fraction of each machine's storage is used to store compressed backup images of other machines. Since the only thing needing backup is variable configuration/user data, this should be fairly efficient. System files etc. would not need backup, since you should have a bunch of dvds/memory sticks with the "blessed" image floating around anyway ;) \

The backup scripts need to randomly chose the time of day to move backup data around, and there should be realtime-like guarantees as to the traffic shape (packet sizes, maximum throughput) so that the "network" is not brought down. That's fairly easy to do, you only need to think about it first. If everything is set up so that say no more than 15% of bandwidth is used for backup traffic, and the packets are always large (1400+ bytes), this should work just fine. Of course not everything would be backed up every day, but still if there's enough capacity on average, things should work fine.

I've been running linux on pretty low-end hardware and crappy network for a good while, and it was just fine. Heck, even had a web server on a 486/66 machine for a while. It was more than enough to keep the 128kbit DSL (IIRC a finnish HIS system) saturated, while doing squid caching for internal users, too. I think it had 32MB of RAM, and ran a custom 2.0 kernel (it was a while ago).

Cheers!

Severity One · 2009-02-02 Reply Admin

Now I've been working with various Unix flavours (Solaris, *BSD, SVr4, Ultrix and what have you) since 1990, and I get lost after only a paragraph and a half.It's just too complex.

If your personnel costs are more or less the license costs of Linux (in other words, zero), such an approach would be cost-effective. In all other cases, it wouldn't.

Now I wouldn't run a serious deployment (whether a web server, an application server or a database) on a Windows machine if my life depended on it. If nothing else, the stupid licensing so that only a few people can be logged in to Windows Server 2003, or you have to pay a lot of money. But to assume that Linux can replace Windows in an office environment is ignorant at best.

Yes, it can be made to work, and at lower capital expenditure than a Windows installation, but the additional operational expenditure will easily nullify that, and more. It will always be a bespoke solution, that needs to be documented really well, and it will still require more time to maintain.

2009-02-02 Reply Admin

Procedural:
I think Massimo gave up waaaaay too easily here.

Didn't look like Massimo gave up did it?? I thought he presented his findings and the agency put the project off.

...maybe I'm just reading too deeply into this one.

DaveK · 2009-02-03 Reply Admin

Procedural:
See, this is exactly the kind of flawed thinking that created this mess in the first place.
As I also like making unsubstantiated conclusions and rely on repetition to give them credence

I agree. Your flawed thinking is entirely because you like making unsubstantiated conclusions and relying on repetition to give them credence. Fortunately I am here to help you, by schooling you from your ill-trained ways.

Procedural:
Elvis IS Alive JFK Was Killed By The Mob
and, of course:

Elvis IS Alive JFK Was Killed By The Mob

It would of course be an ad hominem of me to claim that because you make blatantly false claims, this implies *all* claims you make are blatantly false. So I won't do that.

Procedural:
For all this talk, you propose to do all of this fine, inquisitive, professional work on the production environment, and herald this as a pure example of professionalism, just so you can save yourself from imaging an old server and doing your experimental work on it ?

Oh dear, you so haven't been reading what I actually said. No, I do not for one moment propose to "do all of this fine, inquisitive, professional work" on the production environment. I propose to do it on my own workstation. By firing up a browser and surfing MSDN. Remember, we're only disagreeing about whether you should RTFM and get the relevant information at your fingertips before you start or not, not what you should do after that.

Procedural:
How will you find out whether that KB article failed you ?

The exact same way you would find out if your random guesswork failed you, only with some extra information to base my inferences on. You are arguing that it's a bad thing to RTFM first, remember, not about what we do after that.

Procedural:
Common examples from the under-endowed would include:

I'll defer to your obviously superior knowledge of what the under-endowed would do.

Procedural:
1) Hide next to the support desk with fingers crossed hoping that the call volume won't suddenly jump (and that the stalked employee won't press charges against that weird stalky unshowered guy);

Nah, they won't, I told them you don't mean any harm really.

Procedural:
2) Making live changes, rebooting the server, and "get a feel for it" (you know, look at the GUI, checking that services are "running", etc.);

*koff*

Excuse me.

That is your plan. Of course I'm not going to defend it.

It was your idea to just load up an API monitor. You didn't bother to explain in detail what you were going to do with that; you've just been asserting ever since that it would be some kind of panacea. But what exactly were you proposing to do with it? Let me remind you: you're going to look at process activities, at registry accesses in the HKLM/CCS/Services subtree, and at a whole bunch of similar stuff, none of which is actually labelled "ThisIsTheThingThatMakesIISThinkExchangeIsStillInstalled."

In other words, it is YOUR plan to just somehow magically by intuition "get a feel for it", and it is MY plan to go and RTFM and get some actual factual information on which to base my decisions, rather than just assuming it'll all somehow work out.

Procedural:
3) Writing a beautiful memo laying the blame on Microsoft so that they'll keep paying you.

I propose to earn my keep by *knowing* what I'm doing for my employer. You assume that you are so valuable that your employer should he happy to pay you to sit around playing with your favourite toys all day to try and solve some abstract logic puzzle for your own sheer personal self-gratification. Which one of us is really trying to get paid for nothing here?

Procedural:
Any way you want to look at it, you will have to interfere with the system to validate your actions before staging and deployment.

As I already pointed out, but apparently too subtly: that goes just as much for what you're saying as it does for what I'm saying. Whether you VM image the system and attempt your fix there before trying it on the live system or not, you can *still* choose either my method or yours to attempt on either the live or imaged system, as appropriate. This is not an argument for not RTFMing first.

Procedural:
Your assumption that this can't be done because you can't run a VMWare image on a PII is faulty for many reasons: 1) Yes, it will run;

Oh dear. You have totally proved the superiority of the RTFM-first approach I am advocating. Here's what the VMWare installation manual says re: host requirements -

rtfm:
ESX Server 3 Requirements ESX Server requires a computer with the following specifications: ? At least two processors of one of the following types: ? 1500MHz Intel Xeon and later, or AMD Opteron (32-bit mode) ? 1500MHz Intel Viiv or AMD A64 x2 dual-core processors ? 1GB RAM minimum ? One or more Ethernet controllers ? Direct attached or networked storage devices with unpartitioned space

So, you wasted time downloading VMWare and attempting to install it to a machine that's years too old to be anywhere near capable of it, and now you're back at square one with no idea what to do next. Not a winning strategy IMO.

Procedural:
2) Why is this your only option ? You don't have a laptop or some other device with which you can test things ? You actually want to run the VM on the same machine that you are testing ?

You're talking like you've had a sudden attack of amnesia. Have you forgotten that we're discussing a post on TDWTF? That some guy at the place where Massimo was working had broken a bunch of machines by attempting to half-heartedly uninstall Exchange in such a way that IIS couldn't install its SMTP server? Yes? Good, you remember that. Have you forgotten Massimo's description of the place? You know, the whole thing about how all the machines are old P2s with no RAM? How the network is held together by baked bean tins and string? How there's NO BUDGET FOR ANYTHING AT ALL? And yet you come out with this grandiose plan, which requires 1) not doing the background research first, 2) hardware and budgetary requirements that far exceed what was remotely plausible to obtain in the circumstances, 3) a great deal of sheer luck and guesswork. *Why* is this a better idea?

Procedural:
See, this is exactly the kind of flawed thinking that created this mess in the first place.

Yes, it sure is. Only with bigger flaws in the logic.

Procedural:
(much, much bigger)

Absolutely; I couldn't agree more.

DaveK · 2009-02-03 Reply Admin

Pardon, should just clarify this:

DaveK:
Procedural:
2) Making live changes, rebooting the server, and "get a feel for it" (you know, look at the GUI, checking that services are "running", etc.);
*koff*
Excuse me.

That is your plan. Of course I'm not going to defend it.

It was your idea to just load up an API monitor.

I forgot to mention one thing here. I should have said

DaveK:
It was your *only* idea to just load up an API monitor and guess what to do.

You didn't even pull this whole "in a VM" qualifier out of thin air until well into the thread, and that is why it is *particularly* relevant that both my fix *and* yours could separately and EQUALLY be tested or implemented either inside or outside a VM. You can't go making up new conditions on the discussion half-way through. (Unless you're emulating that earlier TWDTF story about the job interviewer who kept on coming out with ever more and more ludicrously implausible "but suppose that didn't work" objections to every single idea the interviewee suggested).

C4I_Officer · 2009-02-05 Reply Admin

Kuba:
What a load of FUD.

OK, do you really consider any of that stuff to be aasier/simpler/more foolproof than just having to do a CHKDSK/repair installation? Come on, let's get serious.

Kuba:
<Stuff about image-based networked backups>

While valid, this approach is just impractical: it would work if the bitty boxes were somewhat standardized and used exactly the same OS with exactly the same settings, on exactly the same hardware, and that extended to ALL boxes in the whole jurisdiction of the army corps, every unit, every office..

Unfortunately we have to deal with machines belonging to different domains, workgroups, or even stand-alone machines coming from an office in some distant unit, set up by who-knows-who in who-knows-when-and-where. The hardware can also vary wildly, and they are not even using the same version of windows. What recovery image am I supposed to use then?

Keeping a working backup of each and every machine would pretty much mean keeping the very least a specialized slipstreamed version of windows for EVERY possible configuration (for machines not storing data locally).

In front of this incredible diversity of configurations and hardware, "restoring" a PC should not, actually, it MUST NOT be any more complex than running CHKDSK, performing a repair installation or at most running some generic disk fixing utilities and being able to boot again. What you described is just mind-blowing, it can only applied in a carefully planned environment.

BTW, does any Linux distro still install with journaling off by default, as they did in the past?

Addendum (2009-02-05 18:55):

Kuba:
What a load of FUD.

OK, do you really consider any of that stuff to be easier/simpler/more foolproof than just having to do a CHKDSK/repair installation? Come on, let's get serious.

Kuba:
<Stuff about image-based networked backups>

While valid, this approach is just impractical: it would work if the bitty boxes were somewhat standardized and used exactly the same OS with exactly the same settings, on exactly the same hardware, and that extended to ALL boxes in the whole jurisdiction of the army corps, every unit, every office..

Unfortunately we have to deal with machines belonging to different domains, workgroups, or even stand-alone machines coming from an office in some distant unit, set up by who-knows-who in who-knows-when-and-where. The hardware can also vary wildly, and they are not even using the same version of windows. What recovery image am I supposed to use then?

Keeping a working backup of each and every machine would pretty much mean keeping the very least a specialized slipstreamed version of windows for EVERY possible configuration (for machines not storing data locally).

In front of this incredible diversity of configurations and hardware, "restoring" a PC should not, actually, it MUST NOT be any more complex than running CHKDSK, performing a repair installation or at most running some generic disk fixing utilities and being able to boot again. What you described is just mind-blowing, it can only applied in a carefully planned environment.

Speaking of which, there's the need to teach to newbie "grunts" how to perform low-level maintenance on the incoming broken machines. Restoring a Windows installation is ultra-simple compared to having to become a Linux guru.

Remember, in the Army you can only work with what you are issued and with whatever personnel you are assigned. I can't rely on being assigned Linux-savvy assistants (which will eventually finish their draft or be reassigned somewhere else), while I can reasonably teach any monkey to perform certain low-level tasks in a Windows environment, and succeed at that.

BTW, does any Linux distro still install with journaling off by default, as they did in the past?

2017-10-24 Reply Admin

the first part of this story reminded me of a quote from the Diskworld book "thief of time": "any big project requires organization, but when an organization has been around for a long time, many members lose sight of the project and start thinking that organizing the organization ITSELF is ALL that matters."

2018-08-14 Reply Admin

oh, that part about the "Exchanged!" servers reminded me of a silly story: some employees had been playing a game called "command and conquer" at work. when a manager found out, instead of properly uninstalling the game, she deleted every folder and file named "command", including an important windows file called "command.com"...

Critical Condition

Leave a comment on “Critical Condition”