• Unomi (unregistered)

    As somebody already stated:

    The WTF is not using a decent testing environment. That means a duplicate of your production environment. Double the price, double the effort, but less mistakes.

    The loss in value of losing a production database-server is much more than the price of a testing environment. Everybody knows that. But they want to take the risk.

    Having a safety net underneath your critical tasks makes you sweat less and concentrate better.

    After tested the whole procedure of patches /service packs, you can print a test result to the deployment department and demand these procedures to be executed.

    What is so difficult? Trying to convince the managers involved.... But that's always a first class WTF.

    • Unomi -
  • Sad B***ard (unregistered) in reply to Rob
    Rob:
    Ah yes. SQL Server service packs.

    We have an SQL Server 2005 server, which was running with SP1 installed since the start. Then we decided to upgrade to SP2. No problems there. Well ok, there were some, but unrelated.

    Anyway, a few days later our network engineer contacted me that the database backup had shrunk quite a bit. So I checked: there was only one backup per database instead of the 7 (1 week) that was supposed to be there.

    So I checked the maintenance plans, and immediately saw the problem: the SP2 upgrade changed all the "remove files older than 1 week" to "remove files older than 1 day". I spent half an hour changing all the maintenance plans. Then, a few days later, the Microsoft Update on my work station showed an update to this problem.

    I still have to install the update on the server itself, and already fear that all will be set to "remove files older than 1 month"...

    I am now going to be sad and give some support - but I think I should just ensure you know this.

    This is a known problem with the original version of SQL Server SP2. The problem is in the updated admin tool (not the server software). It corrupts the cleanup interval of SQL Server 2005 maintenance plans when they are edited. This does not apply to the current version of the patch available since March 05, 2007, and there is a GDR patch which should be applied to all machines running the admin tools which were patched with the original SP2.

    See http://support.microsoft.com/kb/933508/en-us for details.

    On an aside I do apply service packs - but unless they fix a problem I have, I have been known to wait before doing it. Still actually don't have SQL Server SP2 on my machine - but it is always behind NAT firewalls and clients are not on SP2 yet either.

  • Marcin (unregistered)

    Thanks for that salutary tale, Daily What The Just So Stories! Tomorrow, can we have the one about the little coder who wouldn't write unit tests?

  • (cs)

    Wait, I think I used to work for this company. I used to work for a company that didn't like patches being installed.

    Claimed exactly this reason too....patches would break it.

    Though to be fair, I did release a beta (of a commercial application) once that did break once the patch I submitted for glibc was released into the wild. But, to get the beta out the door, I had to make it work with the bug....and it wasn't really my fault that the tester waited 6 months before finally trying to run it.

    I also remember the days where software was only supported on the exact version of some dependent application or operating system....a full QA test cycle was required each time something major got patched.... And, SQL server was one of the dependent apps....though IIS and iPlanet were also big. It took worms that exploited IIS and SQL server to change that thinking.

    The office had gotten shutdown for days due to worms affecting Windows vulnerability (though my machine always made it through, because I kept it patched).....then it became mandatory to update daily, etc.

    Odd how I miss those days....

  • (cs) in reply to s
    s:
    Installing service packs and updates on my home PC would immediately render it unusable. I mean, Windows Genuine Advantage is known to conflict with all windows installs with the serial number I use.
    Interesting. What is that serial number?

    <whisper>Don't worry. I work for Microsoft [snicker]</whisper>

  • (cs) in reply to ParkinT
    ParkinT:
    s:
    Installing service packs and updates on my home PC would immediately render it unusable. I mean, Windows Genuine Advantage is known to conflict with all windows installs with the serial number I use.
    Interesting. What is that serial number?

    <whisper>Don't worry. I work for Microsoft [snicker]</whisper>

    Well there are a series of numbers that MS either blacklisted as stolen or their hash values came out to something similar or something, the details are not really known to me. This made some perfectly legit serial numbers show up as bad. MS has since updated their serial generation to not print numbers that fall into that category, but this does nothing for those people with legitimately purchased copies that are giving a false reason of being stolen simply because they purchased to early before the number check got put in place.

  • Junkman (unregistered) in reply to Sgt. Zim

    I would have agreed, had I not just started working with someone who was EXACTLY like this...

    "Service packs are inherently dangerous; problems might arise!" - is just the kind of thing he would say...

    and this from the same guy who pulled the wrong drive in a two disk array when the other one had failed .

  • andy brons (unregistered) in reply to XML Hater
    XML Hater:
    I have actually seen MS patches fix one security hole only to open up a completely different sometimes-more-serious one.

    I thought that was a WTF in itself.

    A SQL Server Patch and Service Pack are two entirely differnt animals. In the past SQL Server Service pack have caused problems. That really doesn't happen anymore. The only problem I know recently was SQL 2000 SP4 for 64 bit machines. The problem was all over the blogs and SQL Service packs revert gracefully.

    You are by far better of installing them on SQL Server than not.

  • (cs) in reply to tin
    tin:
    mr_ed:
    I was mildly surprised that my Linux install is Genuine Windows.

    I guess it only checks if you aren't genuine then. Perhaps it should be called "non-genuine disadvantage"? Clearly your Linux install isn't a pirated copy of Windows, so that's all they need to know. Stoopid MS.

    that is correct. MS maintains a list of "compromised" product keys. your product is "genuine" if it's not on this list. if your product key becomes compromised, then you could be genuine one day and not the next.

  • (cs) in reply to oldami
    oldami:
    MikeBeer:
    Bean:
    I can definetely defend the "bad service packs" line. To this day, Installshield 10 SP1 will crash if you add a merge module to an installation and try to build it. Any merge module. Any installation.

    I've heard that removing installshield and installing the slipstream IS X + SP1 together works, but we don't have a support contract so we can't get it. Installshield X without SP1 works just fine for us though.

    http://community.installshield.com/showthread.php?t=138263

    Any version of InstallShield with any service pack will crash whenever you try to do anything if you don't perform the proper sacrifices each morning at sunrise. At least that's been my experience.

    Thats why we switched to Nullsoft scriptable install system. It works, and its open source.

    Or to InnoSetup, also works, also open source.

  • (cs) in reply to bramster
    bramster:
    How many copies of Vista are sold because it's XP with all the service packs?
    It's not (otherwise three of my favorites programs wouldn't be broken on Vista).
  • (cs) in reply to KattMan
    KattMan:
    This made some perfectly legit serial numbers show up as bad.
    There is even funnier: we received a nice development PC, but with Windows not yet activated, and most annoying of all, needing a msdn key, not a normal Windows (nor VL) (hell, I didn't even know there were versions to activate with an MSDN key), so after some hurries to renew our long-forgotten msdn subscription, the mighty box arrives.

    Happy like kids to the thought we might finally be able to try our new 4-core computer, we register online, but bam, database error. Naively, we retry but the number is no longuer allowed. We call and end up after 10 minute on some hungarian who hangs up on us as soon as we say "activate".

    ..and as for today, we're still trying to activate that ****.

  • Franz Kafka (unregistered) in reply to Rob
    Rob:
    Ah yes. SQL Server service packs.

    We have an SQL Server 2005 server, which was running with SP1 installed since the start. Then we decided to upgrade to SP2. No problems there. Well ok, there were some, but unrelated.

    Anyway, a few days later our network engineer contacted me that the database backup had shrunk quite a bit. So I checked: there was only one backup per database instead of the 7 (1 week) that was supposed to be there.

    So I checked the maintenance plans, and immediately saw the problem: the SP2 upgrade changed all the "remove files older than 1 week" to "remove files older than 1 day". I spent half an hour changing all the maintenance plans. Then, a few days later, the Microsoft Update on my work station showed an update to this problem.

    I still have to install the update on the server itself, and already fear that all will be set to "remove files older than 1 month"...

    This is why you validate patches on an isolated system before deploying.

  • Thomas (unregistered) in reply to Franz Kafka

    The simple reason for keeping your production servers current on service packs is that your developers will! It is always best to develop against a fully patched system to head off any problems that might be caused by additional security features or to benefit from additional functionality. So the net effect is that the code goes into QA and it breaks and the developers say "install the latest service pack". If QA says (rightly) that they won't until it is install on production, the tell them you'll happily wait on any new builds until they do. Installing a service pack is FAR less expensive than having developers hack around the fact that it is not installed. You don't have to be upto the day, but IMO you shouldn't let it go more than a couple of months.

  • Tomer Gabel (unregistered)

    Happy ending! Rock on!

  • Doesn't Break It? (unregistered)

    We had a client install an update to SQL Server a few years ago (SP2 to SP3 for some SQL Server version) and it most certainly did break things.

    And not in some easy to fix way, either. It broke the application inside some java library from a third party which meant that the application did not work at all.

  • Tbee (unregistered)

    You people want a WTF? How about this one: windows administrators are per default administrators on all databases in a SQLServer.

    So... The guy who build / maintains the building of a bank also has full access to its vault. Microsoft logic. Hence the $ instead of s, I guess.

  • Bobby Brady (unregistered) in reply to AndrewB
    AndrewB:
    Dax:
    As always the development guys think they know everything about everything, when in reality they don't know shit about shit. You realize that you need to stage a deployment, then do testing and QA and THEN you release the service pack on to production. Who do you think is going to be staying late all weekend to rebuild your precious SQL server when the latest service pack breaks something? The developers? I don't think so.

    Everyone always bitches about OPs moving slowly and being over cautious about changes to production, but then when shit hits the fan it always gets turned into "Why on earth did you do THAT"

    Its so fucking simple it deserves its own WTF "If it ain't broke, don't fucking fix it!"

    I think Mr. Irons here just set a new record for bitterness on this site.

    I agree but he does have a point - the real culprit here is the law of averages. A person is smart - people as a collective are usually stupid. Going from one extreme to the other is not a solution and I can plainly see the end results of this story were not the real goal of the person who started the complaints in the first place. The real solution would be to have a sanity check before applying the patch to a production environment - staging anyone? (but you know how some people in power are - to them it is usually a black and white issue - get a bunch of them in a room together and watch the fires start ablazin) - the end result of this story sadly usually flies in an environment where there are too many cooks that cannot smell the shit burning are in the kitchen, which always results in the same outcome, people get burned. In my experience the larger the corporation - the more ignorant and slower the environment.

Leave a comment on “Fixing It Would Break It”

Log In or post as a guest

Replying to comment #:

« Return to Article