• (cs)

    Ouch. We keep more production stuff on Dev servers than we should, but those are every bit as secure and backed up as the production ones. Frankly, I'm surprised that in the year and a half they used this unholy setup, no one had to ask for anything to be restored...<font size="2"></font>

  • Dazed (unregistered)

    Wow! I was pretty horrified once to find that a team of seven people from an expensive consulting company hadn't done a back-up for seven weeks. But this knocks that into a cocked hat.

  • (cs)

    It seems there was a lack of "expert" communication from the PiMP. Another one bites the dust for being a complete dimwit. I bet this PiMP didn't even know how to use anything other than this "expert team collaboration software".

  • DEVY (unregistered)

    <FONT face=Arial size=1>HAHA, </FONT>

    <FONT face=Arial size=1>We had something similar..</FONT>

    <FONT face=Arial size=1>We have a development server that we now have to back up as some one decided to put several production sites on it... This also means that we cant reboot it etc... basically its no longer a dev server.</FONT>

    <FONT face=Arial size=1></FONT> 

    <FONT face=Arial size=1>david.</FONT>

    <FONT face=Arial size=1></FONT> 

    <FONT face=Arial size=1>PS... Im not second!</FONT>

  • (cs) in reply to Dazed

    I love it when people ask me for files - uselesss to me - they emailed me months ago for whatever reason - "hey, my HDD got wiped so I need those files." Like I'm still keeping them around for you...

  • Lazy to register (unregistered)

    First to wonder: who is the WTFer here????  The executives who hired the Expert?  The expert who turned a demonstration into the bakbone of the company's biggest project?  The executives who didn't realize this?? The policy of no backup on dev temp servers???

    OT: captcha: "bedtime"  I wish!

  • DEVY (unregistered) in reply to DEVY
    Anonymous:

    <FONT face=Arial size=1>PS... Im not second!</FONT>

    Guess I ook far to long writing that Post!

  • Demaestro (unregistered)

    4 letters.... O...U...C....H....

    To be fair to the PM though he did ask for a deticated server originally and they weren't up for shelling it out. I would say that for a project of this size that you give the PM the tools he requests to get the job done.

    This is like like hiring a pro electrition for doing a huge buildings rewiring but not giving him a tool to test if wires are live then going..... WTF how did he electricute himself?

    Give your workers the tools they need. And listen to your staff when they tell you something is wrong. Getting the software and hardware for the PM would have solved this. Listening to the developer's concerns also would have helped.

    Save a penny lose a dollar.

  • (cs)

    This reminds me of something that happened to me in college!

    It was my final semester and my final project was coming along very nicely.  The guy next to me was creating a sort of email type thing,  I don't exactly remember. 

    Well he didn't secure it.  Some spammers got ahold of the script and sent out mass amounts of SPAM email.  The ISP shut the web server down.  After a few weeks, the server was finally online again.  The guy never secured his code, so of course it went down again.  Eventually my teacher decided to take the server and move it into the classroom where we could at least work on our projects locally. 

    We moved the server in the classroom, hooked it up, turned it on, and nothing happened.  All the data was corrupted somehow, everybody lost everything they ever created for the past 4 years.  I don't remember but I think we were able to recover some of our work.

    in the end, the school payed me back my tuition.   But I still have to live with the fact that I wasted 4 years of my life because of some idiot who doesnt know how to secure a PHP mailer.

  • Dazed (unregistered) in reply to Lazy to register
    Anonymous:
    First to wonder: who is the WTFer here????  The executives who hired the Expert?  The expert who turned a demonstration into the bakbone of the company's biggest project?  The executives who didn't realize this?? The policy of no backup on dev temp servers???

    The Network Operations people are also decidedly guilty. Even an unbacked-up dev server might have a few days work of one or two people on it, and shouldn't be junked without checking. The briefest check would have shown a lot of recently modified files.

  • (cs)

    Not surprised. Is anyone else? Some big hot-shot manager gets brought in, and recommends some software they are accustomed to, but when it gets shot down, they use it anyway. Of course, since they are all big and hot-shotty, they eventually force everyone else to use it through their own determination.  The system thus ends up being used even though it's a demo system, and there are no backups or anything of the like, and IT of couse knows nothing about the fact that a demo system is being used for a production system. Finally, the whole thing blows up in their face. Frankly, I'm surprised that it took the system over a year and a half to do so.

    Wow... all those words and they don't really even comment, just summarize...

    I guess the "true WTF" would be that the hotshot failed to mention to IT that a demo server was being used for what were production purposes. I'm sure there are other WTFs.... I know there are.... But that one is probably the biggest.

    sigh

    Alex: I want my code WTFs back. I don't care if you #&@% up the code conversions anymore. Just don't change them after the original posting in the thread (even if there are NUMEROUS LARGE ERRORS). If your WTFs are what most ppl are commenting on, oh well. Just BRING THEM BACK!!!

  • (cs) in reply to cronthenoob
    cronthenoob:

    in the end, the school payed me back my tuition.   But I still have to live with the fact that I wasted 4 years of my life because of some idiot who doesnt know how to secure a PHP mailer.


    In your case, I would rather put the blame on the non-existing backups than on the lad who is unable to write secure PHP code. Why did it run on the same server anyway?
  • (cs) in reply to ammoQ
    ammoQ:
    cronthenoob:

    in the end, the school payed me back my tuition.   But I still have to live with the fact that I wasted 4 years of my life because of some idiot who doesnt know how to secure a PHP mailer.


    In your case, I would rather put the blame on the non-existing backups than on the lad who is unable to write secure PHP code. Why did it run on the same server anyway?


    It was a web programming course.
  • (cs)

    They did not want to promote it to a "production" server because then they would have been required to buy the license for the "demo" collaboration software.  I wonder if the software vendor has tried to collect 18 months of back licensing fees?  Anonymous emailers are great for reporting this type of theft.

  • (cs)

    So let me get this straight. The PiMP had the Company's  €2.3 Million front to back integration plan on a dev server running expensive, tough unlicensed software. I think the IT guys did the company a favor. Wiping that box before an audit probably saved some jobs. That's the WTF here.

    I hate it when outside contractors install software on my equipment without checking it out. This chucklehead decided to run the project on demo software. I only wish it had been a time-bombed copy that died just days before go-live. That would really make me smile.

    Can't blame the server-guys for this one. We don't have the tape, nor the bacdwidth to backup dev servers. That's why we run CMS and back that up. If anything dies in dev, just reload from Source Safe.

    Just my three cents.

  • (cs)

    Oh man. . . I got a sinking feeling in my stomach just reading the last couple of paragraphs.  I can't even imagine how the Expert Project Manager felt.

    And Goat Cheez:  chill.  This is as good as any code wtf.  I think Alex puts up the best of what is submitted, so if you want code wtfs, submit some good ones!

    Bean bag girl is back, and she's baaaad.  Gentlemen prefer blondes.

  • Backed up against the wall (unregistered)

    For the team mentioned in the WTF, can anyone say 'BaseCampHQ dot com' ?

    (I don't want to get in trouble for looking like I am linking to a commercial website)

    For the poor guy who lost 4 YEARS of work- can you say USB Flash drive?

    (Sorry, no sympathy here for that one)

  • (cs) in reply to Backed up against the wall
    Anonymous:
    For the team mentioned in the WTF, can anyone say 'BaseCampHQ dot com' ?

    (I don't want to get in trouble for looking like I am linking to a commercial website)

    For the poor guy who lost 4 YEARS of work- can you say USB Flash drive?

    (Sorry, no sympathy here for that one)


    We were just naive kids with an ignorant teacher. 
  • ewhac (unregistered) in reply to EJCorcoran
    We don't have the tape, nor the bacdwidth to backup dev servers. That's why we run CMS and back that up. If anything dies in dev, just reload from Source Safe. [emphasis mine]

    You're fired.

    Ensign, get this repository transferred to Perforce...

  • Charles Perreault (unregistered) in reply to Dazed
    Anonymous:
    Anonymous:
    First to wonder: who is the WTFer here????  The executives who hired the Expert?  The expert who turned a demonstration into the bakbone of the company's biggest project?  The executives who didn't realize this?? The policy of no backup on dev temp servers???
    The Network Operations people are also decidedly guilty. Even an unbacked-up dev server might have a few days work of one or two people on it, and shouldn't be junked without checking. The briefest check would have shown a lot of recently modified files.


    I totally agree.  I work in a not-WTF university.  There isn't any server or workstation that is wiped without first being ghosted.  The ghosts are kept at least 30 days and then deleted if nobody complained.  Even if there does not seem to be any new/recently modified file on a server, it's actual internal state (configuration, registry, software versions) is precious.  I remember having to restore a dev server only to check the IIS configuration because all guys working on the project were fired.

    Still... 1 year without any backup is somewhat evil.
  • Dazed (unregistered) in reply to EJCorcoran
    EJCorcoran:
    Can't blame the server-guys for this one. We don't have the tape, nor the bacdwidth to backup dev servers.

    Oh, they surely bear a portion of the blame (though of course a much smaller part than the PM). If they made it clear there were no back-ups and there had been a disk crash, then it would indeed not have been their fault. But if the server was powered up and users had accounts on it, then wiping it without any sort of check was negligent. If they were convinced the server was unused they should have powered it off, or disabled all log-ons, months earlier.

  • kuroshin (unregistered) in reply to ewhac
    Anonymous:
    We don't have the tape, nor the bacdwidth to backup dev servers. That's why we run CMS and back that up. If anything dies in dev, just reload from Source Safe. [emphasis mine]

    You're fired.

    Ensign, get this repository transferred to Perforce...

    Atleast he didnt say that Source Safe was a real version control system.

    Thank God for that.

  • (cs) in reply to Charles Perreault

    And this, my friends, is why the tried-and-true method of printing everything out to build cubicle wall partitions can also be a valuable activity for the company.

  • (cs) in reply to cronthenoob
    cronthenoob:
    <snip>

    in the end, the school payed me back my tuition.   But I still have to live with the fact that I wasted 4 years of my life because of some idiot who doesnt know how to secure a PHP mailer.


    If the school had been remotely good they would have been paying enough attention to students and their projects to be able to assign a grade anyway. In the case of an external exam board said exam board would also probably do the same thing.
  • He Sed Awk (unregistered) in reply to cronthenoob

    Wait a minute --- you had 4_years worth of work on a server and had no_backups  ... of ANY KIND?

    Sounds like a major wtf right there ... I still have code that I wrote for projects in school from 10 years ago - either electronically or hardcopy.

  • 8bitwizard (unregistered)

    Wow.  Unlicensed software (it was a "demo" because management hadn't authorized payment), dev server, a whole year, and even with Alex's anonymization, presumably it did in fact have a devbox machine name.  That's some pretty harsh karma for software piracy.

    There's this guy I've heard about who can tell you all about karma.  His name is Earl.

  • (cs) in reply to snoofle
    snoofle:
    And this, my friends, is why the tried-and-true method of printing everything out to build cubicle wall partitions can also be a valuable activity for the company.


    Yea, especially when it's binary (read compilers and build environments; I version them as well).
  • (cs) in reply to m0ffx
    m0ffx:


    If the school had been remotely good they would have been paying enough attention to students and their projects to be able to assign a grade anyway. In the case of an external exam board said exam board would also probably do the same thing.


    The program was relativly new to the school, and I think after that fiasco, they completely dumped it.  So basically I should have done more research, because that school was horrible.
  • JS (unregistered)

    Lots of blame to go around on this one. I agree that even the cursory of checks before wiping this machine would have prevented this (as a last defense of course). Just shoot an email out that says "Hey, anyone using DEVT09? We're gonna wipe it at 5PM on Friday." Simple. Plus you've covered your ass.

  • (cs) in reply to rmr
    rmr:
    Oh man. . . I got a sinking feeling in my stomach just reading the last couple of paragraphs.  I can't even imagine how the Expert Project Manager felt.

    And Goat Cheez:  chill.  This is as good as any code wtf.  I think Alex puts up the best of what is submitted, so if you want code wtfs, submit some good ones!

    Bean bag girl is back, and she's baaaad.  Gentlemen prefer blondes.


    Normally I wouldn't complain so much.... no wait... I would.... Still though, the last Code WTF was:

    8-14-2006, 3:55 PM 86019 
    							</td>
						</tr>
					</tbody></table>
				</h4>
				<table style="table-layout: auto;" border="0" cellpadding="0" cellspacing="0" width="100%">
				    <tbody><tr valign="top">
				        <td class="wtf_ForumPostSubhead" style="width: 48px; border-right-width: 0px;">
				            <img src="../Themes/default/thedailywtf/defaultAvatar.gif" border="0" height="50" width="1">
				        </td>
				        <td class="wtf_ForumPostSubhead" style="width: 100%; border-left-width: 0px;">
				            <table style="table-layout: auto;" border="0" cellpadding="0" cellspacing="0" width="100%">
				                <tbody><tr>
				                    <td colspan="2">
							            <h4 class="ForumPostTitle">
								            
								            Enterprise Beautification
							            </h4><br></td></tr></tbody></table></td></tr></tbody></table>

  • zip (unregistered) in reply to cronthenoob

    cronthenoob:
    m0ffx:


    If the school had been remotely good they would have been paying enough attention to students and their projects to be able to assign a grade anyway. In the case of an external exam board said exam board would also probably do the same thing.


    The program was relativly new to the school, and I think after that fiasco, they completely dumped it.  So basically I should have done more research, because that school was horrible.

    * waits patiently in the hope cron will name the school in question *

  • doc0tis (unregistered) in reply to Dazed
    Dazed:

    The Network Operations people are also decidedly guilty. Even an unbacked-up dev server might have a few days work of one or two people on it, and shouldn't be junked without checking. The briefest check would have shown a lot of recently modified files.


    I disagree. Why should they look at this server before wiping it. It's a temp server which (I assume) to their knowledge (and they're the ones in charge) no one is using. Why bother looking at it? My company doesn't have the resources to look at the files just for the heck of it. Sure in this case it would have saved a ton of time/money, but if it wasn't documented that there is something running on that server, and if the PM was willing to let all critical documents go unbacked up, then 100% of the responsibility is on the PM.

    --doc0tis
  • 8bitwizard (unregistered) in reply to JS
    Anonymous:
    Lots of blame to go around on this one. I agree that even the cursory of checks before wiping this machine would have prevented this (as a last defense of course). Just shoot an email out that says "Hey, anyone using DEVT09? We're gonna wipe it at 5PM on Friday." Simple. Plus you've covered your ass.

    Hell, just unplugging the network cable or admin-downing its switch port a week before re-imaging the machine would have been something.  Dev-temp box or not, you don't just pick a random box and start re-imaging it.

    Of course if they had a web checkout page for dev-temp servers and this one had long ago expired, then the net-ops people are a lot less at fault.
  • (cs) in reply to doc0tis
    Anonymous:


    I disagree. Why should they look at this server before wiping it. It's a temp server which (I assume) to their knowledge (and they're the ones in charge) no one is using. Why bother looking at it? My company doesn't have the resources to look at the files just for the heck of it. Sure in this case it would have saved a ton of time/money, but if it wasn't documented that there is something running on that server, and if the PM was willing to let all critical documents go unbacked up, then 100% of the responsibility is on the PM.

    --doc0tis


    I'm inclined to agree. It's mentioned as a "dev temp" server. Don't know the specific application of this phrase to their company, but it sounds an awful lot like what we refer to as the sandbox server. Nothing of any consequence should go on it, and it's understood that it can get wiped at any time with no warning whatsoever.
  • (cs) in reply to 8bitwizard
    Anonymous:
    Anonymous:
    Lots of blame to go around on this one. I agree that even the cursory of checks before wiping this machine would have prevented this (as a last defense of course). Just shoot an email out that says "Hey, anyone using DEVT09? We're gonna wipe it at 5PM on Friday." Simple. Plus you've covered your ass.

    Hell, just unplugging the network cable or admin-downing its switch port a week before re-imaging the machine would have been something.  Dev-temp box or not, you don't just pick a random box and start re-imaging it.

    Of course if they had a web checkout page for dev-temp servers and this one had long ago expired, then the net-ops people are a lot less at fault.


    Technically, it wasn't random at all. The last time that machine was ever used for anything was probably when the PM asked for a demo server for the software they were using. When they wiped that machine, that request was over a year and a half old. The PM should have sent a memo to IT telling them that even though the machine was a dev machine, it had valuable infomation on it and should be treated like a server as much as possible. That is the biggest WTF imo.

    There were things that IT could have done to prevent what happened, but none of them were WTFs. IT didn't disobey company policy.

    The PM decided to use a dev box for something that should be on production. A big WTF yes, but not as big of a WTF as him not deciding to tell IT that he is doing so.
  • (cs) in reply to cronthenoob

    cronthenoob:

    in the end, the school payed me back my tuition.   But I still have to live with the fact that I wasted 4 years of my life because of some idiot who doesnt know how to secure a PHP mailer.

    I'm not so sure it was wasted.  It sounds like you got the best education a programmer could get.... learning the value of backups.

     

  • (cs) in reply to JS
    JS:
    Just shoot an email out that says "Hey, anyone using DEVT09? We're gonna wipe it at 5PM on Friday." Simple. Plus you've covered your ass.

    Bad idea. You will get at least three "I'm not sure, please wait till I've checked" answers (of course without any follow-ups), therefore you never get any work done.
  • (cs) in reply to GrandmasterB
    GrandmasterB:

    I'm not so sure it was wasted.  It sounds like you got the best education a programmer could get.... learning the value of backups.

     

    I learned all about that from beanbag girl's good friend, "backup girl."
  • Anonymous Coward (unregistered)

    Of course, the real WTF is that they didn't use Hair-Pulling Girl's services.....
    I mean, it's only 1 cent/MB/month!

  • d4 (unregistered) in reply to ammoQ

    Nope, it's a good idea. If anyone feeds you a I'm-not-sure line, just reply with "Well, be sure to check before 5PM Friday and save anything you need."

  • lw (unregistered) in reply to Lazy to register
    Anonymous:
    First to wonder: who is the WTFer here????  The executives who hired the Expert?  The expert who turned a demonstration into the bakbone of the company's biggest project?  The executives who didn't realize this?? The policy of no backup on dev temp servers???


    Or the Network Operations staff who apparently didn't check with anyone before wiping out a server that would obviously contain many recently updated files.

    "Dev temp server" might not get backed up, but under what scenarios would it be acceptable to just blow it away to field a run of the mill request.  Presumably no one canceled the other request.

    Every place I've been at, rededicating even an unused old NT 4.0 Server is announced in emails for at least a few days.

    I'd give the Network Ops staff their share of the blame here, too.

  • LRB (unregistered)

    BTW there sure seem to be a lot of WTF's being committed in this thread.  Seems lots of posters are ASS-You-Me-ing that temporary means this or PM didn't do this etc.  Well that may or may not have happened.  However, when in IT in business it is almost never the best policy to assume.  It doesn't matter if no significant data should have been on that server by company policy or not.  No policy is perfect and no policy is perfectly followed.  The most important point is that you don't destroy resources that could potentially be worth millions or even more without even doing a simply check with the people last using that resource to see if it is valuable or not.  The objective is not to be some policy lawyer, but to do the best to secure and promote the company's best interests. 

  • (cs) in reply to GoatCheez

    GoatCheez:

    Normally I wouldn't complain so much.... no wait... I would.... Still though, the last Code WTF was: [Enterprise Beautification (2006-08-14)]

    There were two "code-related" ones last week:

    Two "code-related" articles are scheduled for this week. It's difficult to do "All Code" articles because I haven't received a new bad pattern lately (I question if there are any left), just repeats of the "for-case paradigm," "pointless conversion code," etc. What remains are convoluted messes of code that require dechypering lots of lines of often language-specific code just to understand what's wrong with it. Go take a look at the Side Bar for examples of what I'm talking about. I get the impression that most readers would prefer not to dig through the comments just to understand the Java, PHP, C++, C#, Perl, etc and why it's so bad to do that in that specific language.

  • (cs) in reply to John Bigboote

    John Bigboote:
    Anonymous:


    I disagree. Why should they look at this server before wiping it. It's a temp server which (I assume) to their knowledge (and they're the ones in charge) no one is using. Why bother looking at it? My company doesn't have the resources to look at the files just for the heck of it. Sure in this case it would have saved a ton of time/money, but if it wasn't documented that there is something running on that server, and if the PM was willing to let all critical documents go unbacked up, then 100% of the responsibility is on the PM.

    --doc0tis


    I'm inclined to agree. It's mentioned as a "dev temp" server. Don't know the specific application of this phrase to their company, but it sounds an awful lot like what we refer to as the sandbox server. Nothing of any consequence should go on it, and it's understood that it can get wiped at any time with no warning whatsoever.

    What would be the point of a dev server that "Nothing of any consequence should go on"?  If it was a "temp" server and not a "dev temp" server then I would be inclined to agree that it may have been a sand box.  But servers that can dissapear at any moment in time without warning are essentially useless to a dev team.

    A simple "We will be pulling this machine at 2pm today" would have sufficed.  If no one responds then you go ahead with it.  However, I think IT is on the bottom of the list of people who WTF'd in this story.  They didn't leave a multi-million dollar project on a "temp" server, but I still think they should have given some notice.

  • (cs)

    the same happened to the Duke Nukem Forever dev server heh.

  • Russ (unregistered) in reply to cronthenoob
    cronthenoob:
    This reminds me of something that happened to me in college!

    It was my final semester and my final project was coming along very nicely.  The guy next to me was creating a sort of email type thing,  I don't exactly remember. 

    Well he didn't secure it.  Some spammers got ahold of the script and sent out mass amounts of SPAM email.  The ISP shut the web server down.  After a few weeks, the server was finally online again.  The guy never secured his code, so of course it went down again.  Eventually my teacher decided to take the server and move it into the classroom where we could at least work on our projects locally. 

    We moved the server in the classroom, hooked it up, turned it on, and nothing happened.  All the data was corrupted somehow, everybody lost everything they ever created for the past 4 years.  I don't remember but I think we were able to recover some of our work.

    in the end, the school payed me back my tuition.   But I still have to live with the fact that I wasted 4 years of my life because of some idiot who doesnt know how to secure a PHP mailer.


    What the hell kind of school did you go to where your 4 years of education hung on 1 project?  That's kind of rediculous...
  • (cs) in reply to DigitalLogic
    DigitalLogic:

    John Bigboote:


    I'm inclined to agree. It's mentioned as a "dev temp" server. Don't know the specific application of this phrase to their company, but it sounds an awful lot like what we refer to as the sandbox server. Nothing of any consequence should go on it, and it's understood that it can get wiped at any time with no warning whatsoever.

    What would be the point of a dev server that "Nothing of any consequence should go on"? 



    In our case, playing with a new technology that might never see the light of production, or in any case won't see production in its current form. We've sandboxed around with BizTalk, Team Foundation Server, CMS, SharePoint, as well as several products where the mere act of installation might completely boof one of our permanent development servers.
  • (cs) in reply to ProfMikey
    ProfMikey:
    the same happened to the Duke Nukem Forever dev server heh.


    5 or 6 times, apparently.
  • Brad (unregistered) in reply to cronthenoob
    cronthenoob:

    But I still have to live with the fact that I wasted 4 years of my life because of some idiot who doesnt know how to secure a PHP mailer.


    No; you wasted 4 years because you didn't back it up, right?  Just like the PM.
  • Ran (unregistered)

    I agree with Dazed (above). Policy or no policy, it's moronic to wipe an in-use computer without backing up its contents just in case.

Leave a comment on “Back That Thang Up”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article