- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
Lucky you! :D
Actually, to be fair, it's a bazillion times better than in the past... forgive my exaggerations!
Admin
Too many "nom"s.
Anyway I knew as soon as I read that the server guys are above the devs, instead of working with them, that there'd be some major WTFage. I didn't expect it to be "we need to use Java and a dedicated server to display an order form and generate PDFs." It seems in this case, the server guys may have actually done some good... I don't know what to think anymore.
Admin
The rules of how applets work, that's who. Can't open a socket to anything other than the webserver that delivered them.
Even then, who's stopping anybody using any old DoS tools they fancy, and pointing them at your server? This is like a shop taking knives off their shelves so they can't be robbed at knifepoint
Admin
Indeed, that is TRWTF. I missed it at first myself.
In general, I'd say the server guys are being paranoid. But depending on the importance of the data, some paranoia may be a good thing. To be honest, that the final say on what is run on the servers is taken by people with any technical knowledge at all - as opposed to managers who know nothing about computers - is probably better than most companies.
Admin
Admin
Will they grow into some magic bins talk?
Admin
You're getting one side of the story.
Yes, yanking an applet offline without warning over a potential DoS that never happened is a WTF, but I don't know if I believe that part.
The reason I figure they wanted a new server was to run something other than their normal webserver. So, assuming the normal server is a LAMP and the new one would be a java server. That's an incredibly stupid thing to ask for just to be able to transform XML to PDF.
Admin
It depends. If the application was uploaded contrary to proper procedures (which seems to be the case since the server guys had veto power), then deleting it is the proper move. To the extent this disrupts customer experience, that's the fault of the people who uploaded it improperly in the first place.
Admin
Me neither.
True. To which a good response would have been "Is there any need for a dedicated server here? Can't this work be carried out on the existing app server?" or better yet "I presume you've got a cost/benefit analysis that justifies the new server for this". Not "I have bigger corporate teeth than you, do as I tell you".
Admin
Ho ho ho
Admin
Yanking a customer facing application without warning is only ever appropriate if an actual security issue has occurred... and even then one should carefully consider it. I don't care how it got to the point of being a customer facing application. If the devs uploaded it without following proper procedures, then that's a separate issue to deal with (and dismissal of the devs is not unreasonable in that case), but in no way does that justify yanking the application in this manner. Saying "that's the fault of the people who uploaded it" is just an "us vs. them" response, and is totally inappropriate in a business environment.
Admin
Admin
I wish our server guys were this picky
Admin
That's it Sponjk! Report to my office to turn in your union card. You have failed to rehabilitate from your anti-XML ways for the last time.
Admin
I don't see that the devs uploading this would be grounds for dismissal. It shouldn't be physically possible for them to do so in the first place. I have no way of putting something into actual production, and nor do I want it. My developer work ends when I hand something over to the QA and say "I think this is finished". Only once it gets through UAT is it even feasible that it will see production.
The proper procedures should be such that it can't happen. Sounds like, in this case, "proper procedures" were nowhere to be seen. Bit odd, considering how draconian these server guys were in other areas.
Admin
But..but...but..XML solves everything! If it ain't working for you, I suspect you're simply not using enough.....
Admin
Server teams and development teams really need to work closely together to keep stuff like this from happening. If the server team didn't like it, then the devs should have known about that before deploying. Also, the server team pulling an application offline like that without even telling the devs who maintain it is just ridiculous. On the other hand, having a dedicated server just to handle a feature that most people probably wouldn't even use, is definitely a WTF.
Often server teams and devs speak totally different languages. Personally I think that devs need to be well-informed about server operations and hardware so they can understand why server teams make the decisions they do. Likewise, I wish that server teams were composed of competent programmers so they can understand that the things we devs do aren't really as crazy or unsafe as they think.
Our server team only has one former dev (that I know of) and he did all his work in C++ and a little .NET 1.1. Needless to say we've had to occasionally butt heads because he's worried that my .NET 3.5 web sites are vulnerable to buffer overflow errors that might allow random Internet users to wipe out our Active Directory database...
Admin
That would be a disastrous policy. If any dev uploaded anything facing the customer to the server, then it would have to stay up until an actual security flaw could be found. You would, essentially, be staking the security of your entire website on your team being faster and more creative than the entire internet full of happy hackers. This, to you, is a good idea?
Admin
Regarding the "add a new server" issue, it appears the development team had somehow been led to believe the solution must have minimal impact on the existing systems. In this sense, it may have seemed that setting up a dedicated box was the appropriate approach.
Also, it is never stated that they didn't get their dedicated server. Only that they got PHP, maybe-JVM, and FTP instead of Tomcat, JBoss, and "other fancy-schmancy Java stuff."
Admin
Until you hit the virtual address space limit. Then you're screwed.
Admin
That's also the main problem these days. Why do you think everything from desktop applications to operating systems cost more hardware to do the same tasks? I'm not bashing anyone in particular, and sometimes the optimization effort is all for nought. However, developers should be able to easily profile for bottlenecks and be able to recognize potential waste. Even so.. there needs to be more coordination between everyone instead of the server guys just stamping their foot when they don't like something.
Admin
Note the "yanking without warning". I'm not saying the application couldn't be removed, only that doing so in a way that resulted in customers getting unexpected 404 errors is NOT the way to handle the situation. Alternatives:
Keep the application online while the developer corrects the issues.
Keep the application online for a period of time with a notification of it's pending removal.
Admin
A word of explanation from submitter:
Devs did not deploy anything by themselves on production - they do not want to, and, actually, they simply can't do that.
About 'XML to PDF generator', which is a bit unclear in article - devs did not want another dedicated server just to run this step - they wanted to create dedicated webservice hosted on the same machine, but deployed on some application server other than Tomcat - PHP alreay running other parts of application there. Server guys said 'NO JAVA ON OUR SERVERS! Get yourselves another, dedicated one!'. Yup, it looked like that.
I submitted some more sotries, which did not do into article. Let's wait, maybe you will read some day about server guys deleting parts of applications (because of security reasons) a few days after they deployed them, and replaced them with self - written ones (which were working incorrectly, BTW). Or about devs being cut off from test environments, because they generated too much traffic and HDD activity.
Do not forget, that DoS activity was ONE HTTP REQUEST per page access - just as if there was an IMG tag with invalid path as 'src'.
I am surprised to see so little 'The real WTF is submitter' posts. Guys, do not let me down!
Cheers
Admin
The one concession I happily give Server Guys is that when my app screws up, I annoy a few managers. When the server screws up, the entire operation grinds to a halt (since the servers maintain order entry, the warehouse, etc.) So, I'm willing to cut them some slack when they say changing something on the server is a Bad Idea.
That said, I would expect them to have some sort of alternative in mind. (Our local IT guys are great for figuring out ways to make things work without screwing up the infrastructure. Our head office IT screws up the servers just fine without my help.)
Admin
Or, to quote the OP:
"Hobson's team decided to redirect the XML that would normally be used to a web service running on a dedicated server running Tomcat that would apply the data to a template and spit back a nice, pretty PDF."
Yup, that appears to be a sound architecture to me.
For once, I'm with the server guys on this one. It's a weakness of the system. If you can't persuade the PHBs to fire each and every one of the architecture astronauts who build this rubbish, then your next best bet is a human firewall.
Admin
I am a backroom server guy, but jeez, lighten up and take the rod, out of the butt of the rod you have your butts!
Argh! All of 50MB consumed in a single process on a dedicated server! Good job they ain't running the latest Oracle, the executable alone is 120MB before it even grabs a chunk of memory usually consisting of the 2GB+ mark!
There's being studious and then there's being anal!
Admin
Admin
I don't know if it's the situation in this case, but in my last job, there were rules about sharing servers across applications and cost was not in the equation (hail process); it may have been a corporate requirement to put it on a dedicated server. It may have nothing to do with a developers' decision.
As far as the server guys prohibiting certain implementations? I can see an architect (or team) reviewing design, but since when do guys that build and manage servers also manage the architecture of the applications running thereon? It's possible these guys had a dual-function role, but I've never seen that before, at least not like this.
Admin
Actually because it's a US holiday, this is the one case I can finally say it's PROPERLY spelled "Labor"!
Admin
Why do the server guys even get a voice? They run the hardware. The hardware belongs to whatever group is using it, and why the hell are you so keen on php? It's not like we can't install tomcat.
Admin
"just in case they network problems"
this sentence no verb.
;o)
Admin
It's also a Canadian holiday called Labour Day, so let's split the difference, eh?
Admin
Honestly, I don't blame them for no-go'ing Tomcat. The thing is a total memory hog, and refuses to do anything (logging, file layout, etc.) the way every other Unix-y service does it. If I could banish it from my servers, I would.
Admin
Structured Query Language doesn't allow anonymous access? Wow, had no idea that was in the spec.
Admin
FTP is insecure as hell, at the network level. Go look for straight FTP usage in a PCI compliant network: It isn't going to happen. At the bare minimum, the file transfer must be done over an SSH tunnel, using one of many different ways to do it.
Admin
As others have stated - there may very well be sound arguments against the use of Java, Tomcat, a dedicated server, whatever. That's not the issue here. The issue here is that the sysadmins were allowed to make that call unilaterally, as opposed to an architect, project lead, department lead, whatever.
The job of a sysadmin is to ensure that every other department has the system resources it needs to function. That involves procuring and maintaining the hardware, operating systems, and off-the-shelf software. It may also occasionally involve implementing IT policies - firewalls, acceptable use, etc. - but only to the extent that those policies have been approved by a CIO/CTO or other executive. If the execs want everybody to be able to store their entire (legal, of course) music collections on their home directories, then it is up to the sysadmins to make the resources available or suggest (not enforce) an alternative, not put in 1 GB quotas anyway and tell everybody else to go to hell.
When a company employs a development staff, it is never the role of a sysadmin to question a system's design. That is not what they are trained for. They may assign a cost to that design - such as, it will cost us $2000 and approximately 20 man-hours per month to maintain a Tomcat server - but it is up to the people writing the checks to decide whether or not that warrants putting the boot on it. If a $2000 server and 20 man-hours of sysadmin time can offset 1000 hours of developer time (conservatively estimated at $25,000), then that is what you implement.
Some of you obviously spent way too much time reading crap like BOFH. The real world doesn't work like that. The only time that sysadmins should ever be making a stink is when it's a legal or ethical issue, and even then, that doesn't confer the right to act unilaterally.
I'll bet that these people originally came from a college or university IT department where the sysadmins are essentially given free reign, because nobody above them has the time or inclination to review their work. That is typically the environment where you see this kind of arbitrary and draconian policymaking and enforcement.
Admin
What's the alternative? Don't deploy unless you can prove the absence of flaws? Sounds a little impossible. I'm not willing to yank an app because it might contain a flaw.
Admin
What strikes me as odd is that there is a complete disconnect between your prod ops team and your developers.
You do see this sometimes, when 'IT' and development fall under different 'silos'.
What is the current server architecture in place?
Why would a developer develop an application outside of this environment!? (i.e if you are a LAMP shop, (with PHP) your applications should be in php. If you are a Java shop, you develop in Java, which will then already be on the servers.)
Work cross departmentally, communicate.
Admin
Perhaps there are multiple stacks in use - history, pilot project, etc. regardless, it isn't up to the admins to decide what's acceptable.
Admin
The real WTFs here are:
1.) Using PHP in an apparently large scale web app environment.
2.) Dev guys asking to deploy entirely new services (Tomcat, etc.) to add one feature to an existing app. Unless we're missing some details, this would have been a substantially inefficient path to take.
3.) Server guys who have the authority to dictate specific application architectures. This is the equivalent of devs getting to dictate IT infrastructure. The specific example here turns an elegant, easy to develop, test and maintain solution into something significantly more expensive. This was wasted money.
4.) Server guys bitching about 50mb of RAM (which is likely <1% of the servers RAM capacity), likely resulting in 100s of hours of dev/QA/etc. time. This was a LOT of wasted money.
The root failure is the dev and server guys lack of looking at big picture ROI of their decisions, and instead taking a "minimize my hassle" approach.
Admin
Today's spellchecker was sponsored by the Server Guys.
Admin
New manager. Two years ago I was brought in as the new Technology head. There was an IT team that was simply known as the No team. I called the first IT up and asked him the settings for my laptop. He said, "No personal laptops are allowed." I said, "You have one more chance here; what are the settings for my laptop." He repeated the policy and I told him he could wait in the front waiting area as we were going to have a meeting there. I had security meet him as he was fired. When the manager of IT came to my office to complain I ignored his complaints and told him that he too would have one chance to configure my laptop. He too refused and I fired him on the spot. I then interviewed the remaining IT staff until I found one guy who had been chafing to actually help people and promoted him to head from the very bottom. Not only did I get a huge bonus due to the huge number of people who voted for me in the company "moral boosters" program but my new head of IT fired an additional 4 IT people. Not one of the positions needed to be restaffed as along with the 6 people to go went a pile of stupid policies and paperwork that had kept 6 people busy.
A perfect example of the people we let go was that out of the last 500 system upgrade requests not a single upgrade had been approved. Yet the top three people in IT all had the highest of high end machines(quad core in 2007) for mostly ssh usage while some people were still using low end pentium 4s to do photo editing. My new IT guy kept his crap machine and sent the now free top end machines to those who really needed them. Plus the now vacant positions freed up enough IT budget to immediately start rotating out the worst machines.
HR told me that most were trying to use our company as a reference.
Admin
Or ever turn on...
Admin
Why would any organization run things by their IT people before implementing them? The IT guys in our lab are (barely) just responsible for popping CDs in drives and patching machines. Sounds like a runaway IT department needs a bitchslapping or two.
Admin
If we write in IPA, we would need provisions for 'r-droppers'!
ˈleʲbɘ⁽ʴ⁾ ˈdeʲ
Admin
Congratulations on ruffling your feathers, young cock, but this is a ridiculuous way to manage people.
Admin
Yes! Yes! Yes!
Admin
Sometimes you have no choice. People acting in ridiculously incompetent/obstructionist manners while getting paid to do a job will eventually get treated ridiculously.
I've seen cases where tens of thousands of dollars get lost in single incidents because individuals (be it IT, Management or Dev), or even entire teams engage in prima donna policies that establish power positions and dick-measuring (the very thing you accuse Emperor of). That's just what inept people do for job security.
Money gets lost in lost opportunities, increased running cost, lost of talent (talented people won't take crap beyond a certain point). Hell, I've even seen incompetent people suggesting (or even all-out threatening) bodily harm to those who dare to question their modus operandi. I shit you not.
Until you know the specifics, don't think this was a disproportionate way to manage.
Admin
Every time I read The Daily WTF I feel like I'm grading a high school essay. Anyone know how to get red ink off a monitor?
Admin
Not that I believe that the story is true, but firing a lower-level employee for following his managers reasonable policy is never appropriate. Personal laptops are a significant security risk, so it's sensible to not allow them.
But he doesn't have to worry about security anyways, he's got an IT staff that doesn't worry about that "pile of stupid policies".