- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
I could see it increasing performance reducing fragmentation and slack space...if we were talking about this 20 years ago. :-D
Admin
I've got some of those old books as well. Hell, in college I had textbooks which went into such issues. They're the sort of thing that were great at curing bottlenecks 30-40 years ago, but would save you a microsecond or so these days if the drives' firmware and better os-level filesystem code didn't render most of these optimizations obsolete.
Admin
Have you ever considered taking a class for that?
Admin
Lemme guess: you've got certifications?
Admin
I wondered if he stated that all drives were to be the same rev hardware with the latest and greatest drivers.
Admin
At one time this kind of IO juggling was really important. But things change.
Hardware RAID is way better than it used to be, don't try to outsmart it. Get lots of spindles, put them all in one raid 10 volume. Done. Maybe logs on another partition, but few people really need that.
Manually managing placement of db data on disks is extreme micro optimization that ads a lot of complexity, for very small gains in performance. It's just not worth it.
DBAs who tell you it's necessary are the ones who learned their trade 15 years ago, and refuse to keep up with current trends.
Admin
Yeah, of course it is. So your database has only been running for a week then? It would be 6 Petabytes by next month.
Seeing as Walmart are running at an estimated 2.5 Petabytes I call BS. If you were in charge of a database that size your would be working for a pretty major organisation and I can't imagine you would feel the need to scavenge for bragging rights on TDWTF!
Admin
I'd agree with Brad, not all fingers are same. Unless you are willing to learn, truly, you will never learn. No matter how 'learned' or 'certified' we are, we will still learn something new, something good, or just something... that would make us better; iff, obviously, we are willing to learn!
Btw, good write! (0:
Admin
Hilarious article, I love the irony. If this story was true, I would have to ask the DBA what he was using the 10 mirrored concatenated partitions for. I mean why not 9 or 11? 10 performance tiers on the same spindles? haha
Not to berate BradC as an individual, but rather all the advocates of raid 10 > raid 5, I will say this: you are irresponsible. Depending on the application (such as QA databases), raid 5 is at times BETTER than raid 10. The simple reason is cost. If you were deaf, would you pay $100 more for the premium speaker package on a new car?
For those that want a technical explaination of my opinion and not an analogy, the common comparisons has to be detailed from the perspectives of someone (or company) who is completely unconcerned with money and one who is ultimately concerned with money. First, lets establish a price difference. RAID 5 can be configured from 3 to, oh lets just use EMC's 7+1 as an example. To match the capacity of RAID 5, you will need to spend 33-75% more per usable byte. For the home user buying SATA drives, $500 worth of RAID 5 will then be $666 to $875 for RAID 10. For the enterprise user, the $15000 your company spends on a rack of disks running raid 5 will be $19950-26250. Now lets see what these additional monies buy you.
A common comparison is performance: RAID 10 does not have parity calculations that is involved with RAID 5. First of all, I will say that RAID 10 with misalignment or a poor choice in stripe size will perform worse than raid 5 with a properly aligned partition and a stripe size accomodating to your application's IO sizes. Even BAARF would admit, the performance degradation is with writes to disk and that read performance is comparable to RAID 10. This isn't to say writes to RAID 5 is poor, it's just not as good. However, what if your application is simply video streaming or the app is only a file share for 5 users? Consider to yourself if the price point of write performance is still worth it if you are the proverbial deaf person. For the enterprise user, you likely pay for an array that has available write cache. Writes then are at the speed of electronics and not spindle. What the price point of RAID 10 will buy you is sustainable high write throughput. Some app's need this, however a lot do not.
IO performance during recovery and recovery time in RAID 10 is better than RAID 5, I can agree. If we are to talk about drive failures, then we must then talk about mean time between failures. Lets use a low mtbf estimate of 300,000 hours. This doesn't mean after 300k hours, all your drives will fail. This means your drives will likely fail between 0-600k hours, or 0 to about 68 years. Day to day, you have a 1 in 25000 chance of a drive reaching it's point of failure. Additionally, if it takes 6 hours to rebuild, you can guarentee 99.9% full performance in your SLA. There are apps that require a better SLA and for those, RAID 10 is the better choice, but for QA, development, etc it would be the wrong choice. The deciding factor is whether or not the money lost from such downtime is greater than the money spent offering this security.
What about losing a second drive? Again, an MTBF issue. Day to day, if your chances of 1 drive failure per day is 1 in 25000, but the question assumes you have already lost 1 drive. So 2 disk failures within the hours of RAID 5 rebuilding will be 3 to 7 in 100,000. If your data requires the uptime and you want better odds against second drive failure, RAID 10 is for you. Don't let anyone fool you that there isn't a 1 in 100,000 chance that the second failure won't be the mirror of the failure though.
On a different topic, BradC also points out dedicated LUNs are better than shared LUNs. First of all, a LUN is not a physical disk, spindle, or any moving part of it. Like the acronym suggests, Logical Unit Number... its a SCSI thing. It is never shared, except with clusters. A file system on the LUN could be shared, such as a... file share. What he meant to say, so as not to confuse people, is dedicated spindles vs shared spindles. I will argue this point as well. In shared, centralized storage environments, not every server owner requires storage for performance. (Some require better scalability, reliability, etc) If I had a database that generates 800 IOPS sustained, that I can fit on 4 spindles and a data warehousing application generating 60 IOPS peak that requires 8 spindles, I would confidently have them share disks. If your disks are capable of 150 IOPS each, the data warehousing application would leave your database with 145 IOPS per spindle or 1740 IOPS peak; leaving you under 50% utilized. If you used 4 dedicated LUNs for 600 IOPS peak, your database would stop running. You must be intelligent about it, learn to balance workload.
Ultimately, nothing, not even RAID 10 provides you with 100% uptime or 100% full performance. You can get closer and closer to 100%, but you will never touch it. You will need a disaster recovery site (doubling your investment) to protect against site failures. Or how about 3 way DR in case of multiple site disasters like the blackout that encompassed all of the east coast? If there is a 2012 situation, you will surely need a satellite uplink to protect your data in space and I'm sure EMC will develop and sell it to you for the right price. 3 way mirrors and even 4 way mirrors offer more than a 2 way raid 10, so why not advocate that? Solid state disk offer better performance than spindles, so why don't individuals like BradC want to phase out spindles? In fact, the price point and (hopeful) mtbf of solid state is so high, it is most commonly implemented in RAID 5. As the price of SSDs fall, the next latest and greatest technology will come along and for the same 2 reasons, consumers will implement it using a lower cost RAID 5. Happy users, happy DBA, happy CIO.
I only have 3 years of industry experience and am in no way saying there aren't things I do not know about this subject and my statistics skills are definitely lacking. Surely you get the idea though. This is merely a counterpoint.
Admin
I hope he also changed the dd and backup commands.
Admin
Now Google is a slightly odd fish. It does tons of queries and not many writes (relatively speaking), and the queries don't have to be synchronous with the writes - because one thing writes to the DB, and other things read from it, and the things reading have no interest in when writes were made. This is different from most database applications.
But, Google does show something - what is good for one system may not be best overall.
Google want lots of query performance, and don't care about reliability or performance from a single PC. It is much more effective for them to have 10 $300 desktop PCs than 1 $3000 server PC. So what if a disk explodes - that same data is replicated on 5 zillion other disks. It doesn't matter than a desktop PC can only do half the queries of a server PC, because you've got 10 times as many for the same price, so you get 5 times the performance.
Admin
Admin
Well done!
We DBAs are not all complete prats like this guy in the story!
I am a certified ( should be certified? ) DBA, but when I tune I involve everyone I think has something valid to say and will help make my ideas become better. The SAN admins and Unix admins are at the top of my list of great resources. I expect the SAs to question everything I ask for, so I can be sure that I have it right. I wouldn't feel 100% if what I said, went straight in without someone asking me why?
There was a time when you need to think about physical disk placement on local storage, but those days are fading fast with huge caches and monster, ripper SAN systems. You still need good planning, but the days of wasting 75% of the platter are long gone.
Admin
The physical and logical configuration of the entire I/O subsystem should be kept in mind.
In short, it not necessarily all about the RAID. It is about understanding the entire life-cycle of an I/O request.
For example, in many large-cache storage frames we've not been able to tell the difference between RAID5 disk and mirrored.
Naturally, your mileage may vary.
Admin
Mine goes up to 10^(-i)!
(captcha: eros)
Admin
I didn't get the feeling that the story was saying that all DBAs are stupid, only this particular one ^^
Admin
Admin
Don't you mean Moron?
Admin
The #1 problem I have with database guys (not just DBAs but database guys in general) is their belief that there is only ever ONE way to do it (their way) even when there may be other ways that are better.
Admin
ARGH NO.
The moral of the story is back up your crazy complex configurations with COLD HARD BENCHMARKS.
Admin
BradC, what kind of a moron (not moran) thinks that an amusing anecdote about an imbecile of a DBA is somehow an attempt to indict all DBAs?
Admin
RE: 2010-03-23 10:21 • by Mikkel
As fun as it was to read, I really hate to say this, but..., I agree. The sysadmin did the wrong thing.
If they have an issue later that requires troubleshooting the disk i/o system, they are going to be going on faulty feedback from a "fake" system command. This will cause endless headaches until someone realizes they've been slipped a mickey... that their system df command is a trojan.
Paul should simply have let the system hobble along until they got rid of the DBA and replaced him with someone competent. Either that, or (better), he should have come clean about what he'd done. Heck, if he had, HE would have been the hero of the story. Maybe not in the eyes of the C.DBA, but in the eyes of his company's management. And that's where it would count anyway!
Admin
This is like the third or fourth reply, just in this thread, where people wanting to reply to a comment seem to feel the need to do so in a snide, disparaging fashion.
It's like we've lost all sense of basic civility and professionalism, and it's really disturbing.
Somehow, I just don't imagine really top-notch performers acting this way. Can you imagine this mindset in the Apollo 13 control room? No.
It's perfectly OK to disagree with someone, but you can at least disagree on the technical merits without dragging in ad-hoc insults.
In other words... grow up! This isn't high school anymore (some of) you guys.
Admin
What the hell is a moran? Is it similar to a moron?
Admin
Boring! Boring! Boring!
Admin
Ha Ha. Very Funny. It is to laugh! ;-)
Admin
Welcome to the Internet. Things you should know:
Admin
So, some fuckwad can't be bothered to learn the language he's using and since I don't care or know about every idiot's personal handicaps in the head department and don't protest against every one of their stupidities, I am now required accept them all? Yeah, right.
Btw, was the mullet guy's name "mike" by any chance?
Admin
He couldn't find a wooden tabletop
Admin
http://www.urbandictionary.com/define.php?term=moran
Admin
whoah there Sparky, Paul didn't allow anything, the Certified DBA took it upon himself to use the screen shot of the fictional directory structure as a basis for an article. What was Paul supposed to do, out him? What, and miss all the fun? LOL... it certainly does hightlight a lack of scholarship in whatever publication it got published in, but you know a lot of these pubs PC week, etc. they just try to fill space.
TRWTF is that DBA prolly just winged this stuff to "look smart" and got away with it for as long as he did. After all, don't all RDBMS's have standard disk tuning procedures?
Admin
Pretty interesting stuff, Terry. Thanks.
Admin
Hm...
Admin
This just goes to show that "years of experience" (most likely the same exact experience 15 years over) does not equal seniority, talent or good use of common sense. Certified Morons abound!
Admin
It sounds like Paul needs to grow a pair of balls
Admin
Actually, Sysadmin, DBA or not, you do layout storage differently wether you are going for best serial throughput or insanely high volumes of small size random i/o. (But most sysadmins already know this.) The real issue here is when the sysadmin asked the question:
"I don't see how this improves performance..."
They guy did NOT answer the question. ('I've been doing it X years' is not an answer. Especially to a sysadmin that, as a general rule, do know a lot about disk performance.)
If you want something setup a special way: You better KNOW why it will be faster, and better be WILLING to explain it. (Doesn't matter who you are, or your rank, or job description.)
If you can't defend why it's faster, then, in my opinion, you are full of cr@p. (The corollary is: What happened the last time you ran a benchmark? Oh... you have never benchmarked this. Ahh, I see....)
Admin
It probably has been said in the comments already...but honestly 5 pages of comments is a little much to read.
It is important never to assume that your "methods" that worked on the last system are going to work on the current system. They are a good baseline but your configuration either network, server, SAN and anything else might throw your "expertise" out the window. The methods that DBA's used 2 years ago for disk tuning are different they they are today. Advances in technology and the way systems work can mess with what you know. Yes, disk reads are different on different areas of the disk...but does your vendor account for that currently? How does the disk management system your systems administration people use integrate into the database system?
Oracle currently has a feature called ASM....Automatic Storage Management. Basically tell it what disks you have and their characteristics and it will put the indexes, datafiles, and all the other needed files where it "knows" is best for the system. I currently choose not to do this....I sure hope that they people we have on the storage team know the system better than me and I can work with them to get the best performance....I don't want the responsibility for the storage...but I am glad to work with them to get the best possible bang for our buck.
Regards Tim
Admin
I'm going to invoke my psychic powers and pronounce that the "Certified DBA" was working for Sybase. We had one of those constantly "tuning" our poor-performing OLTP database... I can't begin to describe the stupid shit he pulled to get the DB to perform decently. (For anyone who's ever had the "pleasure" of working with Sybase, you may be familiar with raw devices and all the joy they bring.)
Last year we replaced our Sybase servers with MSSQL 2008 boxes using RAID and standard file systems. Strangely enough, we've had exactly zero database performance issues since then.
Admin
unless the DBA understands the caching algorithms and the striping algorithms used in the disk subsystem then he should keep well clear.
DBA's like the one described (and most in my experience) expect no RAID or caching, or have been 'taught' by someone with a specific product limitations in mind - often that product is now years gone.
DBA's should keep to the DB, or if they want to be included in the discussions on how the systems are configured they should devote regular effort to understanding the technology changes that happen in our industry (and so should the SA!).
There are too many bright people involved in the development of new HW and SW products for anyone to assume that knowledge learned in the past can be applied verbatim to current products. Even the basic principles change on some products and anyone who fails to devote significant effort to keeping up with the tools they are expected to work with, deserves sacking - without this effort all that happens is that the 'profession' we are in is (sometimes deservedly) seen as a bunch of cowboys
Admin
cough
Admin
jagoff! ;-)
Admin
I realize I'm horribly late to the party, but one thing I've been noticing is how some of y'all seem to be missing the part where "who has the clout and who does not with management" plays a role.
So, if the DBA who can't configure hard drives for anything is the one who can throw their weight around and get other people fired, is Paul the admin supposed to fall on his sword for y'all to be satisfied that one day in the distant future some management person will see the light, fire the DBA, and rehire Paul?
C'mon. The real world doesn't work this way. No hierarchically-structured system of human relations that depends on anything other than strictly measurable metrics of ability to perform will work this way. There's always influence peddling, empire building, plain old jealousy and a thousand other emotional factors that get in the way of "oh well, Paul should have fallen on his sword for the greater good of an efficient database".
How many of you would sacrifice a decent-paying job for someone ELSE's abstract moral principle? Certainly not me. If I had someone else joggling my elbow all the time when I knew I could get something done faster by end-running about without that person knowing, I damn well would. I'm not paid to fight battles with people who can get me fired if I contradict their idea of a pet project in front of the bigwigs.
Admin
i am a god
Admin
You mean maron, I guess... Or maran?
Admin
GET A BRAIN, MORANS.
GO USA!