The Daily WTF: Curious Perversions in Information Technology

2006-05-22 Reply Admin

Anonymous:
why does everyone get so excited about database blunders? there's plenty of highly skilled programmers who don't have a clue about databases. as a games programmer, i have zero need to ever learn the pain of db design, so why bother? looks kind of boring to me!

So which part of a game isn't a database?

Really, a good game programmer can learn a lot from the RDBMS guys, like when to page your textures in and out, how to cache the game objects that you don't have memory to hang onto just now...

Did you ever read Gems?

2006-05-22 Reply Admin

This is not a WTF.. this a NFW (No *&^%$# Way ! )

Another reason to take a look at the code before you sign a contract.

2006-05-22 Reply Admin

firewireguy:
Oh dear God! I thought relational databases were day one of all CS degrees!

I am currently studying CS in Holland (University Of Leiden), en Databases is a first year required course, as indeed I think it should be...

I don't know all these universities you are mentioning here, not living in the USA, but doesn't it seem to be pretty unnerving to you all that someone can graduate in CS while never having had at least an introduction to relational databases and the major relevant concepts about them?!?

2006-05-22 Reply Admin

Anonymous:
Anonymous:
why does everyone get so excited about database blunders? there's plenty of highly skilled programmers who don't have a clue about databases. as a games programmer, i have zero need to ever learn the pain of db design, so why bother? looks kind of boring to me!

So which part of a game isn't a database?

Really, a good game programmer can learn a lot from the RDBMS guys, like when to page your textures in and out, how to cache the game objects that you don't have memory to hang onto just now...

Did you ever read Gems?

The video game industry is well above learning those pesky uninteresting things like properly organizing data. All that's really important is how to push more polygons with more complex shaders, and if some interesting gameplay feature cannot be done because the data model sucks, then well, too bad.

The grandparent poster's attitude is a good illustration of what's wrong with the video game industry. Bunch of people who fail to realize that most of the problems they have to solve have already been solved for years in other branches of computer science, but too proud to try to learn about it. They are mostly still in the prehistoric age, reinventing everything themselves, poorly.

And indeed, most of the video game development is about massaging a bunch of data, which is created and edited concurrently by a bunch of different people.
But no, RDBMS is a lowly business-oriented technology. Video game programmers are real programmers, they write their own storage systems.

2006-05-22 Reply Admin

It's not that bad... if you are getting paid per database table :)

2006-05-22 Reply Admin

Anonymous:
Anonymous:
Anonymous:
why does everyone get so excited about database blunders? there's plenty of highly skilled programmers who don't have a clue about databases. as a games programmer, i have zero need to ever learn the pain of db design, so why bother? looks kind of boring to me!

So which part of a game isn't a database?

Really, a good game programmer can learn a lot from the RDBMS guys, like when to page your textures in and out, how to cache the game objects that you don't have memory to hang onto just now...

Did you ever read Gems?

The video game industry is well above learning those pesky uninteresting things like properly organizing data. All that's really important is how to push more polygons with more complex shaders, and if some interesting gameplay feature cannot be done because the data model sucks, then well, too bad.

The grandparent poster's attitude is a good illustration of what's wrong with the video game industry. Bunch of people who fail to realize that most of the problems they have to solve have already been solved for years in other branches of computer science, but too proud to try to learn about it. They are mostly still in the prehistoric age, reinventing everything themselves, poorly.

And indeed, most of the video game development is about massaging a bunch of data, which is created and edited concurrently by a bunch of different people.
But no, RDBMS is a lowly business-oriented technology. Video game programmers are real programmers, they write their own storage systems.

I don't mean to offend you, but most games that make it on to the market actually have very well written and efficient data models. They have to, they need to manipulate that data in real time. Game programming is not exclusively about polygons and shaders. The problem with videogames today is not the data model, it's the costs of providing enough fun content.

As for the applications whose goals are specifically to handle data, not so many of them are written profesionally. As a game programmer, I can learn one important thing from the author of such code: how not to do it.

Are you trying to find faults with the game development field just because you're unhappy about the mess in 'enterprise' DB programming? Don't worry: if you work hard enough, maybe in five-ten years you'll end up in a rewarding well paid solution developer position, with plenty of fun challenges ;)

Devildog

2006-05-22 Reply Admin

Anonymous:
I don't mean to offend you, but most games that make it on to the market actually have very well written and efficient data models. They have to, they need to manipulate that data in real time. Game programming is not exclusively about polygons and shaders. The problem with videogames today is not the data model, it's the costs of providing enough fun content.

The fact that a large part of the industry continuously reinvent the basic cogs that make a game run doesn't help driving that cost down. Scripting engines, level editors, content management, collision, exporters, high level game engine, how often does the requirements for those things actually change from a game to another? Yet they are routinely thrown away and rewritten.
Except when they're such a mess of WTF-ridden spaghetti code that they drag the development down. In that case, they do seem to be reused.
Maybe I have just been unlucky, but out of the 4 "game that did hit the market" that I previously worked on and the one I'm working on now, I have worked only with rube goldberg asset production chains, messy code, poorly designed asset management system that seem to crappily reinvent all kind of wheels, ad-hoc tools hastily put together, shitty hardcoded stuff all over the C++ sources, and I can barely count the number of times I had to tell a designer "Would be nice. However, the engine works in this and that way and those kind of object just cannot be used to do XXX without a painful hack"

Of course, I have been part of the problem in a lot of these cases, but the game I'm working on takes the cake in terms of being a fucking mess, and this time it's hardly my fault since most of the tech and tools were actually done before I joined the company.
It has been received very well at E3 though, thank you very much. Coding quality has nothing to do with the success of a game.

Are you trying to find faults with the game development field just because you're unhappy about the mess in 'enterprise' DB programming?

No, I find faults in the game development field because I work in it.

2006-05-22 Reply Admin

Anonymous:
This seems to be very common - even Wordpress.com (http://photomatt.net/2006/03/01/wordpress-and-lyceum/) does this. WTF aside, is there *any* real reason why this is a good idea?

I'll bite:

If the sum of rows is HUGE and you're getting crappy index statistics because of it, breaking the data into separate tables gives better index and performance.
Blatanly ignoring the aggregation issue, but still.

ammoQ · 2006-05-22 Reply Admin

Anonymous:
Anonymous:
This seems to be very common - even Wordpress.com (http://photomatt.net/2006/03/01/wordpress-and-lyceum/) does this. WTF aside, is there *any* real reason why this is a good idea?

I'll bite:

If the sum of rows is HUGE and you're getting crappy index statistics because of it, breaking the data into separate tables gives better index and performance.
Blatanly ignoring the aggregation issue, but still.

Which database system?

2006-05-22 Reply Admin

Zlodo: the poster who complained about the boredom of seeing the same database wtf's over and over on this site, and almost nothing else, has a point. I for one would like to see something different. So if you work with such messy wtf-y code, why not try to submit some? Let's all have a laugh and chill, right?

The mechanics of a large game project is a side issue, and I sure don't like messy systems and the incompetent who make them even more so, but I don't think reusing everything is a good solution, and certainly RDBMS logic won't help. A good designer has to know the power of the dev's tools. And if a gameplay mechanic is worth implementing, a skilled programmer team will take the effort to 'bend' the engine a little. As long as they have access to the code that is..

And about the "I don't mean to offend you" part, please replace it with "Actually" :)

2006-05-22 Reply Admin

It could be worse.....they could've set their tables up like Remedy....

You see, that's funny, because I've seen Remedy tables in another life.

Quite ugly, those.

2006-05-22 Reply Admin

Anonymous:
If the sum of rows is HUGE and you're getting crappy index statistics because of it, breaking the data into separate tables gives better index and performance.
Blatanly ignoring the aggregation issue, but still.

Breaking data into separate tables is fine if each "customer" or client or user's application can be treated as a standalone instance. But, if you really need to aggregate data there is a great way to do this with the more advanced SQL systems: horizontal table partitioning. This is even available in PostgreSQL now. Physically, you have separate tables, even on separate physical disks, but you also have a layer that binds these together to present a unified table. And this is not just a view on a giant UNION query, but an under-the-covers implementation that allows you complete logical access to all the tables as one. Think of it as SQL RAID.

2006-05-22 Reply Admin

Anonymous:
I am currently studying CS in Holland (University Of Leiden), en Databases is a first year required course, as indeed I think it should be...

I don't know all these universities you are mentioning here, not living in the USA, but doesn't it seem to be pretty unnerving to you all that someone can graduate in CS while never having had at least an introduction to relational databases and the major relevant concepts about them?!?

When I was getting my Masters in CS (I'm a convert with a non-technical undergrad), I asked about getting into database course (I went to Washington University in St. Louis). My professor/mentor looked at me funny and said, "We don't have database courses. If you want, you can take one of the classes offered by the Information Systems Department in the Business School."

I found that attitude to be common across the professors in the CS Department. I realize it wasn't relevant to me while getting a Master's, but not having a CS database course seemed a little out of touch with what's going on in the business world.

Chalk it up to Academia and their Ivory Towers, I guess.

Zlodo · 2006-05-22 Reply Admin

Anonymous:
Zlodo: the poster who complained about the boredom of seeing the same database wtf's over and over on this site, and almost nothing else, has a point. I for one would like to see something different. So if you work with such messy wtf-y code, why not try to submit some? Let's all have a laugh and chill, right?

I'm actually a coward. Most of my colleagues read thedailywtf, and none of us have the balls to post code from our codebase, which is too easily recognizable.

The mechanics of a large game project is a side issue, and I sure don't like messy systems and the incompetent who make them even more so, but I don't think reusing everything is a good solution, and certainly RDBMS logic won't help.

I got indeed kinda sidetracked off topic with the reusability issue, but basically my gripe is that the video game industry has a way of ignoring useful techniques from other IT application domains, and poor reusability is one many symptoms of this.

As for RDBMS, I'm maintaining a tool that is used to store and edit assets in a common datastore for the whole team here. Syncing, concurrent reading and writing, locking, all these things have been reinvented, poorly.
I continuously pull my hairs because of some inter-assets references breaks because some sync issue occured. We have lock contentions and performance problems all over the place, and since all the content designers here have to go through this tool to export their data, performance is a productivity issue aswell. And don't get me started on race conditions and data corruption when something crashes.
All those things are problems that have been solved by RDBMS (we do have some stuff in oracle, too, but not everything, just as to make things more complicated without actually buying performance or robustness)

As another example of what happens when a game programmer do database stuff with a lacking DB knowledge, I know a mmorpg that stores the list of items that a player has as a binary array in a blob. Want to know how many players have item xxx ? Sure thing, just read each blob sequentially and see if it's in there. Very efficient.

A good designer has to know the power of the dev's tools. And if a gameplay mechanic is worth implementing, a skilled programmer team will take the effort to 'bend' the engine a little. As long as they have access to the code that is..

The programming team may have the will to do it, but if it's physically unable to do it because the budget/time constraint already make it a challenge to stabilize the game, if the feature require to hammer the engine into the proper shape, it gets cut. If it's cheap to do, then it can make it.

And about the "I don't mean to offend you" part, please replace it with "Actually" :)

No offense taken. I'm a little cranky because I wasted a lot of time in the past working on stuff that has been poorly done because of blinkered programmers.

And by the way, I didn't have any database courses in my industrial computing degree, and I think that even there, it's an error.
You can have to manage boatloads of data in about any IT application in any domain. I have a friend who works on industrial test benches, for instance. You need databases to collect and exploit the vast amount of data from the sensors.

I guess that in scientific/research/medical applications, there are likely lots of situations where you need to process/store/collate vast amounts of data. RDBMs should really be part of every IT cursus.

TomCo · 2006-05-22 Reply Admin

Eureka!

*R-E-A-L* WTF = unpadded numeric values in table names!

tables before:

infoset836
infoset84
infoset843

tables after:

infoset000084
infoset000836
infoset000843

Now they're in proper numeric order though it now makes this WTF not the least bit funny. :-P

Maybe we should just keep this app -as is- for "funny's" sake. ;)

Enterprise-TRIcorder-SQL>

select
wtfery.planet,
count(distinct wtfery.life_signs) as distinct_wtf_total
from
(
 /* hopefully, all of these tables have the same layout */
 /*
 also, note that we've don't have a place to note the
 solar system - we'll need another set of tables
 for that.
*/
 select 'mercury' as planet, * from planet1 union all
select 'venus' as planet, * from planet2 union all
select 'earth' as planet, * from planet3 union all
select 'mars' as planet, * from planet4 union all
select 'jupiter' as planet, * from planet5 union all
select 'saturn' as planet, * from planet6 union all
select 'uranus' as planet, * from planet7 union all
select 'neptune' as planet, * from planet8 union all
select 'pluto' as planet, * from planet9 union all
select 'popeye' as planet, * from planet10
 /*
 hopefully there will be no more
 discoveries of new heavenly bodies
 anytime in the future
*/
) wtfery
where
wtfery.application = 'enterprisey'
and
wtfery.intelligence is null
group by
wtfery.planet
;

 planet distinct_wtf_total
 ------ ------------------
earth 326654 <-- that few? data incorrect?
 popeye 1

 2 record(s) selected.

2006-05-22 Reply Admin

that's not exactly a good idea. what if some clueless un-neutered programmer-wannabe is out of ideas and goes trawling in the net for ideas - and happens to find this brillant design?

2006-05-22 Reply Admin

frosty:
Anonymous:
Third year course at University of Pennsylvania. Not a required one at that.

At Northwestern University (the real one in Evanston, IL), there's only one DB class, it's optional (as are the compiler, OS, and networking classes; you pick 15 from a pool of classes), and it's taught by whoever gets stuck with it (atleast, 2 years ago this was the case).

That's the same situation here at Northeastern University (Boston).

chrismcb · 2006-05-22 Reply Admin

Anonymous:
Well, in that case the real WTF is the American education system. If nobody passes the fucking test, they should just fail everybody. I still have nightmares (no, really) about my Fluid Mechanics exams where the grades were between 1 and 5 (out of ten) and the 2 people with the 5s passed.

Well its not that simple. One of the main reasons to grade on a curve, is to protect the students from bad teachers. Was the reason the students failed, was because they failed to learn, or failed to be taught? Or did the teacher botch the test?

2006-05-22 Reply Admin

I work at a company that has grown tremendously over the past few years. Based on the size of our customers' data, we shouldn't still be using MySQL, but we probably still use MySQL because with all the data we have, it is difficult to transition to a better database.

We have so much data that MySQL would DIE if we did not break the tables up by customer id. The tables would be so large that they would not fit on one server. I can't speak to newer versions of MySQL, but the version we use does not support having one logical table located across several physical servers. This is a limitation that we had to deal with, and our solution was to break the data up into many different tables with the same schema, and name the tables based on customer id.

Personally, I think we should move to a real database, but the point is if you are in a situation where you must use MySQL (or any other database that can't support REALLY large tables), you may have to break your data up into many tables based on customer id and/or date, depending on the primary key.

ammoQ · 2006-05-22 Reply Admin

Anonymous:

We have so much data that MySQL would DIE if we did not break the tables up by customer id. The tables would be so large that they would not fit on one server.

This must be quite a lot of data you are having... it sounds insane to have a database of that size in MySQL.

Anyway, some companies put their databases on SANs, so they have no physical limit on the table size.

2006-05-22 Reply Admin

Zlodo:

I continuously pull my hairs because of some inter-assets references breaks because some sync issue occured. We have lock contentions and performance problems all over the place, and since all the content designers here have to go through this tool to export their data, performance is a productivity issue aswell. And don't get me started on race conditions and data corruption when something crashes.

With every post you make, I become more and more conviced that you and I work on the same game.

But, this sort of problem may be far more common across the game industry than I realize.

2006-05-22 Reply Admin

Anonymous:
This seems to be very common - even Wordpress.com (http://photomatt.net/2006/03/01/wordpress-and-lyceum/) does this. WTF aside, is there *any* real reason why this is a good idea?

Not trying to justify it or anything ... but... the only idea I could come up with would be to implement row-level access control on a customer by customer basis. Since you can't do this with SQL or Oracle, best you can do is ring-fence the data in seperate tables.

Saying that, you could even do that with views. I give up!

ammoQ · 2006-05-22 Reply Admin

Anonymous:
Anonymous:
This seems to be very common - even Wordpress.com (http://photomatt.net/2006/03/01/wordpress-and-lyceum/) does this. WTF aside, is there *any* real reason why this is a good idea?

Not trying to justify it or anything ... but... the only idea I could come up with would be to implement row-level access control on a customer by customer basis. Since you can't do this with SQL or Oracle, best you can do is ring-fence the data in seperate tables.

Saying that, you could even do that with views. I give up!

On Oracle, you can use Oracle Label Security to get that level of access control, if you have the required $$$

jrenaut · 2006-05-22 Reply Admin

"Déjà-vu" anyone?

2006-05-22 Reply Admin

Anonymous:
You can't even try to store a string data on a float. The compiler will stop you with a lovable error, and show warnings all around. This Is A God Thing.

No... If trying to store a string value in a float caused a bolt of lightning to strike you from the blue sky, that would be a God Thing.

impslayer · 2006-05-22 Reply Admin

Anonymous:
Zlodo:

I continuously pull my hairs because of some inter-assets references breaks because some sync issue occured. We have lock contentions and performance problems all over the place, and since all the content designers here have to go through this tool to export their data, performance is a productivity issue aswell. And don't get me started on race conditions and data corruption when something crashes.

With every post you make, I become more and more conviced that you and I work on the same game.

But, this sort of problem may be far more common across the game industry than I realize.

Zip and Zlodo? Sounds you're IN the same game...

2006-05-22 Reply Admin

How would the professor know that he had created a test that an average student should score 75% on? He'd have to give the test to hundreds of people of known skill level before giving it to the class. Curving is a far more useful solution.

He'd know by using similar material to what he had used the previous year. It doesn't matter if the median score isn't 75 - what matters is that the test covers the appropriate material. Students aren't owed a grade.

Gene Wirchenko · 2006-05-22 Reply Admin

Anonymous:
How would the professor know that he had created a test that an average student should score 75% on? He'd have to give the test to hundreds of people of known skill level before giving it to the class. Curving is a far more useful solution.

He'd know by using similar material to what he had used the previous year. It doesn't matter if the median score isn't 75 - what matters is that the test covers the appropriate material. Students aren't owed a grade.

Sure they are. They pay tuition, go through the course, and get a grade.

While I prefer no adjustments, what do you do if the question is incorrect? I had a midterm like that. The instructor reviewed the midterm with each of us. He noted to me that no one in the class had gotten a certain question right. He thought he was asking which of the following six answers was an invalid variable name. A pity the question said "datatype" instead.

Sincerely,

Gene Wirchenko

2006-05-22 Reply Admin

Gene Wirchenko:
what do you do if the question is incorrect?

You invalidate the question entirely and give extra points to any students that wrote "This question has no meaningful answer" on their test.

Students that spent the entire test period banging their heads trying to answer an invalid question, at the expense of looking at other questions, will deservedly fail. Life is full of such situations.

2006-05-22 Reply Admin

Sure they are. They pay tuition, go through the course, and get a grade.

Sorry, I was being unclear. They don't deserve a passing grade.

While I prefer no adjustments, what do you do if the question is incorrect?

Mark it as such and tell the instructor separately.

2006-05-22 Reply Admin

Anonymous:
Anonymous:
This seems to be very common - even Wordpress.com (http://photomatt.net/2006/03/01/wordpress-and-lyceum/) does this. WTF aside, is there *any* real reason why this is a good idea?

Not trying to justify it or anything ... but... the only idea I could come up with would be to implement row-level access control on a customer by customer basis. Since you can't do this with SQL or Oracle, best you can do is ring-fence the data in seperate tables.

Saying that, you could even do that with views. I give up!

Yes, it is quite easy to do that with views. I posted some examples of how this with PostgreSQL at another forum.

2006-05-23 Reply Admin

I have about 4-5 years of experience with in scientific application codebases. In my experience, databases really aren't used by the scientific computational community outside of bioinformatics. For the most part, everyone uses fread, fwrite, printf, and scanf to read-write their data files. Data is organized by file structure. I've worked on 3 different projects that generated so many files that 'ls' would take minutes to run.

Scietific computing lags about 5-10 years behind the newest software engineering trends. Only now are people abandoning FORTRAN in favor of object oriented languages.

I've considered sending in some of the FORTRAN wtfs I've seen in my day. Who doesn't love argument lists with 500 parameters?

unlisted_error · 2006-05-23 Reply Admin

Gene Wirchenko:
Alex Papadimoulis:
Instead of using "CustomerId" columns on several of the tables, the application uses ... tables ... with the CustomerId appended to the end of the table. I'll leave it as an exercise for the reader to envision the query needed to aggregate all "infoset" tables.
One long database query maybe, but many calls to the directory functions to get the table filenames. Since the latter call do not get considered, the design is considered efficient. Is that about it?
Sincerely,

Gene Wirchenko

This has got to be a troll. Why not go and extoll the virtues of the design while your at it. Seems some people just want to start flame wars for the fun of it.

_unlisted_error_

unlisted_error · 2006-05-23 Reply Admin

Anonymous:
"cubicle mate"

That in itself is a WTF, as if cubicles are not bad enough when you are the only person in one.

Where I work we sit four people to a cubicle. Makes for a "chummy" atmosphere. "Hey, move your elbow, I need to grab that book over there."

_unlisted_error_

Gene Wirchenko · 2006-05-23 Reply Admin

unlisted_error:
Gene Wirchenko:
Alex Papadimoulis:
Instead of using "CustomerId" columns on several of the tables, the application uses ... tables ... with the CustomerId appended to the end of the table. I'll leave it as an exercise for the reader to envision the query needed to aggregate all "infoset" tables.
One long database query maybe, but many calls to the directory functions to get the table filenames. Since the latter call do not get considered, the design is considered efficient. Is that about it?

This has got to be a troll. Why not go and extoll the virtues of the design while your at it. Seems some people just want to start flame wars for the fun of it.

No, it is not. I am just illustrating the mindset of some. A factor that they do not consider has no cost. This is commonly known as "Out of sight, out of mind."

Sincerely,

Gene Wirchenko

jsmith · 2006-05-23 Reply Admin

Anonymous:

How would the professor know that he had created a test that an average student should score 75% on? He'd have to give the test to hundreds of people of known skill level before giving it to the class. Curving is a far more useful solution.

He'd know by using similar material to what he had used the previous year. It doesn't matter if the median score isn't 75 - what matters is that the test covers the appropriate material. Students aren't owed a grade.

I'm not saying students are owed a grade. I'm saying that it's very difficult to construct a test that B students get a B on and C students get a C on. When a class only has two tests, it's important to get it right. There is far more variability in the difficulty level of a test (even when a professor tries to make it a specific difficulty) than there is variability in the skill level of the average student. For a test given the first time, the most accurate way to grade it is on a curve. After a test has collected a few hundred result sets, then an absolute grading scale can be constructed. It is also best to put a few questions in the test that are well-known questions to guage the skill level of the target population.

Projects are a different story. They are a more thorough and unbiased guage of skill, but much harder to grade. A's and F's are pretty easy, but the grades in the middle are hard to differentiate. It also takes a long time to grade projects from 150 students in 5 different classes (all of different subjects/levels). I've worked places where grades were required to be reported back to the student within a week and the week before finals was when all projects were due. You could easily put in a 100 hour week grading the projects. BTW, grading is unpaid work.

impslayer · 2006-05-24 Reply Admin

jsmith:

You could easily put in a 100 hour week grading the projects. BTW, grading is unpaid work.

100 hours unpaid work in one week, now THAT sounds like a major (MAJOR) WTF...

WTF?!?!?!?

(Hang on, you mean with the grading your 'normal' week got up to 100 hour... Sounds more plausible, still a major WTF though...)

2007-11-27 Reply Admin

It can actually make some sense in a mult-tenancy point of view for a SaaS company's database for a particular application if each customer is allowed to customize their schema. (If more limited customization was needed than extension tables or adding columns per tenant could be done in conjunction with a tenant ID column)

Its alot more efficient in terms of scalability (on the provider side) than having a database for each tenant since concurrent database connections are limited.

Just use some meta-data and a service interface to generate the queries and pass them along to the Db.

I've only seen a few posts here that actually recognize where this could be used.

Though using my described system properly is a alot of work in itself (and requires alot of specialized knowledge to really optimize it for scalability and ease-of-use)

2021-10-06