Comment On Manager of the Data Dump

Please do not add comments/feedback about HiddenNetwork.com to this thread. You can add comments to the HiddenNetwork.com discussion here: http://thedailywtf.com/forums/thread/94341.aspx.

And sorry about the ad that will appear right below this text. That's a bug. It's only supposed to show under the articles.

Snotty and stupid... that's my favorite kind of dev.

When I want to muck about with CSV data in a database, I'll start out with varchar 255 for every field.

I then immediately analyze the data to figure out the correct datatypes for each row, convert, and add indexes.

Clearly, that analysiss stage is just a waste of time!
 

The SQL standard needs to be changed so that its more difficult to use.  It should be made difficult to design tables without Primary/Foreign keys.  I've seen far too many of them.

I was going to comment on the WTF.

.... and then I got distracted by certain, ahem, parts of the BSG ad to the right.

Sorry guys.

Alex Papadimoulis:

J.T.,

Fine,  <snip>

 

....would have the point at which I called him out for his attitude.

 

 

Roooh lovely, all fields as strings, I can't wait to see what the code that typecasts all afterwards looks like :-)

 

Captcha: awesomeness as in brillant

Anonymous:

Roooh lovely, all fields as strings, I can't wait to see what the code that typecasts all afterwards looks like :-)

 

Captcha: awesomeness as in brillant

Wait a minute, you mean there is a company out there with a real DBA that has enough power to actually enforce the rules?  I don't think we will get to many WTF's from that company at all if the rest of the team is like this.

 P.S. Is this company one of the ones listed on HiddenNetwork.com?

P.P.S. Hmm BSG girls.  And both of these were cylons.  Kinda like an everyready girlfriend. She just keeps...  oh sorry wrong blog.

Seems to me like things are working just fine.  And, as long as the DBA keeps being the janitor, things will continue to work fine. 

DBAs crack me up.  They're like, if a garbage man really thought he was a sanitation engineer, and ran around saying, "If I didn't have to pick up after all these people, I could get some engineering done."

  

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

XH:

The dev is probably a PHP-tard, so no cast is necessary.  I find this sort of thinking to be very common across the PHP developer-base.

No, it said in the original post that he was a ".NET Wizard".

Captcha: giggity.  As in the BSG girls... Giggity, giggity goo!

DBA's sucks, they only try to keep us developers down. Therefore, modern databases supports XML as column type so you can use beautiful XML for all your needs.

CREATE TABLE [DtxSurveys] 
  ([DataInfo] XML NULL)

Anonymous:

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

 

I'm absolutely speechless!  Data integrtity is the job of the database, it's what they are designed to do.  Why write a lot of code to handle it in your app when it's taken care of for you if you use the database as you are meant to?  Indexes should be used on any fields you are going to be searching on regularly, otherwise you have a load of full table scans going on for simple queries which will kill performance.

 (Captcha: bedtime, why yes, I think it is!)
 

<sarcasm>

 Bah! Everyone knows that putting restraints on data being put into a database does nothing more than irritate the developer. It's not like those restraints are ever used for anything useful! Also, a 3800% increase is NOTHING. It was probably a mistake in the timing mechanism. A more believable value would be something like a 105% increase. There's just no way that it increased the runtime by that much! Bah! It's all J.T.'s fault for having silly DB restrictions and guidelines!!!

</sarcasm>
 

I believe you mean "decreased the run time to 52 milliseconds", instead of "increased".

As a DBA, I see a lot of junk like your Frank examples.  A LOT.

Best example I can think of off the top of my head is a query that went something like this:

select OrderID, DateOrdered, OrderQty
from Orders
where dateordered between '1/1/2006' and '31/1/2006'
union all
select OrderID, DateOrdered, OrderQty
from Orders
where dateordered between '2/1/2006' and '2/31/2006'
...
where dateordered between '6/1/2006' and '6/31/2006'

All done as Unions, 1 per month (and yes, the original query request included February 31st; it was obviously a copy-and-past job).  The reason?  "We want it to be in order by month."

I, for some reason, felt that this might work better:

select OrderID, DateOrdered, OrderQty
from Orders
where dateordered between '1/1/2006' and '6/30/2006'
order by dateordered

D. Skinner:

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

 Next week's WTF brought to you courtesy of one D. Skinner... ;)
 

Let me follow up my previous comment, re-reading the comment I posted to, D Skinner claimed that keys introduce overhead:  Trust me, your app-based data integrity code will be orders of magnitude slower than that of all modern RDBMS.  The people who develop databases are some of the best and brightest there are (they have to be, looking after other people's data is a damn big responsability).

 

(Captcha: paula (is brilliant)) 

Anonymous:

Data integrity is the responsibility of the app, not of the database.

That may be workable in a standalone application, but when you have a database being used by multiple applications concurrently, the database becomes the only place that you can reliably enforce data integrity. Same goes for ad-hoc dynamic queries vs. stored procedures and tight security / access rules.
 

Anonymous:

Anonymous:

Roooh lovely, all fields as strings, I can't wait to see what the code that typecasts all afterwards looks like :-)

 

Captcha: awesomeness as in brillant

The dev is probably a PHP-tard, so no cast is necessary.  I find this sort of thinking to be very common across the PHP developer-base.

Hell, I remember being shocked to even discover that in intrinsic cast from smallint to int was a performance killer. I cringe to think of this mess. 

I love the "That's how we're doing it in dev." part.

I admit that I - as a PHP dev :P - may have done (it's better nower days ;) some horrible database designs but never ever have I started out with all columns being varchars.

Anonymous:

Seems to me like things are working just fine.  And, as long as the DBA keeps being the janitor, things will continue to work fine. 

DBAs crack me up.  They're like, if a garbage man really thought he was a sanitation engineer, and ran around saying, "If I didn't have to pick up after all these people, I could get some engineering done."

  

 

Please use your name, so I can pre-screen any future employers for your presence.

 

and appropriately, the CAPTCHA is : hacker 

 

I doubt the DBA here would catch the WTF's in the table definition or the Select statment.

No, let me correct that -- not a chance she would catch it. But she does check to make sure the FK and Index names on MS SQL Server have all been changed from MS default (FK_xxx, IX_xxx) to Oracle standard (xxx_FK, xxx_IX).

Thank goddness I'm a contractor and about out of here! Hopefully J.T. will be at the next location, sounds like a DBA that knows his stuff.

Ahh.  Another arrogant, stuck-up, amateur to be broken in.

 

Drill Sergeant:  Holy God!  Private Pyle!  What is this in your table definition?

Private Pyle:  Sir, it is a varchar(8000) field, Sir!

Drill Sergeant:  And are you allowed to have varchar(8000) fields in your table definitions, Private Pyle?

Private Pyle: Sir, no Sir!

Drill Sergeant:   Why is that Private Pyle?

Private Pyle:  Sir, because my tables are too wide, Sir!

Drill Sergeant:    Because your tables are disgusting fatbodies!

Anonymous:

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

I remember a recent boss who had that "the app will maintain data integrity" philosophy. Needless to say, the data was a giant turdball. Too bad all the major database players have these key things that kill performance. What were they thinking? why didn't they consult someone like you before they went and did all of that?

 

I think (god let us hope...) he was being sarcastic. 

Anonymous:

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

 Err...What?  Have you ever used a database?  I hope there was scarcasim there, but I didn't detect it.

 Keys create some overhead on inserts, but if you ever really want to use the data you're storing it surely wouldn't be a good idea to do an all rows scan for any sort of subset.

Anonymous:

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

 

"Data integrity is the responsibility of the app" hmmm?

RDBMS

The R stands for Relational. Taking all of the data integrity out of the database, which is optimized for it, and putting it into an app where the developer has to account for every possible change to the data, is not a grand idea. In fact, it stinks. Can you guarantee that ALL data changes will only take place through the application? If you can't then there is the possibility that data may change not according to the rules of the application.

Data integrity is absolutely the responsibility of the database AND the app. Neither should work alone. The database should enforce some rules and the application should enforce others.

 

-- J.T.

 

Disgruntled DBA:

Ahh.  Another arrogant, stuck-up, amateur to be broken in.

 

Drill Sergeant:  Holy God!  Private Pyle!  What is this in your table definition?

Private Pyle:  Sir, it is a varchar(8000) field, Sir!

Drill Sergeant:  And are you allowed to have varchar(8000) fields in your table definitions, Private Pyle?

Private Pyle: Sir, no Sir!

Drill Sergeant:   Why is that Private Pyle?

Private Pyle:  Sir, because my tables are too wide, Sir!

Drill Sergeant:    Because your tables are disgusting fatbodies!

 

ROTFLMAO! 

I just printed that out and posted it on my wall.

And sent a copy to the data people.

You, Disgruntled DBA,  have made my day.

Anonymous:

Let me follow up my previous comment, re-reading the comment I posted to, D Skinner claimed that keys introduce overhead:  Trust me, your app-based data integrity code will be orders of magnitude slower than that of all modern RDBMS.  The people who develop databases are some of the best and brightest there are (they have to be, looking after other people's data is a damn big responsability).

(Captcha: paula (is brilliant)) 

First, hats off to J.T. - we need more of you in this business

Second, I respectfully disagree on this point. While some of the people who develop DBs are very likely extremely bright, you could arguably make the claim that some of the ones that aren't too bright also develop DBs. Some of the folks who create the WTF's featured in this forum (or similar thereto) undoubtedly work for Oracle/Sybase/...

We all trust the banks/stock-exchanges to safeguard our money, yet...

The same goes for database vendors.

It's probably a side effect of specialization and compartimentialization of jobs responsibility.

There's the team that know how to talk to the client.

There's the team that know how to generate code (.net wizard is appropriate here)

And there's the team that know how to design database (that team is on vacation)

It looks like in this case, the "developer" could have used a little help from the DBA to fine tune the design in 'dev'. Then again, because it works in 'dev' doe not mean it's good design. That also says something about peer reviews and set of skills held by the developer team. Maybe a whole bunch of apes hitting the keyboard. A lucky one could come up with the Shakespeare's database.

 Captcha: chocobot
 

JustThat:

Anonymous:

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

 

"Data integrity is the responsibility of the app" hmmm?

RDBMS

The R stands for Relational. Taking all of the data integrity out of the database, which is optimized for it, and putting it into an app where the developer has to account for every possible change to the data, is not a grand idea. In fact, it stinks. Can you guarantee that ALL data changes will only take place through the application? If you can't then there is the possibility that data may change not according to the rules of the application.

Data integrity is absolutely the responsibility of the database AND the app. Neither should work alone. The database should enforce some rules and the application should enforce others.

 

-- J.T.

 

I kind of agree with you here but not totally.  The RDBMS should enforce ALL rules.  The app can enforce these rules also.  For example, set a field to NOT NULL so the database requires an entry, and the app should verify there is data before trying to save.  Why?  Like you said, you can't be sure the update will always happen in the application but why make the database work harder than it needs to when you can easily check for proper data in your business layer.

ISNULL == Frank

Anonymous:

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

Each sentence is a little WTF in itself.
 

Anonymous:

It's probably a side effect of specialization and compartimentialization of jobs responsibility.

There's the team that know how to talk to the client.

There's the team that know how to generate code (.net wizard is appropriate here)

And there's the team that know how to design database (that team is on vacation)

It looks like in this case, the "developer" could have used a little help from the DBA to fine tune the design in 'dev'. Then again, because it works in 'dev' doe not mean it's good design. That also says something about peer reviews and set of skills held by the developer team. Maybe a whole bunch of apes hitting the keyboard. A lucky one could come up with the Shakespeare's database.

 Captcha: chocobot

Would these be the same folks who typed in the code and sent the printout to the client to re-enter, compile and deploy?

Anonymous:

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

 

Well it's not often the WTF makes me horrified TWICE!

Every single time I've seen a developer try to do something better than the database they end up doing it worse. This is not to say
you can't help the database. Give the database hints, help it perform better... but leaving data integrity up to the app?!

 CAPTCHA: error

Anonymous:

Since keys create overhead (sometimes a
lot), they should not be mandated or even favored by any
standards.  Frankly, I'm of the philosophy that keys are almost
always bad.  Data integrity is the responsibility of the app, not
of the database.  (Indexes, of course, are usually necessary for
large tables, but even they should be used sparingly.)

Come on, this IS a troll.  Nobody could be THAT clueless! 

OK,
I'll bite:  why in hell are you paying big bucks and a server for
a RDBMS if all you want is a flat file?  Get enterprisey and store
everything in a single XML file instead :-P  Imagine all thosepaid hours spent at coding the I/O routines and XML searching methods!  You could make meeelions!

 

J.T. should have used ISKINDANULL, ISMAYBENULL or at least ISVERYNULL

Anonymous:

JustThat:

Anonymous:

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

 

"Data integrity is the responsibility of the app" hmmm?

RDBMS

The R stands for Relational. Taking all of the data integrity out of the database, which is optimized for it, and putting it into an app where the developer has to account for every possible change to the data, is not a grand idea. In fact, it stinks. Can you guarantee that ALL data changes will only take place through the application? If you can't then there is the possibility that data may change not according to the rules of the application.

Data integrity is absolutely the responsibility of the database AND the app. Neither should work alone. The database should enforce some rules and the application should enforce others.

 

-- J.T.

 

I kind of agree with you here but not totally.  The RDBMS should enforce ALL rules.  The app can enforce these rules also.  For example, set a field to NOT NULL so the database requires an entry, and the app should verify there is data before trying to save.  Why?  Like you said, you can't be sure the update will always happen in the application but why make the database work harder than it needs to when you can easily check for proper data in your business layer.

I agree with what you said above but I was thinking more of the line between data integrity rules and pure business rules that may be difficult or impossible to implement within the database. For example, something like a ticketing system where a request needs to be signed of on by a manager before work can proceed. The database really isn't set up to do anything with that information. Sure, it can be stored as a value in the database but will the database know what it means? Not likely. It would take the front-end to look at the value in the database and ensure it meets the business rules before proceeding.

In general, I believe rules dealing with data integrity should always be as you stated: a partnership between the app and the database with the database acting as the gatekeeper and having the final say (i.e. the app can't override the database). Rules dealing with business process are sometimes best put in an application which uses data known to be "clean" because it is in a well-designed database with proper RDI.

 Does that make sense? Or am I talking out my Frank?
 

Anonymous:

Anonymous:

Since keys create overhead (sometimes a
lot), they should not be mandated or even favored by any
standards.  Frankly, I'm of the philosophy that keys are almost
always bad.  Data integrity is the responsibility of the app, not
of the database.  (Indexes, of course, are usually necessary for
large tables, but even they should be used sparingly.)

Come on, this IS a troll.  Nobody could be THAT clueless! 

OK,
I'll bite:  why in hell are you paying big bucks and a server for
a RDBMS if all you want is a flat file?  Get enterprisey and store
everything in a single XML file instead :-P  Imagine all thosepaid hours spent at coding the I/O routines and XML searching methods!  You could make meeelions!

 

I have joked that another development group we work with should have the motto: "We take the R out of RDBMS every day"

19 tables and not a primary key, foreign key or index in sight.
 

Just that,

Nope, you are talking Frankly.  I think we are in agreement there.

Leave integrety in the database supported by the application, but business rules are left to the business layer.

I have noticed that the OP for this topic hasn't returned.  I'm guessing he's checking his code to see how much may be getting submitted here in the near future.

Anonymous:

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

 

I hope I never work where you work. 

Anonymous:

Anonymous:

Since keys create overhead (sometimes a lot), they should not be mandated or even favored by any standards.  Frankly, I'm of the philosophy that keys are almost always bad.  Data integrity is the responsibility of the app, not of the database.  (Indexes, of course, are usually necessary for large tables, but even they should be used sparingly.)

 Err...What?  Have you ever used a database?  I hope there was scarcasim there, but I didn't detect it.

 Keys create some overhead on inserts, but if you ever really want to use the data you're storing it surely wouldn't be a good idea to do an all rows scan for any sort of subset.

 

This is just a recap of the mysql philosophy from a few years back, before they added transactions.

You mean you actually have data integrity?  At my work we have
to code data integrity, sorting, etc. and everything a REAL object
relational database should be doing for us.  This is because the
clients (users/supervisors) get to design the databases for the apps we
build for them.  I've gotten to suggest a few things a couple
times which happened to make sense to them like having a primary key or
having a field by numeric instead of varchar.  Apparently they
NEED to have every field updatable, so I just slip a sequenced field in
there for a primary key to make sure  hope it doesn't make my app blow up.

If we had a DBA like that with authority like that, I don't think I would care what they paid me. I would actually LOVE my job.

"That's how we're doing it in dev."

I'll store this sentence in my mental table of useful phrases. It deserves a place alongside "Works as coded." and "Beanbag girl forever!".

What's more, they all fit in a varchar(8000) field, nicely concatenated and separated with a pipe.

This thread perfectly illustrates a common conflict that can happen in the business world today, the DBA vs the developer.

You have the two extremes, of course.  On one hand, you have the DBA, who thinks the DB should drive the entire application, and all the developer does is write an application shell that calls the stored procedureds he has graciously provided.  The DB is law.

On the other hand, you have the developer.  He wants to be able to write code to have an application work.  The DB is a storage dump to persist his data.  Nothing more.  The Application is law.

Both of these approaches are just plain wrong.

You have to work from the middle.  In order for this to happen, you need DBAs who understand the developer's needs (such as enough flexibility to provide adequate search functionality, for example) and developers who understand the DBA's needs (that is, don't write SELECT N + 1 queries that trash performance).  Without this mediation, you have a huge bottleneck that can be very difficult to overcome.

All too often, you end up with one of the extremes.  In this WTF's case, you have the developer who could give a crap about silly things like data integrity, indexes, or those "key" things.

I'm sure there could be another set of WTFs out there for the other, that is, the all-too-controlling DBA who forces stored procudures, etc on to the development crew and the process involves jumping through 500 burning hoops to get any kind of work done.

biziclop:

"That's how we're doing it in dev."

I'll store this sentence in my mental table of useful phrases. It deserves a place alongside "Works as coded." and "Beanbag girl forever!".

What's more, they all fit in a varchar(8000) field, nicely concatenated and separated with a pipe.

And the real WTF..

Allowing users to post all on one line and avoid word wrapping by adding scrollbars so you have to keep scrolling left and right when they decide to actually add a carriage return.  Just plain silly.

Anonymous:

Wait a minute, you mean there is a company out there with a real DBA that has enough power to actually enforce the rules?  I don't think we will get to many WTF's from that company at all if the rest of the team is like this.

 P.S. Is this company one of the ones listed on HiddenNetwork.com?

P.P.S. Hmm BSG girls.  And both of these were cylons.  Kinda like an everyready girlfriend. She just keeps...  oh sorry wrong blog.

I used to work at a company where the DBA not only enforced rules but could also be invited for a coffee and a chat about how to solve a DB-related problem.

As a result, he got well-designed data structures, optimized  queries, and in turn I've got caffeine poisoning because a good DBA can consume enormous quantities of coffee. :)

(Anyway, a month's learning from him was worth a year at a university.)

Disgruntled DBA:

Ahh.  Another arrogant, stuck-up, amateur to be broken in.

 

Drill Sergeant:  Holy God!  Private Pyle!  What is this in your table definition?

Private Pyle:  Sir, it is a varchar(8000) field, Sir!

Drill Sergeant:  And are you allowed to have varchar(8000) fields in your table definitions, Private Pyle?

Private Pyle: Sir, no Sir!

Drill Sergeant:   Why is that Private Pyle?

Private Pyle:  Sir, because my tables are too wide, Sir!

Drill Sergeant:    Because your tables are disgusting fatbodies!

 
8000 chars is beau coup data. How about 255 for all of them?