Comment On We Don't Need No Stinkin' Database

J.M.'s company landed a contract to fix up a mid-sized university's student ID card scanning system that had just finished being rolled out. It was a few-year deployment process, first starting with the dormitories, then moving on to the copy machines, cafeteria, local businesses, and, finally, student events. Unfortunately, it hadn't been a very smooth ride, with the final straw being the original vendor giving the IT Director the choice of "2-3 card scans" or "nightly batch scan processing." The director chose to bring in another company. [expand full text]
« PrevPage 1 | Page 2 | Page 3Next »

Re: We Don't Need No Stinkin' Database

2006-03-16 15:03 • by JS
Which would you rather be: The company that got it right, or the company that made (I'm assuming) several times the money?

Re: We Don't Need No Stinkin' Database

2006-03-16 15:04 • by no name
sqrt(1^2) + 1 (damn captcha)

:D

Re: We Don't Need No Stinkin' Database

2006-03-16 15:05 • by Cooper
Not getting it - the WTF is a poorly written obsolete system???

OIC

WTF??

Re: We Don't Need No Stinkin' Database

2006-03-16 15:05 • by bramster
It wasn't Purdue, by any chance?


Re: We Don't Need No Stinkin' Database

2006-03-16 15:05 • by Gene Wirchenko
64261 in reply to 64257
Anonymous:
Which would you rather be: The company that got it right, or the company that made (I'm assuming) several times the money?


The former.  I find that having a reflection in a mirror helps greatly with shaving.

"middleware" seems too often to have the wrong initial vowel.  "muddleware" would be a better fit.

Sincerely,

Gene Wirchenko

Re: We Don't Need No Stinkin' Database

2006-03-16 15:07 • by mrsticks1982
64262 in reply to 64257

Anonymous:
Which would you rather be: The company that got it right, or the company that made (I'm assuming) several times the money?


 


I would like to be the company that got it right cause when the university needs some maintenance or other systems created then I will be the company that made several times more money.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:07 • by connected
Mother of g*d! Now I understand why I'm having trouble getting into the building every day...and all this time I just though the guards didn't like me.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:07 • by Colin
I've been reading this site for a while now and pretty much every
posting has this in common: they either don't know of a tool or if they
do then they don't know how to use it.



But both paths can be quite comical though.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:08 • by R.Flowers

Wow; a company that lands a contract for an enterprise-level job, and they seem either ignorant of the advantages of a database, or suspicious of using one.


I would like to see the inside of one of those 'text_files'.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:08 • by synesthetia
64267 in reply to 64260
I would hope it wasn't Purdue.  The students there could do 10X better than this.

That is, however, exactly what this sounds like.  It sounds like the quality of work you would get with a mediocre school's senior project.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:08 • by Pastor_Of_Muppets
64268 in reply to 64257
Many genuinely frightening things happen with these school card systems.  I was still in school in 2003 when my school rolled out it's new ID card.  I took mine to work, ran it through a magstripe reader, and low and behold, the only thing on the card was my student id number, and a counter so it read something like 99999999901.  The student id is printed on the card itself, and not something that is kept secret because it was no longer the SSN.  If the student lost their card, the system was 'secure' because the new card was issued as 99999999902.  You've gotta love that security.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:09 • by mrsticks1982
Good lordy, how would they explain this to other places they were going to work ..Hey we can code just don't expect any new fancy, I make you life easier stuff!!!!

Re: We Don't Need No Stinkin' Database

2006-03-16 15:10 • by haveworld
64270 in reply to 64257
Anonymous:
Which would you rather be: The company that got it right, or the company that made (I'm assuming) several times the money?


the company that got it right because reputation is very important in the long term and could lead to highly paid contracts in the future.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:10 • by Ken Nipper
Were the original "programmers" former students?

Re: We Don't Need No Stinkin' Database

2006-03-16 15:11 • by Pastor_Of_Muppets
64273 in reply to 64268
s/it's/its

Re: We Don't Need No Stinkin' Database

2006-03-16 15:13 • by connected
64275 in reply to 64265

R.Flowers:
I would like to see the inside of one of those 'text_files'.


I don't think there was anything in the files themselves...they just represent values that SHOULD have been inserted into columns...in a table...in a [fill in the blank here].

Re: We Don't Need No Stinkin' Database

2006-03-16 15:14 • by emurphy
64276 in reply to 64257
Anonymous:
Which would you rather be: The company that got it right, or the company that made (I'm assuming) several times the money?


The former.  They have a good reference and a future source of income.  Meanwhile, the latter has killed their golden goose and damaged their credit with the goose vendor.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:15 • by R.Flowers
64277 in reply to 64264

Anonymous:
I've been reading this site for a while now and pretty much every posting has this in common: they either don't know of a tool or if they do then they don't know how to use it.


You've heard "to a hammer, every problem's a nail." Well, this company has a nail, but no clue what a hammer is.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:16 • by frosty
64278 in reply to 64257
And I took all of that stuff for granted when I was going to school.

The bit about the students figuring out that debits weren't processed until the next day is funny.  The closest we had was an ATM that spit out $10s and $20s and some dolt put $20s in the $10 container.  People were allowed to do 3 withdraws before going to the end of the line (well, until the party was broken up).

Re: We Don't Need No Stinkin' Database

2006-03-16 15:17 • by jackass
64279 in reply to 64257
It depends, if work was scarce in my area I would rather be the one that made more money.  You can always pack up, move on and leave your reputation behind.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:21 • by R.Flowers
64282 in reply to 64275
connected:

R.Flowers:
I would like to see the inside of one of those 'text_files'.


I don't think there was anything in the files themselves...they just represent values that SHOULD have been inserted into columns...in a table...in a [fill in the blank here].



I'm assuming the numbers are file names (that roughly correspond to a 'primary key'), and the contents would be column names - values represented in some way. For example, in the file '3040557', the content (in accounts/balances, for example) might be:


balance=124.75
last_transaction_date=03/04/2005


or maybe just


124.75
03/04/2005


Ken Nipper:
Were the original "programmers" former students?


I hope students would do better.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:23 • by emurphy
64283 in reply to 64279
jackass:
It depends, if work was scarce in my area I would rather be the one that made more money.  You can always pack up, move on and leave your reputation behind.


Well, at least you're honest.  Remarkably so, given your username.  :D

Re: We Don't Need No Stinkin' Database

2006-03-16 15:33 • by ParkinT
Overall the University would have been better off if they let the students write the system.  Make a part of large project in a software development or engineering class!

Re: We Don't Need No Stinkin' Database

2006-03-16 15:34 • by Ann Coulter
64285 in reply to 64283
emurphy:
jackass:
It depends, if work was scarce in my area I would rather be the one that made more money.  You can always pack up, move on and leave your reputation behind.


Well, at least you're honest.  Remarkably so, given your username.  :D



There's nothing contradictory about an honest jackass.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:34 • by JS
64286 in reply to 64276
emurphy:
The former.  They have a good reference and a future source of income.  Meanwhile, the latter has killed their golden goose and damaged their credit with the goose vendor.
Ah, but the incompetent company only has to sucker people into buying their services every once in a while to out-earn the other company. A good reputation can help, no doubt, but reputations don't carry forever.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:39 • by Bahamas Boy
64287 in reply to 64262








 Anonymous wrote:




Which would you rather be: The company that got it right, or the company that made (I'm assuming) several times the money?


 


I would like to be the company that got it right cause when the university needs some maintenance or other systems created then I will be the company that made several times more money.


 


Good for you. But now, every day *I* go lauging all the way from the beach to the bank...

Re: We Don't Need No Stinkin' Database

2006-03-16 15:41 • by filesalot
Of course a DBMS is the right tool for this job, but I don't think you can lay the problems of this system on the choice

of flat files in a directory tree.



You can go pretty far with a directory tree (it is a tree index, after
all) as long as you keep the size of the directories small.

For example, if user id #0123456789 was kept in the file



    01/23/45/67/0123456789



Wasteful of space, but not horribly slow.  Subsecond access should
be no problem at all.  And you could distribute this system nicely
by user id, eg the last digit of the id is the backend server number,
0-9, to send the scan to, so there need be no problems with immediately
debiting the user account.



Sounds like the performance problem was due to the muddleware.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:42 • by Krenn
My personal guess...

"Wait a minute, we have to pay a licensing fee to use a database in production?  Screw that, we can implement one on our own."

Re: We Don't Need No Stinkin' Database

2006-03-16 15:42 • by SJ
64290 in reply to 64257
The latter, because everyone else chose the former and I'm cool like that.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:44 • by SJ
64291 in reply to 64290
Anonymous:
The latter, because everyone else chose the former and I'm cool like that.

 

Dammit, Replied instead of quoted.

 

JS:
Which would you rather be: The company that got it right, or the company that made (I'm assuming) several times the money?


The latter, because everyone else chose the former and I'm cool like that.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:45 • by Roger

Alex Papadimoulis:
For being all computerized, it worked slowly, taking about a second or two per scan. When used at student events, it would take 3-5 seconds per scan, if it worked at all. This was a bit of an issue.


I am almost certain that the clients never specified the response time in the original specifications.


Poor specifications == poor product == client's fault.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:45 • by IRRePRESSible
64293 in reply to 64267
Purdue Chicken has a University WTF!

Re: We Don't Need No Stinkin' Database

2006-03-16 15:45 • by Sean
The real WTF is that this is how Visual Sourcesafe worked up until the VS 2005 release.

Re: We Don't Need No Stinkin' Database

2006-03-16 15:53 • by fzammetti

Hmm... where does SQL Server store its data?  Where does Oracle store its data?  I thought the existence of the Ether was disproven almost a century ago, and yet modern RDBMS's seem to be storing their data there...


Oh no, wait, they store it ON THE FILE SYSTEM!  Wow, revelation!


I say we blame the OS vendors for this whole quagmire... obviously, if they had written better-performing disk IO code, this never would have been an issue.


(Sorry, forgot to wrap that all in <sarcasm> tags)

Re: We Don't Need No Stinkin' Database

2006-03-16 15:54 • by cvi
64296 in reply to 64284
Apparently problems with card-scanner-systems are kinda common. "My"
uni had one building all students (some odd 11000) were supposed to
have access to. The system was unable to handle that amount of people
(in a single node/scanner?).



Anyway, the problem was resolved by only giving students with "special reasons" access. (The doors were open daytime, though).

Re: We Don't Need No Stinkin' Database

2006-03-16 16:02 • by loneprogrammer
64297 in reply to 64288
Anonymous:


You can go pretty far with a directory tree (it is a tree index, after
all) as long as you keep the size of the directories small.

Well, that's the key.  Small directories.  That's why this system worked at first, then started slowing down as the directories grew larger.  Nobody saw that they had a bad design until it was too late.

Directories with >10,000 files in them are slow because they are usually stored unsorted, and must be searched linearly.

Putting the data into a DBMS makes it easy to keep the data sorted.  All you have to do is add an index and you'll be able to handle a lot more data, without having to resort to tricks to keep your directories small.

Re: We Don't Need No Stinkin' Database

2006-03-16 16:05 • by Disgruntled DBA
The best part was making the users swipe all of the readers with punch-cards.  Who needs these fancy whatsits with the magneto thingies.  That is sooo 1950's.

Re: We Don't Need No Stinkin' Database

2006-03-16 16:09 • by Runtime Error
64299 in reply to 64292
Anonymous:

Alex Papadimoulis:
For being
all computerized, it worked slowly, taking about a second or two per
scan. When used at student events, it would take 3-5 seconds per scan,
if it worked at all. This was a bit of an issue.


I am almost certain that the clients never specified the response time in the original specifications.


Poor specifications == poor product == client's fault.





So you are saying that the client has to specify that the vendor has to
build something that "isn't retarded" in the specifications? 





Re: We Don't Need No Stinkin' Database

2006-03-16 16:13 • by Another Moose
64301 in reply to 64292

Anonymous:

I am almost certain that the clients never specified the response time in the original specifications.


Poor specifications == poor product == client's fault.


The customer's job is to tell us what they need.
Our job is to work out how to give them something that will actually do what's necessary.
We write the specifications. We create the product. If the product makes the customer unhappy then it's our fault because our job isn't to do the bare minimum we can get away with.


If the customers don't know that they need to decide on an acceptable minimum response time, then a responsible professional would tell them so.

Re: We Don't Need No Stinkin' Database

2006-03-16 16:14 • by [Si]dragon
64302 in reply to 64273
Pastor_Of_Muppets:
s/it's/its




Nah, you got it wrong.  An apostrophe token means "here comes an 's'!!"

Re: We Don't Need No Stinkin' Database

2006-03-16 16:17 • by Another Moose
64304 in reply to 64257

Anonymous:
Which would you rather be: The company that got it right, or the company that made (I'm assuming) several times the money?


I want to be the one who has the greater take-home pay, with which to buy stuff for myself.


The company that charged less probably also required less actual work in order to get the system up and running, which means lower overheads, and less income going on expenses. Revenue doesn't matter - the real question is who made more profit.

Re: We Don't Need No Stinkin' Database

2006-03-16 16:29 • by Jeff S
64306 in reply to 64292
Anonymous:

Alex Papadimoulis:
For being all computerized, it worked slowly, taking about a second or two per scan. When used at student events, it would take 3-5 seconds per scan, if it worked at all. This was a bit of an issue.


I am almost certain that the clients never specified the response time in the original specifications.


Poor specifications == poor product == client's fault.



blaming the client for bad specs == making lame excuses == a really, really bad consultant


Re: We Don't Need No Stinkin' Database

2006-03-16 16:31 • by DiamondDave
64307 in reply to 64257
Anonymous:
Which would you rather be: The company that got
it right, or the company that made (I'm assuming) several times the
money?




I'd much rather be the company that got it right, as that may help to guarantee future contracts...

Re: We Don't Need No Stinkin' Database

2006-03-16 16:33 • by DiamondDave
64308 in reply to 64267
synesthetia:
I would hope it wasn't Purdue.  The students there could do 10X better than this.

That
is, however, exactly what this sounds like.  It sounds like the
quality of work you would get with a mediocre school's senior project.




Just don't expect them to know what "hexidecimal" is...

Re: We Don't Need No Stinkin' Database

2006-03-16 16:35 • by filesalot
64309 in reply to 64297
Right, this should be in a DBMS no question.



But it _could_ have been designed so that the directories never had
more than 100 entries in them, no tricks required, just more
directories in the tree from the start.



Re: We Don't Need No Stinkin' Database

2006-03-16 16:44 • by Anonymoose
64310 in reply to 64297
loneprogrammer:
Anonymous:


You can go pretty far with a directory tree (it is a tree index, after
all) as long as you keep the size of the directories small.

Well,
that's the key.  Small directories.  That's why this system
worked at first, then started slowing down as the directories grew
larger.  Nobody saw that they had a bad design until it was too
late.

Directories with >10,000 files in them are slow because they are usually stored unsorted, and must be searched linearly.

Putting
the data into a DBMS makes it easy to keep the data sorted.  All
you have to do is add an index and you'll be able to handle a lot more
data, without having to resort to tricks to keep your directories small.





Well, to reduce the number of files in each door directory, they should
have put all the valid IDs into a text file, instead of one file for
each student, and then used some sort of O(1) regex to search the text
file! ;)

Re: We Don't Need No Stinkin' Database

2006-03-16 16:51 • by tim
gotta love relational batch files, too

some OS's slow down once you have a few thousand files in a single folder. I remember one system I saw with about 75000 files in a folder (again, a folder/file-based 'database') and that was slow to read or write, had file locking issues and generally got replaced as soon as humanly possible.

the "xml->text->process->xml->text" is fancy :)

Re: We Don't Need No Stinkin' Database

2006-03-16 16:54 • by Woody
64312 in reply to 64301
Anonymous:

Anonymous:

I am almost certain that the clients never specified the response time in the original specifications.


Poor specifications == poor product == client's fault.


The customer's job is to tell us what they need.
Our job is to work out how to give them something that will actually do what's necessary.
We write the specifications. We create the product. If the product makes the customer unhappy then it's our fault because our job isn't to do the bare minimum we can get away with.


If the customers don't know that they need to decide on an acceptable minimum response time, then a responsible professional would tell them so.



It's amazing howoften marketing doesn't understand that.  And that Engineering really needs to treat marketing like that.

"Yes, I hear you say you want X.  What that actually means is that you want Y.  Trust me on this."

They seem to hate it when they remember that you know more about writing code/designing systems than them.

Re: We Don't Need No Stinkin' Database

2006-03-16 16:59 • by chaim79
64313 in reply to 64284

ParkinT:
Overall the University would have been better off if they let the students write the system.  Make a part of large project in a software development or engineering class!


Right.... that would be great, each student on the project would have their own backdoor/easteregg/prank on the system. it may run faster, better, etc, but it'd be worthless as far as the programmers, their friends, their friends friends, the girl in the next dorm... etc


Yah... that'd be great....


Erik of Ekedahl

Re: We Don't Need No Stinkin' Database

2006-03-16 17:15 • by Derek
64315 in reply to 64313
chaim79:
ParkinT:
Overall the University would have been better off if they let the students write the system.  Make a part of large project in a software development or engineering class!

Right.... that would be great, each student on the project would have their own backdoor/easteregg/prank on the system. it may run faster, better, etc, but it'd be worthless as far as the programmers, their friends, their friends friends, the girl in the next dorm... etc


Yah... that'd be great....


Erik of Ekedahl

I studied at FSU in Jena, Germany for a semester a couple of years ago.  They actually did have their own chipcard system produced by their CS dept.  They had a few professors in charge of the thing, but the bulk of it (as I understand) was written by students.  It worked great.  It was deployed across the campus, and I never had any problems, nor heard of anyone who did.

Backdoors/easter eggs/etc?  Do you really think that anyone would be foolish enough to take code from random students and put it in a production system without checking it first?  Besides, you'd presumably only allow students to work on it that 1) you trust and 2) understand that inserting a backdoor into real software is typically illegal.  Illegal electronic entry, criminal negligence, etc.

« PrevPage 1 | Page 2 | Page 3Next »

Add Comment