• Rob (unregistered)

    A couple of comments on cleaning up this mess (hey it keeps you employe-ed!)

    1. what things do you have to have in your target system? What records do you have to have, which columns have to be completed. Focus on getting the required things filled in, then save everything remaining as a memo.
    2. load the stupid excel spreadsheets into staging tables, geez there really is not a decision. (I'd recommend loading everything as char data with maximum lengths in order to avoid having any rows drop out.)
    3. My guess on the two unique keys, one was assigned when the data was extracted, and the other is used a foreign key by other data.
    4. prepare for a lot of clean up activity (if a field is labelled sex and mostly contains 'M' or 'F', then be prepared to report those records whose value is not 'M' or "F'.
    5. do not worry about redundancies in the source data, all that matters is which source column will be used to set the target value. You can only pick one.
    6. do not gripe about the lack of normalization. Most systems work with less than a fully normalized data model. Having room for up to 4 nurseries (in your example) probably handles more than 99.5% of the children, a possible work around would be to create a second record for the same child. Ugly, but given the limited benefit and the cost of a rewrite, it probably was never done.
  • Remy Lebeau (unregistered) in reply to mangobrain
    mangobrain:
    This! A thousand times this. Why do programming departments in general continue to put up with this sort of crap? Salespeople should be held accountable for any promises they make, in such a way that they feel inclined to consult R&D before making outlandish claims. Perhaps commission in such cases should be contingent on the claims actually being met, and if any significant developer effort is required to meet them, a portion of it given to the developers.

    +1000! I run in to this more times than not in my job. The sales dept makes promises for new features and such just to land big deals, then the development dept is expected to implement them, usually in a relatively short amount of time (days, weeks), and we usually don't get bonuses for doing it, just thank-you and good-job emails.

  • The president's child (unregistered) in reply to xaade
    xaade:
    I see why they put him/she in the database, what if you had a genetically mixed kid, they may be legally a boy, but prefer to be referred to as a girl.

    What you need is two fields.

    DNA gender (male, female, mixed male, mixed female) Gender Identity Crises (Yes, No)

    Then, if GIC is set, you flip the pronouns from DNA gender to the "opposite".

    This way you can add in new genders later, like:

    Famele and Mefel

    Of course you'd also need the politically correct "selected gender identity" if they have a separate identity than what's on their BC.

    The question there would be: You're all muscular, but you like to wear dresses, which gender do you see yourself as.

    You flip the answer to that, and then you know what gender they'd like to be.

    Children have sex, words have gender... except... One time in school, someone asked "What do the M and F mean," and the teacher answered "It's asking if you're the mother type or the father type." The teacher didn't even think about Unix, because then he'd know some don't have sex. I got so confused trying to find in the Child table if I have sex or not, it made me sick.

  • Carlos (unregistered)

    I know someone who could have designed something like this, because he has a twisted brain. Not gonna name names, of course.

  • SQL programmer (unregistered) in reply to Friedrice The Great
    Friedrice The Great:
    Pock Suppet:
    I'm having a hard time deciding whether the opposite of "programmer" is "marketer", "executive", or "lawyer", but I'd happily do with a few less of each of them.
    The opposite of "programmer" is "non-programmer".
    The opposite of "programmer" is: VB programmer PHP programmer Go programmer C++ programmer Fortran programmer Cobol programmer PL/I programmer C# programmer Haskell programmer Python programmer Perl programmer APL programmer Brainfuck programmer JavaScript programmer Excel macro programmer sendmail config file programmer bash script programmer JCL catalogued procedure programmer
  • (cs)

    Yes, they ask questions about both 'sex' and 'gender'.

    Just remember that when asked "Sex?", the answer is always "Yes!".

  • George (unregistered)

    While the data looks like it is a bit of a mess, in terms of importing the data it looks to be a pretty simple job as far as these things go.

  • QJo (unregistered)

    Child comes home from school after first day. That evening: "Mummy, what's sex?"

    The mother already knows that this child is seriously precocious and educationally advanced, but is a little disturbed that this seems somewhat early to be asking such a question, thinks the mother. The poor child must have been picking up rude talk in the playground. But, being modern and liberated and having made a decision to always, always be honest with her children, she sits the child down and explains the long and lovely story of where babies come from.

    The child holds up a partially-filled-in form, and says, "How am I supposed to fit all that into this little box?"

  • Anoncow (unregistered) in reply to xaade

    One of our customers, a large casino, has been monitoring their data quality. In my country, the first six digits of our national ID number is the birthdate, yymmdd. When a new customer signs up, ID number and birthdate are both required fields. Turns out that last month, fully 5% of customers filled in a DOB which didn't match their ID number.

    People are idiots.

  • (cs) in reply to KattMan
    KattMan:
    There are three other chromosomal.

    You've missed out XXXX, XXXY, XXYY, XYYY, XXXXX, XXXXY, XYYYY. And I'm sure there are more.

    I daresay there is nothing physically preventing XXXXXXYYYYYYY other than statistical unlikelyhood, and high probability of early foetal termination.

  • (cs) in reply to Nagesh
    Nagesh:
    eViLegion:
    Nagesh, you, your company, and your algorithm are all made out of mashed up arseholes.

    Don't be hater. Hate is like acid.

    No hate. You're just an easy target, what with you being so thick.

    Anyway, I like acid.

  • (cs) in reply to eViLegion
    eViLegion:
    Nagesh:
    eViLegion:
    Nagesh, you, your company, and your algorithm are all made out of mashed up arseholes.

    Don't be hater. Hate is like acid.

    No hate. You're just an easy target, what with you being so thick.

    Anyway, I like acid.

    Only thing thick about me is my sinewy arms and thighs.

  • floating ground (unregistered)

    PS. Learn sed, awk, tr, sort and uniq.

  • (cs) in reply to Nagesh
    Nagesh:
    Only thing thick about me is my sinewy arms and thighs.

    The truth comes out: Nagesh is a trained seal.

  • Paul Neumann (unregistered) in reply to herby
    herby:
    Yes, they ask questions about both 'sex' and 'gender'.

    Just remember that when asked "Sex?", the answer is always "Yes!".

    Sex, herby?

    capta: OPTO Herby may want to OPTO[ut] of that offer.

  • Alorinna (unregistered) in reply to Sockatume

    Ah, but you are LOADing the data for MOTHERs, so thus it is indeed a mother-load :)

  • Barf 4Eva (unregistered)

    I fail to see the WTF here.

    But that's only because I've been victimized far worse for so long now by our very own file imports, such that the sales people have literally brain-washed me into believing this is the only way we'll ever function as a company.

  • jay (unregistered) in reply to mangobrain
    mangobrain:
    This! A thousand times this. Why do programming departments in general continue to put up with this sort of crap? Salespeople should be held accountable for any promises they make, in such a way that they feel inclined to consult R&D before making outlandish claims. Perhaps commission in such cases should be contingent on the claims actually being met, and if any significant developer effort is required to meet them, a portion of it given to the developers.

    And then the handsome prince should rescue the beautiful princess from the evil troll, and they get married and live happily ever after while unicorns frolic in the pasture.

    I think that's how fairy tales like this usually end.

  • Not-that-alex (unregistered)

    This is required for internationalisation. If it is a child from germany, the table holds

    Column Name Sample Data gender "Junge", "Mädchen" heshe "er", "sie" himher "ihm", "ihr" hisher "sein", "ihr" sondaughter "Sohn", "Tochter"

    This is clever, isn't it? I mean, have a look at http://arnoldzwicky.files.wordpress.com/2012/04/germanarticles.jpg. Do you really want to implement that?

  • (cs) in reply to mangobrain
    mangobrain:
    This! A thousand times this. Why do programming departments in general continue to put up with this sort of crap? Salespeople should be held accountable for any promises they make, in such a way that they feel inclined to consult R&D before making outlandish claims. Perhaps commission in such cases should be contingent on the claims actually being met, and if any significant developer effort is required to meet them, a portion of it given to the developers.
    My employer considers our ability to do customer data conversion a core competency. None of our competitors do it. We write custom programs to migrate the data, then when they're ready to go live they can run that program against their current data and be up and running on our system with all their records intact. We charge for this service, and the customers have all thought it well worth the cost. It's an advantage when attracting new customers.
  • amy (unregistered)

    I'm crying a little bit right now...

  • B00nbuster (unregistered)

    What a crappy design. However, a few "SELECT DISTINCT" statements should give you all the values for a proper ETL process with conversion.

    Then you add a few validation queries.

    Problem solved.

  • Author (unregistered)

    I absolutely agree that importing this data is not really more complex then any properly normalized database. But the fact that these tables were used in a production environment of a paid (and expensive) application is a bit sad to say the least... "Does it work?" -Yes it works. "Then do release!" -Uh okay...

    There are many accidents luring just around the corner for databases like this, especially for the new guy...

  • John tan (unregistered)

    this is probably the easiest database optimization and migration I ever seen.

    ur so lucky....getting good money for this kind of stupid work..

Leave a comment on “It's Nobody's Baby”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article