• (cs)
    Though Eric didn't get in any trouble for single-handedly crashing the company's mission-critical system

    Crashing the mission-critical system: thousands worth of lost work.

    Crashing it without getting in trouble: priceless.

  • Getmeaguiness (unregistered)

    Makes you wonder how MSAccess got to be the platform of choice

  • Cybercat (unregistered)

    The real WTF:

    If the company who programmed this monstrosity could foot a random $285,000 bill, why didn't they put in enough money for a new programming team to write a new application that wasn't such a pile of shit? $285k is enough for one good senior level programmer and 3-4 average to newbie levels for drone work. Problem solved?

  • ratnerstar (unregistered)

    TRWTF is that his company has only one customer, and it's a US auto manufacturer. Better think about diversifying a bit!

  • d000hg (unregistered) in reply to Getmeaguiness
    Getmeaguiness:
    Makes you wonder how MSAccess got to be the platform of choice
    Probably they knocked up a working prototype, and got trapped by it?
  • AC (unregistered)

    And why the burning place where devils "live" did it deadlock when just exporting a table? Is Access supposed to do that?

  • Anon (unregistered) in reply to AC

    Multiple requests + Access

  • (cs)

    You had me* at "Microsoft Access"

    *Laughing out loud

  • SeaDrive (unregistered)

    They have a mission-critical app that teeters on the edge of failure, and they don't tell the new guy that the first 10 things he wants to do are off limits because they will crash the system.

    Management deserves what they get.

  • tekiegreg (unregistered) in reply to AC

    MS Access isn't supposed to be used in these kinds of heavy production environments, period. Even by MS's own strongly worded recommendations.

    It's locking structure is not at all suited for multi-user and it likes to acquire table locks for the simplest of things. And why not? As far as Access is concerned, there should maybe be 1 or 2 people in the db at one time...

    First thing I'm doing if I get there is upsizing to SQL Server and let access run as the front end only (One of it's redeeming features is access functions well as a front end for SQL Server, can preserve the existing access app but have a more robust database behind it).

  • (cs) in reply to AC
    AC:
    And why the burning place where devils "live" did it deadlock when just exporting a table? Is Access supposed to do that?
    Access isn't designed to have many users accessing one .mdb file. In fact, it rarely works as it should when having more than one concurrent access!!

    Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!

  • AMerrickanGirl (unregistered) in reply to danixdefcon5

    Why Oracle? Why not SQL Server? You're already in a Microsoft environment.

    danixdefcon5:
    AC:
    And why the burning place where devils "live" did it deadlock when just exporting a table? Is Access supposed to do that?
    Access isn't designed to have many users accessing one .mdb file. In fact, it rarely works as it should when having more than one concurrent access!!

    Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!

  • Global Warmer (unregistered) in reply to Cybercat

    I have spent most of my career working on this type of crap for the auto industry. $5000 a minute is cheap, it can run allot higher. Even with the high cost of failure though, scary stuff like this happen over and over. These guys are willing to take the risk in order to save a buck.

    Cybercat:
    The real WTF:

    If the company who programmed this monstrosity could foot a random $285,000 bill, why didn't they put in enough money for a new programming team to write a new application that wasn't such a pile of shit? $285k is enough for one good senior level programmer and 3-4 average to newbie levels for drone work. Problem solved?

    They do not usually make you pay this up front, they usually deduct it from what they owe you over time. Also, you can usually negotiate it down.

  • Mr. Smith (unregistered)

    When any company tells me they are using Access I tell them that my 8 yr old niece uses that as well for organizing her tape collection and perhaps they should talk to her if they want someone who ENJOYS using Access.

  • (cs) in reply to ratnerstar
    ratnerstar:
    TRWTF is that his company has only one customer, and it's a US auto manufacturer. Better think about diversifying a bit!

    I heard that they tried adding another customer, but it deadlocked the database.

  • (cs) in reply to AMerrickanGirl
    AMerrickanGirl:
    Why Oracle? Why not SQL Server? You're already in a Microsoft environment.
    Of course its the easiest drop-in replacement; but SQL Server isn't as expensive as the Oracle solution. I was kind of pointing out that with those enormous losses, they're already kicking into high-end expenses without actually improving the system. Ow.
  • jd (unregistered) in reply to danixdefcon5
    danixdefcon5:
    AC:
    And why the burning place where devils "live" did it deadlock when just exporting a table? Is Access supposed to do that?
    Access isn't designed to have many users accessing one .mdb file. In fact, it rarely works as it should when having more than one concurrent access!!

    Create a new mdb with a linked table to the production mdb? Import the report into your new mdb, make the changes, you're good to go.

  • bobzilla (unregistered) in reply to Global Warmer
    Global Warmer:
    ...allot...
    Grammar Nazi Attack: allot? How about 'a lot'?
  • biziclop (unregistered)

    I keep thinking that if it wasn't software but forklifts or trucks in similar condition, nobody'd take the risk.

    We should've called it "f*cks-you-really-hard-if-youre-not-careful-ware" and everybody would take software issues seriously.

  • Rhywun (unregistered) in reply to tekiegreg
    tekiegreg:
    And why not? As far as Access is concerned, there should maybe be 1 or 2 people in the db at one time...

    I've got one with around 20 users working well; it's not terribly heavy though.

    For a "major auto manufacturer" to hang its business in Access, though... wow. My jaw hit the floor on that one. Harder than any other story here. So hard, in fact, that I'm having trouble believing it.

  • (cs)

    About 5 years ago I placed an order for a Ford through my local dealership. It took about 4 months to deliver it from the factory, during which time it was completely untrackable and the dealership had no idea where it was or when it would show up. I always wondered how the fuck such a large company could be such as mess. Now I know.

  • FOSS Rulz (unregistered) in reply to AMerrickanGirl
    AMerrickanGirl:
    Why Oracle? Why not SQL Server? You're already in a Microsoft environment.
    danixdefcon5:
    Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!

    Getting away from the Microsoft environment might be the point.

    However, I'd go with MySQL. If it's good enough for Google and Wikipedia, it's good enough for whatever you're working on.

  • (cs)
    With no development, testing or QA environment for MCL in sight, Eric did the most sensible thing he could think of before developing some minor changes to a report:
    The MOST sensible thing under these conditions would be to start spreading his resumé to every softwarehouse around, and just to be sure he'll land a job somewhere else he should also send it to any other company, be it a bank or a bakery.
  • Stan (unregistered)

    Haw! It HAD to be one of the US big-three. No other automaker is that stupid!

  • ike (unregistered) in reply to danixdefcon5
    danixdefcon5:
    Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!
    At least go and buy a development, testing and QA servers... making changes in the production environment... how stupid... uh... oh, they're using Access for the mission-critical data store... never mind... they are that stupid...
  • Anonymous (unregistered) in reply to FOSS Rulz
    FOSS Rulz:
    Getting away from the Microsoft environment might be the point.

    However, I'd go with MySQL. If it's good enough for Google and Wikipedia, it's good enough for whatever you're working on.

    Google has their own proprietary database systems for anything involving heavy lifting / mission critical apps; MySQL tends to be for minor stuff.

    Some parts of Google (acquired parts) actually do use MS SQL Server, but they try not to do that for too long.

  • jtl (unregistered) in reply to SeaDrive
    SeaDrive:

    Management deserves what they get.

    6 figure salaries?

  • Frenchier than thou (unregistered) in reply to Stan
    Stan:
    Haw! It HAD to be one of the US big-three. No other automaker is that stupid!

    Ah, but you forget that this system is outsourced from the automaker. So it could be any sell-my-own-management-system automaker, if the outsourcer looks enterprisey enough!

  • anon coward (unregistered) in reply to Mr. Smith
    Mr. Smith:
    When any company tells me they are using Access I tell them that my 8 yr old niece uses that as well for organizing her tape collection and perhaps they should talk to her if they want someone who ENJOYS using Access.
    TRWTF is that your niece still has tapes in the 21st century! ;)
  • (cs)

    This can't be real. He never mentions near-daily recovery of the database when Access inevitably corrupts the MDB.

  • Global Warmer (unregistered) in reply to bobzilla
    bobzilla:
    Global Warmer:
    ...allot...
    Grammar Nazi Attack: allot? How about 'a lot'?

    How 'bout it

  • (cs) in reply to tekiegreg
    tekiegreg:
    First thing I'm doing if I get there is upsizing to SQL Server and let access run as the front end only (One of it's redeeming features is access functions well as a front end for SQL Server, can preserve the existing access app but have a more robust database behind it).

    This solution always sounds good, until you realize that there will now be lots of brand-new and unhandled exceptions in your production system.

  • (cs)

    In the 90's, I worked on the system for a (now long gone) airline. At one point my PHB told me that they lost $500,000 per minute of downtime. and this was a medium sized north american carrier. American Airlines would be 2-5 times larger, if not more. Most of that was opportunity cost, but it's still a nice big number.

  • Bob X (unregistered) in reply to Getmeaguiness

    Dude, haven't you noticed how many WTFs are the direct result of choosing MS Access to be the backend of a high-volume multi-user production-critical application like this?

    Frankly, MS should limit Access to 50 tables per DB, 100k records per table, and 20 concurrent users. That would make it great platform for your tape collection DB, but impossible to use for Ford's production control system. As it should be.

    Captcha: luptatum. WTF?

  • Bob X (unregistered) in reply to Rhywun

    Believe it. I work for a major semicon, and our worldwide production control system still runs on a VAX. I poop you not.

  • (cs) in reply to FOSS Rulz
    FOSS Rulz:
    AMerrickanGirl:
    Why Oracle? Why not SQL Server? You're already in a Microsoft environment.
    danixdefcon5:
    Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!

    Getting away from the Microsoft environment might be the point.

    However, I'd go with MySQL. If it's good enough for Google and Wikipedia, it's good enough for whatever you're working on.

    PostgreSQL might be also good. Anyway there are a lot of db which are eighter free/open or much less expensive then the crash...

  • Yesinouhehe (unregistered) in reply to Getmeaguiness

    Is everything you read on the internet true?

    What if: Access was grandfathered in from some rinky dink outfit and Access were only a front end to MSSQL and one application of many in the system and Eric is not a developer but just a poor smuck clerk and Eric hacked Access to get to the data in the first place & Access had nothing to do with the previous big $$$ loss...

    Still no excuse for a p*ss poor system but change 'barely' accessible to 'easily' accessible and 'developer' to 'hacker' to inject some reality.

  • Tony (unregistered) in reply to Anonymous
    Anonymous:
    Some parts of Google (acquired parts) actually do use MS SQL Server, but they try not to do that for too long.

    I dislike MS products just as much as the next guy, but it seems Google's taken "embrace and extend" and turned it into "assimilate and rewrite," which is a WTF unto itself.

  • Soviut (unregistered) in reply to Mr. Smith
    Mr. Smith:
    When any company tells me they are using Access I tell them that my 8 yr old niece uses that as well for organizing her tape collection and perhaps they should talk to her if they want someone who ENJOYS using Access.

    What are tapes? ;) (The real WTF?)

  • Anonymous (unregistered) in reply to Bob X
    Bob X:
    Dude, haven't you noticed how many WTFs are the direct result of choosing MS Access to be the backend of a high-volume multi-user production-critical application like this?

    Most of the time this is the result of the application being used well beyond it's expected life or a prototype being pressed into production. The company often starts off with minimal needs where an Access app is perfectly appropriate. The company grows or is acquired, the application grows, but no one bothers to upsize it to a more enterprisey platform. The WTF is not Access in almost all cases I've seen here - it's unrealistic expectations and poor planning. I guess it's easier to knee jerk a "lol Access" response, though.

  • dgm (unregistered) in reply to uzytkownik
    uzytkownik:
    FOSS Rulz:
    AMerrickanGirl:
    Why Oracle? Why not SQL Server? You're already in a Microsoft environment.
    danixdefcon5:
    Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!

    Getting away from the Microsoft environment might be the point.

    However, I'd go with MySQL. If it's good enough for Google and Wikipedia, it's good enough for whatever you're working on.

    PostgreSQL might be also good. Anyway there are a lot of db which are eighter free/open or much less expensive then the crash...

    Arguments aside over Mysql's suitability for such an enterprise app, I think we'd all agree that it is still a step up from Access!

  • dgm (unregistered)

    The right way to go about this would be to use shadow volume to make a backup copy..

  • (cs) in reply to Global Warmer
    Global Warmer:
    bobzilla:
    Global Warmer:
    ...allot...
    Grammar Nazi Attack: allot? How about 'a lot'?

    How 'bout it

    Grammar Soviet stands firm!

    Now the eyes of Grammar World are upon the Grammar Western Allies to open a second front in the West!

  • Paul (unregistered)

    Does anyone else think of MCL Cafeteria when reading this story?

  • Anonymous Coward (unregistered) in reply to Bob X
    Bob X:
    Frankly, MS should limit Access to 50 tables per DB, 100k records per table, and 20 concurrent users. That would make it great platform for your tape collection DB, but impossible to use for Ford's production control system. As it should be.

    Do you really want to see the ugly ugly workarounds that would exist to get around those limits if that was the case? It might be even worse than example in the article...

    Captcha: consequat

  • Paul Keeble (unregistered) in reply to MET

    6 years ago I came across such an MS Access system in a bank. It held all complaints made against them and did the workflow on resolution of the compliants. With the number of mortgage complaints at an all time high (in the UK) there were about 100 users, doing nothing but using that system and dealing with the complaints, well trying to anyway.

    Not only that but the developer was changing the same mdb file as the users were using it! Not QA, now testing, just save it and see what it did! They had corrupted DBs frequently, obviously serious quality problems and it was a complete mess. What did this bank do about it? Nothing not a thing it just kept creeking along.

    The morale of this story is that you CAN run a mission critical system on MS Access and while as "real" developers we scoff at everything at this approach MS Access is actually working! That is more than I can say for some Java/C#/whatever apps I have seen.

  • Mark (unregistered) in reply to Anonymous
    Anonymous:
    FOSS Rulz:
    Getting away from the Microsoft environment might be the point.

    However, I'd go with MySQL. If it's good enough for Google and Wikipedia, it's good enough for whatever you're working on.

    Google has their own proprietary database systems for anything involving heavy lifting / mission critical apps; MySQL tends to be for minor stuff.

    Some parts of Google (acquired parts) actually do use MS SQL Server, but they try not to do that for too long.

    Yeah, I'm getting tired of the fanboys trumpeting that "Google uses MySQL. And Google is HUGE>!>!! OMG111". IIRC, there was an article in Ars Technica a while back looking into Google's usage of MySQL. Turns out they use it for internal stuff like organizing company picnics, etc.

    BTW, I'm no hater. I use MySQL anytime it's appropriate.

  • Jethris (unregistered)

    Access is a great tool (consider it a small hammer). It's great at hammering nails to hang pictures from, but you wouldn't want to use it to try and bust up concrete.

    I developed an Access app that managed mission critical tapes for a branch of the US Military. Granted, all it did was track the tapes (when they were being recorded, what satellite was operational, how long do they need to be kept, which tapes could be reformated, etc.), but it did those tasks very well.

    It had 1 data entry user, and 5-10 read-only users. Maybe 2 concurrent user, 3 tops.

    SQL Server would have been overkill. Access was cheap, fast, created nice reports, and fit the bill. Downtime for the app had very minimal costs.

    I love using Access for what it was designed for, a desktop database. But, for that many records? The lack of transaction logging itself would be scary.

  • Mark (unregistered) in reply to Walleye
    Walleye:
    tekiegreg:
    First thing I'm doing if I get there is upsizing to SQL Server and let access run as the front end only (One of it's redeeming features is access functions well as a front end for SQL Server, can preserve the existing access app but have a more robust database behind it).

    This solution always sounds good, until you realize that there will now be lots of brand-new and unhandled exceptions in your production system.

    Can someone say "ODBC Error"? Access front end runs this:

    DoCMD.RunSQL "Update table set field = value"

    And it runs it on a linked SQL Server table the user doesn't have UPDATE privileges on. Good luck trapping that.

    OTOH, I agree. Generally, Access does make an easy to develop front-end to SQL Server. You just have to know its limits.

  • Jethris (unregistered)

    Access is a great tool (consider it a small hammer). It's great at hammering nails to hang pictures from, but you wouldn't want to use it to try and bust up concrete.

    I developed an Access app that managed mission critical tapes for a branch of the US Military. Granted, all it did was track the tapes (when they were being recorded, what satellite was operational, how long do they need to be kept, which tapes could be reformated, etc.), but it did those tasks very well.

    It had 1 data entry user, and 5-10 read-only users. Maybe 2 concurrent user, 3 tops.

    SQL Server would have been overkill. Access was cheap, fast, created nice reports, and fit the bill. Downtime for the app had very minimal costs.

    I love using Access for what it was designed for, a desktop database. But, for that many records? The lack of transaction logging itself would be scary.

Leave a comment on “A Barely Accessible System”

Log In or post as a guest

Replying to comment #:

« Return to Article