Comment On A Barely Accessible System

Calculating the true cost of downtime is almost impossible. There's not only the obvious loss of labor to consider, but all sorts of indirect losses like missed opportunity, repair expenses, customer frustration and so on. Fortunately for Eric M.'s company, the management knows exactly how many real dollars it will cost them when their system -- "MCL," as I'll call it -- goes down. Eric's employer is a logistics service provider with a sole customer: a major U.S. automaker. His company is primarily responsible for getting the right auto parts to the right areas in the right plants, on time. Any unexpected delay or shipment error and the entire assembly line can shut down -- and when that happens, the service provider gets to foot the bill to the tune of $5,000 per minute. [expand full text]
« PrevPage 1 | Page 2 | Page 3 | Page 4Next »

Re: A Barely Accessible System

2008-08-14 11:04 • by Justice
Though Eric didn't get in any trouble for single-handedly crashing the company's mission-critical system


Crashing the mission-critical system: thousands worth of lost work.

Crashing it without getting in trouble: priceless.

Re: A Barely Accessible System

2008-08-14 11:09 • by Getmeaguiness (unregistered)
Makes you wonder how MSAccess got to be the platform of choice

Re: A Barely Accessible System

2008-08-14 11:10 • by Cybercat (unregistered)
The real WTF:

If the company who programmed this monstrosity could foot a random $285,000 bill, why didn't they put in enough money for a new programming team to write a new application that wasn't such a pile of shit? $285k is enough for one good senior level programmer and 3-4 average to newbie levels for drone work. Problem solved?

Re: A Barely Accessible System

2008-08-14 11:13 • by ratnerstar (unregistered)
TRWTF is that his company has only one customer, and it's a US auto manufacturer. Better think about diversifying a bit!

Re: A Barely Accessible System

2008-08-14 11:13 • by d000hg (unregistered)
212008 in reply to 212005
Getmeaguiness:
Makes you wonder how MSAccess got to be the platform of choice
Probably they knocked up a working prototype, and got trapped by it?

Re: A Barely Accessible System

2008-08-14 11:20 • by AC (unregistered)
And why the burning place where devils "live" did it deadlock when just exporting a table? Is Access supposed to do that?

Re: A Barely Accessible System

2008-08-14 11:23 • by Anon (unregistered)
212012 in reply to 212009
Multiple requests + Access

Re: A Barely Accessible System

2008-08-14 11:28 • by MET
You had me* at "Microsoft Access"

*Laughing out loud

Re: A Barely Accessible System

2008-08-14 11:28 • by SeaDrive (unregistered)
They have a mission-critical app that teeters on the edge of failure, and they don't tell the new guy that the first 10 things he wants to do are off limits because they will crash the system.

Management deserves what they get.

Re: A Barely Accessible System

2008-08-14 11:28 • by tekiegreg (unregistered)
212016 in reply to 212009
MS Access isn't supposed to be used in these kinds of heavy production environments, period. Even by MS's own strongly worded recommendations.

It's locking structure is not at all suited for multi-user and it likes to acquire table locks for the simplest of things. And why not? As far as Access is concerned, there should maybe be 1 or 2 people in the db at one time...

First thing I'm doing if I get there is upsizing to SQL Server and let access run as the front end only (One of it's redeeming features is access functions well as a front end for SQL Server, can preserve the existing access app but have a more robust database behind it).

Re: A Barely Accessible System

2008-08-14 11:30 • by danixdefcon5
212018 in reply to 212009
AC:
And why the burning place where devils "live" did it deadlock when just exporting a table? Is Access supposed to do that?
Access isn't designed to have many users accessing one .mdb file. In fact, it rarely works as it should when having more than one concurrent access!!

Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!

Re: A Barely Accessible System

2008-08-14 11:36 • by AMerrickanGirl (unregistered)
212021 in reply to 212018
Why Oracle? Why not SQL Server? You're already in a Microsoft environment.

danixdefcon5:
AC:
And why the burning place where devils "live" did it deadlock when just exporting a table? Is Access supposed to do that?
Access isn't designed to have many users accessing one .mdb file. In fact, it rarely works as it should when having more than one concurrent access!!

Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!

Re: A Barely Accessible System

2008-08-14 11:55 • by Global Warmer (unregistered)
212025 in reply to 212006
I have spent most of my career working on this type of crap for the auto industry. $5000 a minute is cheap, it can run allot higher. Even with the high cost of failure though, scary stuff like this happen over and over. These guys are willing to take the risk in order to save a buck.


Cybercat:
The real WTF:

If the company who programmed this monstrosity could foot a random $285,000 bill, why didn't they put in enough money for a new programming team to write a new application that wasn't such a pile of shit? $285k is enough for one good senior level programmer and 3-4 average to newbie levels for drone work. Problem solved?


They do not usually make you pay this up front, they usually deduct it from what they owe you over time. Also, you can usually negotiate it down.

Re: A Barely Accessible System

2008-08-14 11:56 • by Mr. Smith (unregistered)
When any company tells me they are using Access I tell them that my 8 yr old niece uses that as well for organizing her tape collection and perhaps they should talk to her if they want someone who ENJOYS using Access.

Re: A Barely Accessible System

2008-08-14 12:06 • by KNY
212028 in reply to 212007
ratnerstar:
TRWTF is that his company has only one customer, and it's a US auto manufacturer. Better think about diversifying a bit!


I heard that they tried adding another customer, but it deadlocked the database.

Re: A Barely Accessible System

2008-08-14 12:06 • by danixdefcon5
212029 in reply to 212021
AMerrickanGirl:
Why Oracle? Why not SQL Server? You're already in a Microsoft environment.
Of course its the easiest drop-in replacement; but SQL Server isn't as expensive as the Oracle solution. I was kind of pointing out that with those enormous losses, they're already kicking into high-end expenses without actually improving the system. Ow.

Re: A Barely Accessible System

2008-08-14 12:09 • by jd (unregistered)
212030 in reply to 212018
danixdefcon5:
AC:
And why the burning place where devils "live" did it deadlock when just exporting a table? Is Access supposed to do that?
Access isn't designed to have many users accessing one .mdb file. In fact, it rarely works as it should when having more than one concurrent access!!


Create a new mdb with a linked table to the production mdb? Import the report into your new mdb, make the changes, you're good to go.

Re: A Barely Accessible System

2008-08-14 12:10 • by bobzilla (unregistered)
212031 in reply to 212025
Global Warmer:
...allot...

Grammar Nazi Attack: allot? How about 'a lot'?

Re: A Barely Accessible System

2008-08-14 12:25 • by biziclop (unregistered)
I keep thinking that if it wasn't software but forklifts or trucks in similar condition, nobody'd take the risk.

We should've called it "f*cks-you-really-hard-if-youre-not-careful-ware" and everybody would take software issues seriously.

Re: A Barely Accessible System

2008-08-14 12:28 • by Rhywun (unregistered)
212038 in reply to 212016
tekiegreg:
And why not? As far as Access is concerned, there should maybe be 1 or 2 people in the db at one time...


I've got one with around 20 users working well; it's not terribly heavy though.

For a "major auto manufacturer" to hang its business in Access, though... wow. My jaw hit the floor on that one. Harder than any other story here. So hard, in fact, that I'm having trouble believing it.

Re: A Barely Accessible System

2008-08-14 12:44 • by Outlaw Programmer
About 5 years ago I placed an order for a Ford through my local dealership. It took about 4 months to deliver it from the factory, during which time it was completely untrackable and the dealership had no idea where it was or when it would show up. I always wondered how the fuck such a large company could be such as mess. Now I know.

Re: A Barely Accessible System

2008-08-14 12:47 • by FOSS Rulz (unregistered)
212040 in reply to 212021
AMerrickanGirl:
Why Oracle? Why not SQL Server? You're already in a Microsoft environment.

danixdefcon5:
Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!


Getting away from the Microsoft environment might be the point.

However, I'd go with MySQL. If it's good enough for Google and Wikipedia, it's good enough for whatever you're working on.

Re: A Barely Accessible System

2008-08-14 12:52 • by Smash King
With no development, testing or QA environment for MCL in sight, Eric did the most sensible thing he could think of before developing some minor changes to a report:
The MOST sensible thing under these conditions would be to start spreading his resumé to every softwarehouse around, and just to be sure he'll land a job somewhere else he should also send it to any other company, be it a bank or a bakery.

Re: A Barely Accessible System

2008-08-14 12:53 • by Stan (unregistered)
Haw! It HAD to be one of the US big-three. No other automaker is that stupid!

Re: A Barely Accessible System

2008-08-14 13:01 • by ike (unregistered)
212044 in reply to 212018
danixdefcon5:
Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!

At least go and buy a development, testing and QA servers... making changes in the production environment... how stupid... uh... oh, they're using Access for the mission-critical data store... never mind... they are that stupid...

Re: A Barely Accessible System

2008-08-14 13:01 • by Anonymous (unregistered)
212045 in reply to 212040
FOSS Rulz:

Getting away from the Microsoft environment might be the point.

However, I'd go with MySQL. If it's good enough for Google and Wikipedia, it's good enough for whatever you're working on.


Google has their own proprietary database systems for anything involving heavy lifting / mission critical apps; MySQL tends to be for minor stuff.

Some parts of Google (acquired parts) actually do use MS SQL Server, but they try not to do that for too long.

Re: A Barely Accessible System

2008-08-14 13:03 • by jtl (unregistered)
212046 in reply to 212015
SeaDrive:


Management deserves what they get.


6 figure salaries?

Re: A Barely Accessible System

2008-08-14 13:03 • by Frenchier than thou (unregistered)
212047 in reply to 212042
Stan:
Haw! It HAD to be one of the US big-three. No other automaker is that stupid!


Ah, but you forget that this system is *outsourced* from the automaker. So it could be any sell-my-own-management-system automaker, if the outsourcer looks enterprisey enough!

Re: A Barely Accessible System

2008-08-14 13:08 • by anon coward (unregistered)
212049 in reply to 212026
Mr. Smith:
When any company tells me they are using Access I tell them that my 8 yr old niece uses that as well for organizing her tape collection and perhaps they should talk to her if they want someone who ENJOYS using Access.

TRWTF is that your niece still has tapes in the 21st century! ;)

Re: A Barely Accessible System

2008-08-14 13:19 • by Vanders
This can't be real. He never mentions near-daily recovery of the database when Access inevitably corrupts the MDB.

Re: A Barely Accessible System

2008-08-14 13:21 • by Global Warmer (unregistered)
212053 in reply to 212031
bobzilla:
Global Warmer:
...allot...

Grammar Nazi Attack: allot? How about 'a lot'?


How 'bout it

Re: A Barely Accessible System

2008-08-14 13:25 • by Walleye
212054 in reply to 212016
tekiegreg:
First thing I'm doing if I get there is upsizing to SQL Server and let access run as the front end only (One of it's redeeming features is access functions well as a front end for SQL Server, can preserve the existing access app but have a more robust database behind it).


This solution always sounds good, until you realize that there will now be lots of brand-new and unhandled exceptions in your production system.

Re: A Barely Accessible System

2008-08-14 13:27 • by morry
In the 90's, I worked on the system for a (now long gone) airline. At one point my PHB told me that they lost $500,000 per minute of downtime. and this was a medium sized north american carrier. American Airlines would be 2-5 times larger, if not more. Most of that was opportunity cost, but it's still a nice big number.

Re: A Barely Accessible System

2008-08-14 13:33 • by Bob X (unregistered)
212057 in reply to 212005
Dude, haven't you noticed how many WTFs are the direct result of choosing MS Access to be the backend of a high-volume multi-user production-critical application like this?

Frankly, MS should limit Access to 50 tables per DB, 100k records per table, and 20 concurrent users. That would make it great platform for your tape collection DB, but impossible to use for Ford's production control system. As it should be.

Captcha: luptatum. WTF?

Re: A Barely Accessible System

2008-08-14 13:34 • by Bob X (unregistered)
212058 in reply to 212038
Believe it. I work for a major semicon, and our worldwide production control system still runs on a VAX. I poop you not.

Re: A Barely Accessible System

2008-08-14 13:37 • by uzytkownik
212059 in reply to 212040
FOSS Rulz:
AMerrickanGirl:
Why Oracle? Why not SQL Server? You're already in a Microsoft environment.

danixdefcon5:
Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!


Getting away from the Microsoft environment might be the point.

However, I'd go with MySQL. If it's good enough for Google and Wikipedia, it's good enough for whatever you're working on.


PostgreSQL might be also good. Anyway there are a lot of db which are eighter free/open or much less expensive then the crash...

Re: A Barely Accessible System

2008-08-14 13:42 • by Yesinouhehe (unregistered)
212060 in reply to 212005
Is everything you read on the internet true?

What if:
Access was grandfathered in from some rinky dink outfit and
Access were only a front end to MSSQL and
one application of many in the system and
Eric is not a developer but just a poor smuck clerk and
Eric hacked Access to get to the data in the first place &
Access had nothing to do with the previous big $$$ loss...

Still no excuse for a p*ss poor system but change
'barely' accessible to 'easily' accessible and
'developer' to 'hacker' to inject *some* reality.

Re: A Barely Accessible System

2008-08-14 13:44 • by Tony (unregistered)
212061 in reply to 212045
Anonymous:
Some parts of Google (acquired parts) actually do use MS SQL Server, but they try not to do that for too long.


I dislike MS products just as much as the next guy, but it seems Google's taken "embrace and extend" and turned it into "assimilate and rewrite," which is a WTF unto itself.

Re: A Barely Accessible System

2008-08-14 13:45 • by Soviut (unregistered)
212062 in reply to 212026
Mr. Smith:
When any company tells me they are using Access I tell them that my 8 yr old niece uses that as well for organizing her tape collection and perhaps they should talk to her if they want someone who ENJOYS using Access.


What are tapes? ;) (The real WTF?)

Re: A Barely Accessible System

2008-08-14 13:51 • by Anonymous (unregistered)
212063 in reply to 212057
Bob X:
Dude, haven't you noticed how many WTFs are the direct result of choosing MS Access to be the backend of a high-volume multi-user production-critical application like this?


Most of the time this is the result of the application being used well beyond it's expected life or a prototype being pressed into production. The company often starts off with minimal needs where an Access app is perfectly appropriate. The company grows or is acquired, the application grows, but no one bothers to upsize it to a more enterprisey platform. The WTF is not Access in almost all cases I've seen here - it's unrealistic expectations and poor planning. I guess it's easier to knee jerk a "lol Access" response, though.

Re: A Barely Accessible System

2008-08-14 13:52 • by dgm (unregistered)
212064 in reply to 212059
uzytkownik:
FOSS Rulz:
AMerrickanGirl:
Why Oracle? Why not SQL Server? You're already in a Microsoft environment.

danixdefcon5:
Anyway, at $285k losses ... I think it would've been cheaper to just get Oracle 11g and just do a drop-replacement instead!


Getting away from the Microsoft environment might be the point.

However, I'd go with MySQL. If it's good enough for Google and Wikipedia, it's good enough for whatever you're working on.


PostgreSQL might be also good. Anyway there are a lot of db which are eighter free/open or much less expensive then the crash...


Arguments aside over Mysql's suitability for such an enterprise app, I think we'd all agree that it is still a step up from Access!

Re: A Barely Accessible System

2008-08-14 13:54 • by dgm (unregistered)
The right way to go about this would be to use shadow volume to make a backup copy..

Re: A Barely Accessible System

2008-08-14 13:54 • by Bappi
212066 in reply to 212053
Global Warmer:
bobzilla:
Global Warmer:
...allot...

Grammar Nazi Attack: allot? How about 'a lot'?


How 'bout it

Grammar Soviet stands firm!

Now the eyes of Grammar World are upon the Grammar Western Allies to open a second front in the West!

Re: A Barely Accessible System

2008-08-14 13:58 • by Paul (unregistered)
Does anyone else think of MCL Cafeteria when reading this story?

Re: A Barely Accessible System

2008-08-14 13:59 • by Anonymous Coward (unregistered)
212068 in reply to 212057
Bob X:
Frankly, MS should limit Access to 50 tables per DB, 100k records per table, and 20 concurrent users. That would make it great platform for your tape collection DB, but impossible to use for Ford's production control system. As it should be.


Do you really want to see the ugly ugly workarounds that would exist to get around those limits if that was the case? It might be even worse than example in the article...

Captcha: consequat

Even banks do this

2008-08-14 14:01 • by Paul Keeble (unregistered)
212069 in reply to 212014
6 years ago I came across such an MS Access system in a bank. It held all complaints made against them and did the workflow on resolution of the compliants. With the number of mortgage complaints at an all time high (in the UK) there were about 100 users, doing nothing but using that system and dealing with the complaints, well trying to anyway.

Not only that but the developer was changing the same mdb file as the users were using it! Not QA, now testing, just save it and see what it did! They had corrupted DBs frequently, obviously serious quality problems and it was a complete mess. What did this bank do about it? Nothing not a thing it just kept creeking along.

The morale of this story is that you CAN run a mission critical system on MS Access and while as "real" developers we scoff at everything at this approach MS Access is actually working! That is more than I can say for some Java/C#/whatever apps I have seen.

Re: A Barely Accessible System

2008-08-14 14:02 • by Mark (unregistered)
212070 in reply to 212045
Anonymous:
FOSS Rulz:

Getting away from the Microsoft environment might be the point.

However, I'd go with MySQL. If it's good enough for Google and Wikipedia, it's good enough for whatever you're working on.


Google has their own proprietary database systems for anything involving heavy lifting / mission critical apps; MySQL tends to be for minor stuff.

Some parts of Google (acquired parts) actually do use MS SQL Server, but they try not to do that for too long.


Yeah, I'm getting tired of the fanboys trumpeting that "Google uses MySQL. And Google is HUGE>!>!! OMG111". IIRC, there was an article in Ars Technica a while back looking into Google's usage of MySQL. Turns out they use it for internal stuff like organizing company picnics, etc.

BTW, I'm no hater. I use MySQL anytime it's appropriate.

Re: A Barely Accessible System

2008-08-14 14:04 • by Jethris (unregistered)
Access is a great tool (consider it a small hammer). It's great at hammering nails to hang pictures from, but you wouldn't want to use it to try and bust up concrete.

I developed an Access app that managed mission critical tapes for a branch of the US Military. Granted, all it did was track the tapes (when they were being recorded, what satellite was operational, how long do they need to be kept, which tapes could be reformated, etc.), but it did those tasks very well.

It had 1 data entry user, and 5-10 read-only users. Maybe 2 concurrent user, 3 tops.

SQL Server would have been overkill. Access was cheap, fast, created nice reports, and fit the bill. Downtime for the app had very minimal costs.

I love using Access for what it was designed for, a desktop database. But, for that many records? The lack of transaction logging itself would be scary.

Re: A Barely Accessible System

2008-08-14 14:04 • by Mark (unregistered)
212072 in reply to 212054
Walleye:
tekiegreg:
First thing I'm doing if I get there is upsizing to SQL Server and let access run as the front end only (One of it's redeeming features is access functions well as a front end for SQL Server, can preserve the existing access app but have a more robust database behind it).


This solution always sounds good, until you realize that there will now be lots of brand-new and unhandled exceptions in your production system.


Can someone say "ODBC Error"? Access front end runs this:

DoCMD.RunSQL "Update table set field = value"

And it runs it on a linked SQL Server table the user doesn't have UPDATE privileges on. Good luck trapping that.

OTOH, I agree. Generally, Access does make an easy to develop front-end to SQL Server. You just have to know its limits.

Re: A Barely Accessible System

2008-08-14 14:05 • by Jethris (unregistered)
Access is a great tool (consider it a small hammer). It's great at hammering nails to hang pictures from, but you wouldn't want to use it to try and bust up concrete.

I developed an Access app that managed mission critical tapes for a branch of the US Military. Granted, all it did was track the tapes (when they were being recorded, what satellite was operational, how long do they need to be kept, which tapes could be reformated, etc.), but it did those tasks very well.

It had 1 data entry user, and 5-10 read-only users. Maybe 2 concurrent user, 3 tops.

SQL Server would have been overkill. Access was cheap, fast, created nice reports, and fit the bill. Downtime for the app had very minimal costs.

I love using Access for what it was designed for, a desktop database. But, for that many records? The lack of transaction logging itself would be scary.
« PrevPage 1 | Page 2 | Page 3 | Page 4Next »

Add Comment