| « Thank you, Javascript | Random Assortment Transfer » |
Back in the early 1990's, G.R.G. worked at a certain university as a programmer. In addition to breaking into server rooms and deafening cute little chinchillas, G.R.G. built one of the university's first web applications. It was a fairly simple CGI program that provided web-access to the student registration known as Old Yeller.
Even in those days, the university’s student registration system was considered to be ancient. Running on an System/370 that had long since seen its day, Old Yeller was based off of an off-the-shelf system and customized over the years with various COBOL and Assembler patches. Though it wasn't too pretty, Old Yeller reliably handled 85,000 registrations each semester and brought in about $400 Million a year. And all in less than a megabyte of RAM.
When the new-fangled World Wide Web came around, G.R.G. saw an excellent opportunity to modernize the registration department’s operations, help decrease the long lines each semester, and, most importantly, work on a fun, new project. Along with his coworker and manager, G.R.G. proposed that they build a CGI interface to Old Yeller that students and faculty could use. They estimated that it’d cost a few thousand dollars in hardware and software, and a few hundred hours to develop.
Though G.R.G. wasn’t part of the official Application Development Unit, the price was right. The registration department approved the project and G.R.G. and his coworker got started on it right away.
A New Face for Old Yeller
After several long weeks of coding in their dark, windowless office – originally a coat room – they had finally completed a web frontend to Old Yeller. It was very basic and provided students with two simple options: add a course using the cryptic COURSE_ID from the printed catalog, or a drop a course using the even more cryptic COURSE_REGISTRATION_ID from the registration confirmation printout. And it all ran on a measly 66-MHz 486 PC. But it was enough.
Students from the pilot group were more than happy to go to the deserted computer lab and punch in their course numbers instead of having to wait in line for hours. The registration folks liked it too, as it meant that much less work. The next semester, the web frontend got a few more features and was opened up to more students. The semester after that, more features were added and all students could use it. And all the while, the CGI ran smoothly on their dinky old 486.
A few years later, the official Application Development Unit finally caught wind of the Old Yeller interface. “It’s pretty cute,” one of the ADU managers said, “but I think it’s time we build a real self-service interface. And this time, let’s build it right.”
Building it Right
Over the following months, the university’s CTO and some other big cheeses worked hard on the Student Registration System Self-Service Interface Pre-Analysis project. Several times a month, they’d fly out to different universities across the world and wine & dine their technology executives in order to get some good inside information on self-service interfaces to student registration systems.
After nine months, the Pre-Analysis was complete. They determined that, in order to build the self-service system right, they’d need a budget of four million dollars and need the assistance of a team of consultants from a certain three-lettered firm. And since that’s what it would take, the trustees signed off and development of “Mother” – the project’s code name – began.
The kick-off meeting for Mother was held at XXX Corp’s offices in Watchagotchie, Canada. Several managers and developers from the university’s ADU flew out to meet the project managers and the forty consultant developers. They all worked in gleaming offices ensconced in tallest tower in Watchagotchie, and discussed how they planned to write the web-interface using the latest tools, in the latest programming languages, with the latest testing tools, and utilizing the latest project techniques.
Everyone was excited: together with XXX Corp, the university would build the greatest self-service interface ever.
Leaves fell, seasons changed, elections passed, and “Mother” was finally delivered. It was powered by a massive $800,000 mainframe with thirty-two CPUs, water cooling, and 256MB of very hot RAM, all which were housed in the university’s new, raised-floor data center. G.R.G.’s CGI – with its 66Mhz PC and 20Mhz backup PC in a broom closet – went from being “cute” to downright pathetic.
Mother Comes To Life
Every new system has its glitches, and Mother was no different. When the registration department opened up Mother to the students, they noticed that, after a few hours of use, the entire system would crash and need to be restarted. Developers at XXX Corp claimed that this was impossible, as they had thoroughly load-tested Mother, but eventually agreed to investigate their testing code.
As it turned out, one of the developers in the gleaming tower coded a small bug in the core of their advanced load testing tools. Though the tool reported that thousands of users could easily use the system simultaneously, the tool inadvertently ran user simulations in serial, not in parallel. When they fixed the tool to run simultaneous user simulations, Mother would crash within seconds.
Developers in the university’s ADU worked frantically with the developers in Watchagotchie to get it working. They were eventually able to massage Mother into handling, slowly and buggily, a small number of users.
With few other options, the registration department posted notices throughout campus asking students to only use the system on “their” day (A-M on even numbered days, N-Z on odd) and that they try to spread the load around the clock. Even so, the load at 3AM was more than Mother could handle and only a lucky few managed to register for their classes.
All the while, GRG’s old interface to Old Yeller was quietly kept online, and word quickly spread that one could actually register on it and not get a Java stack dump as Mother was prone to do. The university’s ADU was adamant that Mother would soon be fixed, and that GRG’s CGI would soon have to be taken down.
With only two weeks of registration left, someone figured out a workaround to get Mother running. Since Mother had a problem with more than a few simultaneous users, they decided to run as many instances of Mother as possible. ADU developers gathered up hundreds of retired PCs, installed Mother on each of them, and wrote a program to randomly redirect users until a Mother with less than five users was found.
That did the trick. Mother was finally able to run, though still somewhat slow and buggily, but actually handle the load. It only took 4,000 times as much hardware as GRG’s CGI.
In the end, the trustees were happy that a self-service interface was successfuly built, and rewarded the folks in ADU with big bonuses and new offices with real windows. They even sponsored a project to investigate finally replacing Old Yeller. As for GRG and his fellow programmer, they stayed in their small coat-room office and got a 2.3% raise at the end of that year.
Re: The Mother of all Interfaces
2008-04-24 10:22
•
by
Asiago Chow
(unregistered)
|
|
And the moral of the story is: If you want all-expenses-paid trips to Canada and raised floors you've got to come out of the closet.
|
Re: The Mother of all Interfaces
2008-04-24 10:39
•
by
NewbiusMaximus
(unregistered)
|
Yeah, I've been on the other side of that "thoroughly load-tested" statement. Every time some customer was told that, it meant they sat down one tester with (maybe) two client computers and (if you're really lucky) some kind of automated or scripted testing software that they might know how to use. Which, of course, gives you spectacular, catastrophic failure when more than 4 people try to use the application simultaneously. Hilarity ensues. Note to self: get in on the wine-and-dine, "get-a-big-bonus when your shitty $4 million Rube Goldbergian system is finally cobbled together" side of the game instead of the "make things work on a reasonable budget" side of the game. |
|
This sounds remarkably like the University of Waterloo, except that I thought the "dinky old CGI PC" was built and maintained by a CS prof in his office, and rather than interfacing with the "real" registration system, it just printed up official course selection sheets (normally filled out by hand) and the prof would have his grad students gather up the printouts and hand them over in batches rather than having every student stand in line separately.
And when they phased in the new super-duper web based system that didn't work and needed everyone to sign up only during their assigned block of time, the old CS-only system went away because they stopped using the old forms entirely. If it turns out that this IS the same story, and the old system was actually on-line the whole time and I just never heard about it, I'm going to be PISSED. Ed: Based on your description ("U of Waterloo", "Professor"), it's certainly a different place... |
Re: The Mother of all Interfaces
2008-04-24 13:11
•
by
Peter Amstutz
(unregistered)
|
|
Something similar happened when I was a student at the University of Massachusetts. The first couple years I was there (late 90s), the standard registration system for classes was touch-tone phone based. It was clunky, annoying, and tended to get overloaded during peak hours, but at least it did work.
Then somebody got the bright idea to build an online registration system. However, instead of building or adopting a system designed for the needs of a large university, they decided to adopt -- wait for it -- PeopleSoft. Since PeopleSoft is an "enterprisy" HR system, students were now "employees", courses were now "work groups" and semesters were now "fiscal years". There was no integration of the actual course descriptions into the system, so you still needed to refer to a completely different web site (or get the phonebook-sized course guide) to figure out what classes to sign up for. Needless to say, the system was a disaster when it deployed: totally unable to handle the load, there was an outcry from students who were unable to sign up for classes due to never actually being able to log in to the system. The user interface was slow and cryptic, and the service got badly confused when users use exotic web technologies such as the "back" button... A couple more years of work and several million dollars later, the system was almost as good as the crappy telephone system it had replaced. |
Re: The Mother of all Interfaces
2008-04-24 13:35
•
by
vt_mruhlin
|
Read the article again. The primary server was in the coat room and the backup was in the broom closet. That's what I call "off-site backup". |
Re: The Mother of all Interfaces
2008-04-24 14:22
•
by
Christophe
(unregistered)
|
Polymorphism in action!!! |
Re: The Mother of all Interfaces
2008-04-25 01:07
•
by
Melbourne Uni Student
(unregistered)
|
|
When I was studying at the University of Melbourne, we were given a case study on a similarly botched system. Our lecturers could hardly contain their glee when they revealed the case study was on RMIT, a competitor university down the road that lost thousands of students due to their PeopleSoft enrollment system collapsing.
We were actually studying its failure while RMIT was still defending it as 'working'. The RMIT vice-chancellor made a classic comment that explained the problem: "We all thought that this project was actually going OK" |
| « Thank you, Javascript | Random Assortment Transfer » |