As a development manager at a midsize custom software shop, Kyle had seen his share of sticker shock. Fooled by the "Custom Apps, Only $499.99" -- type ads seen in the back of popular tech magazines, naive entrepreneurs would occasionally approach Kyle in the hopes that his company could build the next dot-com wonder-site overnight for $1,000, or maybe $2,000.

For this reason, Kyle had learned to start things off by reminding the seemingly less-serious prospects that custom software is often a five- to seven-figure investment. But before learning this trick, he found out the hard way that many companies have no intention of spending that much on software, no matter how productive their businesses will become or how much money they'll save.

A few years back, Kyle had spent nearly half a week developing an initial proposal for an online, community-based diet-tracking system, only to have Jim, the prospective client, balk at the $70,000 price tag. "Another company quoted us at $6,800," he recalls Jim saying. "Obviously, we're going with them."

Kyle shrugged it off and, for nearly two years, considered it a good lesson learned. But one day, out of the blue, Jim called him back. Having spent more than $100,000 in development alone, Jim's company had little more than a brochureware site that ran painfully slow. They needed some "expert optimization help" to get them back on track.

A Rescue Job

"I realize that we made a pretty big mistake early on," Jim conceded to Kyle. "I mean, we should have never had this thing developed in ASP.NET. It's just far too slow for our needs."

After a few more minutes of conversation, Kyle had a pretty good picture of what had happened. A year after getting started, their diet-tracking Web app was finally delivered, but it never quite made it out of testing. As Jim put it: "Things just acted weird when more than one user accessed the site." So Jim's company cut back on features, and more features, until they finally decided that the real problem was the ASP.NET platform.

A programming team was now hard at work on a PHP-based system, but in the meantime, Jim wanted to launch the company's nearly featureless ASP.NET site. And so Kyle agreed-partly out of curiosity -- to spend a little time looking at their system to see what his company could do.

"Just remember," Jim said, "we're not willing to spend 70 grand like you suggested last time; we've already got a brand-new PHP application on the way for less than half of that."

Kyle remoted-in to the server-the codebase was far too large to send over e-mail-and started poking around. The first thing he noticed was that the app was gigantic. The codebase consisted of two separate and customized versions of DotNetNuke, both of which somehow worked together with several ASP.NET Starter Kits and a mountain of custom code. The majority of the code files were unused and commented-out, partially as a result of the earlier functionality removal and partially to simply get the beast to compile.

Cached Out

While scrolling through the mess of code, Kyle noticed that a large portion of the custom code was responsible for content retrieval and caching. Digging further in, he realized that all content and related files were stored in the database. And not just stored, but represented within a fairly complex "content tree," serialized and then stored.

To render, for example, the ubiquitous logo, the appropriate content tree was retrieved from the database, deserialized and navigated, and then the image's bytes were written to disk and sent to the client. Fortunately, a caching system was developed on top of all this to speed things up.

It didn't take too long for Kyle to figure out that subsystem's fatal flaw: The cache treated the same image-for example, the logo-on different pages as different images. Because each page could have more than 30 content files, including image and CSS files, the cache could grow to an enormous size. That is, if the original developers hadn't limited the cache size to 100.

A cache size of 100 items explained why the site got really slow after clicking through to four or five different pages.

Kyle realized that the Web app had no chance of rescue and needed to be completely rewritten from the ground up. "That's not what we've been told," Jim said upon hearing the bad news. "ASP.NET is just a slow platform, and we simply don't have enough hardware."

Six months later, out of curiosity, Kyle followed up to see how Jim was doing. Unable to find anyone to optimize the site, they had upgraded to four load-balanced servers. As for the faster, PHP version, it was "only a month away."

Cached Out was originally published in Alex's DevDisasters column in the Aug 15, 2008 issue of Redmond Developer News. RDN is a free magazine for influential readers and provides insight into Microsoft's plans, and news on the latest happenings and products in the Windows development marketplace.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!