| « The Hard Way | THIS is a Warning Message! » |
1999 was a big year for Harvard Business School Publishing. In the past few years, they had seen their business model – selling books, journals, articles, case studies, and so forth – transform from being entirely catalogue-based to largely web-based, and it had finally come time for a major re-launch of their website.
HBSP’s new website was slick. On top of a fairly advanced search system, the re-designed site also featured community forums and a section called “Ideas @ Work”, which let users download audio broadcasts from influential business thinkers from around the world. And best of all, despite the rapid development schedule, scope creep, and all of the new bells and whistles, the new site actually worked. In the height of the dot-com era, not too many other sites could claim the same.
One key contributor to the success of Harvard Business School Publishing’s new website was its extensive testing and QA. Analysts developed all sorts of test cases to cover virtually every aspect of the site. They worked closely with HBSP’s logistics department to make sure the tests – searching, fulfillment, account management, etc. – were run. And not just run, but run often.
This aggressive testing strategy ensured that the site would function as intended for years to come. That is, until that one day in 2002. On that day, one of the test cases failed: the “Single Result Search.”
The “Single Result Search” test case was part of a trio of cases designed to test the system’s search logic. Like the “Zero Result Search” case, which had the tester enter a term like “asdfasdf” to produce no results, and the “Many Results Search” case, which had the tester enter a term like “management” to produce pages of results, the “Single Result Search” case had the tester enter a term – specifically, “monkey” – to verify that the system would return exactly one result.
And for three years, “monkey” returned exactly one result: Who's Got the Monkey? (full article text) by William Oncken Jr. Written in 1974 Oncken’s article is for managers who “find themselves running out of time while their subordinates are running out of work.” As for the monkeys, they’re just an analogy for work, not who managers should outsource work to. Apparently, Oncken wasn’t that ahead of his time.
In any case, on that day in 2002, the “monkey” search returned two results. The first, as expected, was Who's Got the Monkey?. The second result was something to the effect of Who’s Got The Monkey Now?, which was an update to HBSP’s run-away best seller, Oncken’s 1974 Who's Got the Monkey?. It seemed obvious: the “Single Result Search” test case just needed to be updated. But then they looked into the matter a bit further.
As part of the aggressive testing strategy mentioned earlier, the HBSP logistics team would fill their down time by executing test cases. First they’d run through the “Zero Result Search” test, then the “Many Result Search” test, then the “Single Result Search”. Then they’d add that single result – Who’s Got the Monkey? – to their shopping cart, create an new account, submit the order, and then fulfill it. Of course, they didn’t actually fulfill it – everyone knew that orders for “Mr. Test Test” and “123 Test St.” were not to be filled. That is, everyone except the marketing department.
When HBSP’s marketing department analyzed the sales trends, they noticed a rather interesting trend. Oncken’s 1974 Who's Got the Monkey? was a run-away best seller! And like any marketing department would, they took the story and ran. HBSP created pamphlets and other distillations of the paper. They even repackaged those little plastic cocktail monkeys as official “Who’s Got the Monkey monkeys”. And finally, sometime in 2002, the updated version of Who’s Got the Monkey? was posted to HBSP, which was then picked up by the searching system, which, in turn, caused the “Single Result Search” test case to fail.
Of course, by this point, there was little anyone could do. The fictional success of Who’s Got the Monkey had already been widely publicized as reality. And with all the subsequent write-ups (many of which are still around to this day), it may have very well become a best-seller. Needless to say, HBSP has since changed their aggressive testing policy.
Some details of the story have been redacted to protect the guilty. Thanks to the two anonymous sources working at HBSP for the inside scoop, and news archives for the rest.
UPDATE: Bill Damon from Harvard Business added, “our web development environment has changed dramatically in the last 9 years – in fact, since 2002, we’ve had a up a test order system specifically to handle these requests, and no testing orders made it into the sales data. Also, I do know from historical sales data that Management Time: Who’s Got the Monkey? was a bestseller long before 1999, and it continues to perform really well in reprint sale. We also don’t merchandise ancillary products such as plastic cocktail monkeys.”
|
About 9 months ago I was working for a reatail consultancy doing a bit of work for one of their clients. There was another team at the consultancy who were doing a major overhaul of a major UK retailers website, specificaly integrating the existing site with a new merchandising engine that did the whole 'Users who brought x, also brought y'
Hundreds of times a day automated load scripts ran on the UAT enviroment for several months leading up to the launch, these scripts ran all day long. The test script developer picked two films out at random, which happend to be an 18-rated porno, a rammstein album and a childrens film, the script purchased them. What he didn't realise was that the UAT system was actually going to be the live system when the switchover took place on the going live date. First day of the launch resulted in the quicked bug fix ever, people looking at childrens dvd's were also being peddled german industrial rock and porno's in the cross merchandising section. Brilliant! Addendum (2008-02-05 10:53): Sorry, Brillant! |
|
"HBSP has since changed their aggressive testing policy."
Why? The test procedure was more effective than required, not only showing that the web site worked, but that marketing's procedures worked, successfully identifying and promoting a best selling article. They need more tests like this to also include inventory, personnel, accounting, enterprise strategy, etc. --Rank |
|
At the end of every year, eBay released a list of the top-ten search terms used on their website. Typical top-ten terms were "BMW" and "Gucci" as well as other higher-end "Rodeo Drive" style items. And every year, USA Today and other mainstream media outlets would write an article about how well the economy must be doing because everyone on eBay was searching for high-ticket items.
Earlier this decade, eBay released an upgraded search product written largely in-house. A monitoring infrastructure was customized to ensure smooth functioning of this new search product. USA Today again wrote their annual report on eBay search terms, but in 2004, there was a surprising change... no longer were the "Rodeo-Drive" items topping the list, no... in the words of the USA Today writer, the sudden appearance of "moon" and "RV" strongly suggested "...that people are returning to their childhoods for comfort in discomforting times and that we are captured once again by dreams of space travel..." http://www.usatoday.com/tech/news/2004-12-29-ebay_x.htm Other journalists concluded that the US Economy was heading for recession as the desire to find "RVs" meant less spending on more frivolous items. The truth of the matter is, the two engineers who developed the monitoring infrastructure for the new search product simply wrote scripts that searched multiple nodes, 1000's of times each day, for search terms that would be considered somewhat improbable by anyone else: their initials or their last name. |
| « The Hard Way | THIS is a Warning Message! » |