Comment On Small But Rapidly Growing

The job listing seemed pretty normal — small-but-rapidly-growing software company needs top-notch developers to help expand flagship product — but there was one little detail that seemed a little... off. Near the bottom of the ad, it read note that we do not make custom-made software. David didn't think much of it at the time, especially since the company's website also reiterated that same point. He figured it must have been some sort of industry thing.

David sent in his résumé and was fortunate enough to get an in-person interview with Frank, the company's CEO. The receptionist smiled and replied in a perky voice, "great! I'll let Frank know; just take a seat and he should be here soon."

As David turned around to head towards the chairs, he almost ran into a man who was standing not eight inches behind him. "You must be David," the well-dressed, middle-aged fellow responded, "I'm Frank, the CEO. Let's head back to my office and chat."

And all told, the interview went exceedlingly well. Frank reiterated that the software was not "custom made", but was instead a pretty neat product that was really starting to pick up on the sales side of things. As Frank put it, the industry that Xirpol Ltd. was a part of was spyware. Not the sleazy, malware kind of spyware, but the "good" kind that computer owners would install on their own machines to spy on the users. This type of software was mostly used by companies to watch over their employees, but schools and occasionally parents could also find a use. Xirpol's flagship program — also named Xirpol — would gather all types of information, push it up to a database, and then allow administrators to view it with lots of pretty graphs and reports.

If schools wanted to know how many computers accessed certain sites during recess, Xirpol could do it. If an IT company wanted to measure the hard disk usage of its workstations so they could predict when to buy more drives, Xirpol could do it. Xirpol could also take screenshots, access user-stored passwords, move the user's cursor remotely, and so on which, depending on the use, could either be used for constructive or destructive purposes. Overall, the software sounded really neat and David was elated when Frank offered him a job as a developer. He accepted the offer and started at Xirpol the following week.

It's Customizable

Because Xirpol was a small company, there wasn't a whole lot of room separating the sales and development teams and, as such, the devs were all too familiar with the just-closed-a-sale gong. However, instead of excitement when the gong rang, the developers would always respond with a collective groan. But not because of the distracting nature of loud and obnoxious sounds in the office, but more because they knew that a sale meant customization.

Although Xirpol did not custom-make software, they would customize their software to the needs of the client. This was an important distinction that only Frank seemed to understand. If the application couldn't open CD drives remotely, then as soon as someone bought that feature, developer would be responsible for adding it in. And since the sales personnel were paid on rather large per-sale bonuses, they'd include whatever features were necessary to close the deal.

One time, when a government contractor client wanted the application to gather usage data from before the application was installed, the sales team wrote it in the sales contract as a "small, insignificant feature that would only take our development team five minutes to write." In reality, retrieving unrecorded data from the past is basically impossible without time travel. Naturally, when the developers were unable to bend time, they ended up taking the blame because, after all, the customer had already paid for the feature upgrade.

Need-to-please Basis

One of David's first lessons at Xirpol was issue prioritiation. With the large number of sales and a small number of developers, the sold-but-not-yet-coded list of features was quickly growing. Because of this, Frank had instituted a policy called "code on a need-to-please basis" that was used to know which features should be implemented first.

The policy simply stated that the first features to be implemented were the ones that Xirpol was currently being sued for non-delivery. Then came the ones they were currently being threatened to be sued. After that was the management changes (such as prettier icons). Finally the changes ones developers wanted (such as encryption for the channel in which sensitive information was being transferred). Interestingly enough, fixing bugs was nowhere to be seen in this policy.

Obfuscation by design

One of the biggest challenges with Xirpol was that maintenance was no picnic. Originally built by Frank, the code was filled with all sorts of obscure features such as "assembly optimized additions" which were allegedly faster than the additions created by the compiler. Another feature was the quintuple-triple-DES encryption, which encrypted data files five times in a row with triple-DES, always with the ultra-secure hardcoded password "X1Rp0L"... which would be clearly visible with any freeware resource viewer.

Because the application was sold to government agencies and ministries and the like, it was marketed as "ultra-secure", and "hacker-immune". In order to achieve this, Xirpol program was bundled in one .exe file and several .dll "module" files. These module files were not actually meant to make the application modular and easy to maintain. Instead, they were there to make it more difficult for would-be hackers to disassemble and understand the application. So instead of module_net.dll and module_core.dll, there was module_00.dll, module_01.dll and so on up to module_15.dll Also, new functions were to be randomly placed in an existing module for further obfuscation.

The obuscation certainly worked: the result was so obfuscated that not even developers knew where to find what they were looking for.

SQL Sentences

If the heading SQL Sentences looks familiar, it's probably because there's an article of the same name that was posted in February of 2008. There's a good reason for that: it was the same company and the same David.

Frank figured that, since a large portion of their application dealt with saving and retrieving data from the database, then a lot of the bugs must be a result of bad SQL queries. Likely, it was things like a LEFT JOIN instead of an INNER JOIN, or a > instead of a >=. And, since their application already used a database, he figured why not store all of the SQL queries in the database?

The idea was handed off to the lead developer, who then worked long and hard to transform it into something far, far worse: the “SQL Sentences” library.

• every single SQL query was to be stored in a table called sql_sentences
• in order to not “hard code” the purpose of a SQL query in its identifier, SQL sentences were given 4-digit hex identifiers - 0x00F9, 0xA004, etc. - instead of GET_ALL_CUSTOMERS
• each sentence has privileges associated with it; the GetSQLSentence() method made several other queries to make sure the logged-on user could execute the query
• to appease the developers who were against storing SQL in the database, a configuration value called “UseHardcodedSQL” was added

All told, these changes required developers to modifiy their code to look like this...

string sql;
// @SQLSentence=0x02f0
if ((bool)(Configuration.GetValue("UseHardcodedSQL")))
{
    sql = "SELECT cust_name, cust_phone, cust_addr " +
          "  FROM customers WHERE cust_type='Active'";
}
else
{
    sql = DataConnector.GetSQLSentence(0x02f0);
}

... and, of course, add that same query to the sql_sentences table. And, should there ever be a change to the query, maintain it in both the code and the gigantic script created the sql_sentences table and the production database.

Versions, oh the Versions

With so many features to implement and the ever-growing sales department overwhelming the stagnant development department, developers had to be able to pull new software versions as fast as possible, each version with a new feature, or a bug fixed. The original idea was that there should be a new version each three months, but that never happened. During busy times, they could have created 10 new versions in a single day.

Versions were created so quickly, and the installer creation process was so convoluted, that sometimes a single developer would spend his entire day only creating these "versions". Of course, this developer would have to prepare for disciplinary action, as Frank would complain that he wrote no code during the day.

Economical Hardships

Because of the nature of Xirpol's feature delivery, lawsuit threats was an everyday thing. More importantly tough, many of the clients would simply stop paying the costly monthly fees until their problems were solved or their features were implemented.

This lack of payment translated into cashflow problems, which in turn translated to late paychecks. Of course, because the bugs and feature delivery problem were always the developers' fault, they were generally the only ones made to suffer by receiving their paychecks weeks late.

Oddly enough, the financial problems seemed to stop there, as Frank had no qualms about buying the entire 31st floor of a downtown building, furbishing it with glass partitions, and adding-in electronically-operated doors.

The Last Straw

David never would have guessed that he'd have so many different reasons to a quit his job. But the final straw for him was the Seating Assignment Projects. When Frank assigned the project to him, it was already three months behind the original deadline and not a single line of code had been written. He didn't mind that so much; the annoying part was when Frank called him up not ten minutes later to scold him for having a project that was three months behind schedule.