greenlightTests. What can you really say about them? They are a royal pain to write and an even bigger pain to keep current as code changes. Nevertheless, we all (should) write them because they help verify that changes to the code don't break something and that the code works properly.

Of course, in order to be of any real value, the tests need to be useful tests. For example, a test to verify that setting the value of an integer to 5 and then immediately reading that integer to see if it's actually 5 doesn't really add any value. The same applies to tests like this:

  @Test
  public void invalidValueCausesExceptionTest() {
    try {
	// do something that causes an exception
    } catch (Exception e) {
      // log that you got the exception
    }
  }

It will always pass because nothing is done to handle the lack of the expected exception.

Numeric tests are interesting beasts. Precision matters. Dividing 4.0 by 2.0 will give a number that can be exactly matched, as-is. Dividing 1.0 by 3.0 without specifying the number of digits of precision, or rounding thereto will likely not match any reasonable value used for comparison.

Derek worked on an extremely high profile two man project that involved pulling together a number of engineering calculations into a single project. The calculations already existed in legacy FORTRAN code which had been written decades earlier in the 70s for mainframes that had less memory than digital watches. To say the code was a nightmare would be an understatement. The programmers did what they could with the resources they had at the time.

Derek was tasked with porting the code and a colleague was tasked with writing all of the tests.

They created a test suite to ensure that all of the ported code produced the same results as the old legacy code. This was their highest priority as these calculations were the backbone to a number of other engineering systems. The results of the new code HAD to be EXACTLY the same as the old results, or at least within reasonably tight tolerances. Otherwise everything the company did (based on those calculations) would be invalid and hence extremely costly.

The legacy code was an out-of-control birds-nest; everything was global and appeared to be dependent upon everything else. It was hard going, and time and time again the tests failed. The reasons why they failed were investigated and solutions implemented. Unfortunately, because most of the calculations were dependent upon each other, tests that once passed would fail when other calculations were ported. It was a nightmare and it was all because of the old coding style.

Although it was very frustrating, over time they started winning the battle; more and more tests were coming up green. More importantly, they were staying green. They started printing off the test reports in all their green beauty and pinned them to the wall for all in the department to see. It was visual evidence that progress was being made and things were getting better. Success was reported up the ladder of management.

There was a new sense of optimism, hope, and happiness. The worst was over. They were on the right track. They could see that there was a light at the end of the tunnel.

Or perhaps it was an on-coming speeding locomotive.

One day while updating the test scripts, Derek noticed something odd in the test output. The calculated values differed between the new code and the old code. The tests were passing and showing up green, but the results were not the same!

When, what, who, why, erm... WTF?!

It turned out that his counterpart, in order to impress his boss, had been increasing the size of the test tolerances on each of the failed calculations in order to make the test pass. In other words rather than fix the calculations, he kept loosening the tolerances of the tests until they passed. It was all a lie. Derek had no idea how long this had been going on, but once the correct tolerances were restored the whole code base failed. Big time!

While management did the right thing and red-lighted fired the tester, they then proceeded to overreact and red-lighted the entire project, thus ensuring that the legacy FORTRAN code lives on.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!