Testing

Typically in Australia we leave testing as a separate stage late in the development cycle. Developers bash away in a crazed attempt to meet their deadlines and throw the results over the wall at the testers. The result is typically unsatisfying to both the developers and the testers.

Testers perform much of their work at the end of the development cycle. Their timelines typically get squeezed and the result is a product that has not undergone adequate testing and review before final release.

An additional problem is that changing requirements are a frequent cause of rework for testers. As a result of this many projects try to freeze requirements early in the development lifecycle. This is a mistake and presents it's own problems.

For large development efforts the time required to run through a test script can be significant. When you need to make several runs through the test script due to defects the cost becomes prohibitive.

If you reach this stage of development without having any controls over your defects in the earlier stages of development you can face the situation where you thought you were 90% complete but are in reality overruning your schedule to a significant degree.

They key to avoiding getting to the end of development and avoiding a seemingly unending spiral of test and fix is simple.

DON'T LET DEFECTS WAIT!

You can not measure progress of a project if you leave the testing until the end of development. Always make sure the tests are found and fixed before proceeding with the development of new features. The reason to do this is not just to control your schedule but also to reduce the total cost of the project.

The longer a defect goes undetected from the point it's introduced the more difficult, and thus costly, it becomes to fix. A defect that makes it all the way into a production system is orders of magnitudes more costly to repair than if it were found straight away.

The most effective way of removing defects quickly is to have the developers themselves finding the defects using automated testing tools. Waiting for a cycle of manual testing by a separate testing group is leaving things too long. Defects can be detected early using some simple but powerful techniques.

Unit testing isn't a new technique but recently methodologies such as XP has created renewed interest in the benefits of automated unit testing.

The key points with unit test are:

• they should be repeatable
• they should be run frequently
• it should be easy to invoke all unit tests for a system and see the results
• all the tests should pass before checking anything in to the code repository
• they should run as quickly as possible

Frameworks such as junit have become incredibly popular and have been ported to a large number of languages. The apeal with junit is the frameworks simplicity. Each tests is independent of the other and so can be run in any order.

The tests can also be structured in hierarchies which allows you to easily invoke all tests for a particular subsystem.

Unit tests should exist for every non-trivial piece of functionality in the system. This means pretty much everything except for one-liners like getters and setters. Expect to have as much test code as production code.

Unit tests are a powerful technique but on their own they do not provide the whole picture. You also need tests for end-to-end functionality.

Like the unit tests these should be automated, repeatable, run reasonably frequently and easy to invoke. Unlike the unit tests, functional tests don't have a single widely recognised unit testing tools.

Developers should be encourage to develop the test before writing the code. This may sound strange but trust us - it works extremely well.

You shouldn't try to test every possible path through the system with functional tests. The main payback with functional tests are to ensure the system operates as a coherent whole. Use the unit tests to exercise fringe cases.

Functional Testing Tools

Test-first is a unit technique where you actually write the test before writing the actual code. This is sometimes refered to as programming by intention because you're focusing on what you need to do before deciding how to do it.

Test-first is actually more of a design technique than a test technique. For a more complete example of what this is about get a hold of one of the XP series of books. There is also a short article on the web that demonstrates the principle rather well.