Everybody seems to be using Fitnesse or Cucumber for integration testing these days. The main idea is that users (testers/business analists and programmers) can write what your program should do in plain text. The programmer then creates a small layer of code which gets interpreted and executes your actual program to see if the specified requirements are met.

Fitnesse example

For example a (hypothetical) piece of code in Fitnesse as it could have been used in the Port of Rotterdam:

!3 Anchorage tests

!|given a ship                                |
|Id  |Name       |Category|CallSign|Shipnumber|
|1001|Maersk Bali|TANKER  |D6DZ8   |7420211   |

|script                                                              |
|When a ship called|MAERSK BALI|reaches the port                     |
|And it reports anchor down                                          |
|ensure            |the ship is shown in the list of anchored vessels|
|check             |the list of anchored vessels now has size|1      |

This is something the testers and product owners can read and understand. Behind this layer of text Fitnesse will try to execute pieces of code. This is left as an exercise for the programmers to complete. They’ll need to write something like:

GivenAShip:
void setId(String id);
void setName(String name);
void setCategory(String category);
void setCallSign(String callSign);
void setShipnumber(String shipnumber);
void execute(); //Stores the ship in the database

AnchorageScenarios:
void whenAShipCalledReachedThePort(String shipName);
void andItReportsAnchorDown();
boolean theShipIsShownInTheListOfAnchoredVessels();
int theListOfAnchoredVesselsNowHasSize();

This is where the problems start, there is a lot of room for typos. Which will send you all over the codebase constantly translating Fitnesse text like “and it reports anchor down” to “andItReportsAnchorDown” during your searches.

Also people always say, because it is ‘plain text’ the testers, analists and product owners can instantly start writing tests. But this just isn’t true… It isn’t just writing down tests/requirements, it is programming with a DSL. Normal human text isn’t good enough, it still needs some DSL structure. They’ll have to learn to program in the Fitnesse DSL language. In almost all project I’ve worked on, it is the programmers writing and maintaining tests, people from the business stay far away from Fitnesse.

If you want to refactor the scenarios/fixtures in Fitnesse you’re in for a treat. Everything is bound by name only, and renaming code for example won’t rename the DSL! Using Fitnesse ‘sounds’ very agile, but it doesn’t really make the code agile at all. Most test code I’ve worked on is fragile, very ridged and not easy to refactor. Why not take advantage of that IDE and statically typed language if the programmers are writing and maintaining the tests?

Using code instead

Today we had a “ShipIt”-day (nautical-pun intended) at work, where we’re free to work on any project for 24 hours straight and we present our findings afterwards. This boosts the creative ideas and in the long run efficiency and innovation.

After using Fitnesse now for three years we’ve become used to it, and don’t ever question its usage. Although we feel the pain of the rigid/fragile test code on a daily basis. I decided to start rethinking the way we do acceptance tests.

My goals was to make our tests:

• Simpler to write
• Easier to refactor
• Maintain the readability
• Easier to reuse
• Faster! (no DSL parsing)

This is what the initial design now looks like. The example used above would become the following:

@Jitness        // Annotation that indicates this is a test class
@SuiteSetUp({   // This is called once for all the methods in the class below
	@Execute(
			type = ShipExecutor.class,
			data = { "1001", "Maersk Bali", "TANKER", "D6DZ8", "7420211" })
	})
})
@SetUp({
	       // @Execute annotations here will be called before each of the methods below
	       // The @SetUp annotation can also be placed on a single method in the test class
})
public class AnchorageTests {

	public void givingCommandAnchorDownCausesShipToAppearInList() {

		final AnchorageScenarios scenarios = new AnchorageScenarios();

		scenarios.whenAShipReachedThePortCalled("Maersk Bali");
		scenarios.andItReportsAnchorDown();
		Assert.assertTrue(scenarios.theShipIsShownInTheListOfAnchoredVessels());
		Assert.assertEquals(1, scenarios.theListOfAnchoredVesselsNowHasSize());
	}

	public void anotherTestHere() {
		... etc ...
	}
}

Using Jitness (working title) the scenarios will be implemented in exactly the same way we’ve already been doing. This makes it easier to transition from Fitnesse to this code.
The only bit of ‘magic’ is the execute annotation. You can write executors like this:

public class ShipExecutor implements Executor {

	private ShipRepository repository = new ShipRepository();

	@Override
	public void before(final String[] args) throws Exception {
                repository.store(parse(args));
	}
	
	@Override
	public void after(final String[] args) throws Exception {
                repository.delete(parse(args));
	}

        private Ship parse(String[] args) {
		return new ShipBuilder()
                         .id(Long.parse(args[0])
                         .name(args[1])
                         .category(args[2])
                         .callSign(args[3])
                         .shipNumber(args[4])
                         .build();
        }
}

A (more elaborate) sample of the output it generates, with failure:

Results: 
OK	BunkeringTests.SuiteSetUp
OK	BunkeringTests.addTypeOfFuelToBunker
OK	BunkeringTests.changeTypeOfFuelToBunker
OK	BunkeringTests.removeTypeOfFuelToBunker
OK	BunkeringTests.bunkeringOnIncomingMovement
OK	BunkeringTests.bunkeringOnOutgoingMovement
OK	BunkeringTests.loadBunkeringForOpenVesselVisit
NOK	BunkeringTests.retrieveListOfBunkerBarges
expected:<1> but was:<2>
OK	BunkeringTests.SuiteTearDown
Took: 128 ms

The main advantages of using this are obviously leveraging the full potential of your IDE, including renaming/refactoring. Changing methods and classes in your production code will automatically show up as changes in Jitness. And everything is easy to refactor because it is just code! This will eventually lead to much more agile code, simpler and easier to refactor. Obviously there is one main drawback, it is harder for testers to read and write. But if they pair up with a programmer this should be no problem I think. I’d rather have the testers learn a bit of coding than the programmers learn a text DSL which leads to fragile tests.

Another big advantage is execution time, using compiled code is always faster than parsing plain text or HTML at runtime. Also this code is compiled during the build, so there is no need for additional classpath (Maven) plugins. I’m really fond of the idea to use code to describe the tests, and I’m very curious how my colleagues are going to rate this tomorrow when we demo the “ShipIt” ideas. With a couple of more days work this can be fully tested and made robust enough to start using it and maybe even open source it.

What are your thoughts? Any ideas about improving the API? Should we just keep using Fitnesse/Cucumber or any other alternative DSL instead?