Comments on: Testing Your Setup Code

By: Symphonious » End To End Testing And The 10 Minute Build

Symphonious » End To End Testing And The 10 Minute Build — Thu, 07 Sep 2006 11:45:45 +0000

[...] At least in my mind, there seems to be a clash of aims in XP. You want to make sure that you have complete confidence in your tests so that you can go faster and reduce the cost of change. To achieve this you write lots and lots of tests - until your fear of something breaking turns to boredom from writing tests you know will pass. Most of those tests are atomic and test a particular component, but fear lies in the gaps between components too so you regularly get recommendations like Ola Ellnestam's on my previous post, Testing Your Setup Code: [...]

By: Adrian Sutton

Adrian Sutton — Wed, 06 Sep 2006 20:51:52 +0000

Olla,
Thanks for that. I can see that an inversion of control library would avoid the problem in your code, but the problem is really just hidden in some third party library which is still a critical part of your app. At least you’re not going to go changing that code so you don’t need the tests as a saftey harness as much but it would still be nice to have a way to effectively test this stuff.

Regarding integration and acceptance tests, I tend to agree - I really like having a very high level of coverage from tests that exercise the whole segment of code, if not end-to-end. The trouble is, these tests are slow and cause your build time to sky rocket. There’s seems to be a lot of people recommending you avoid integration and acceptance tests because of this. I’m not sure what the right answer is, but I’m hoping to think about it some more in a new post in the next couple of days.

By: Ola Ellnestam

Ola Ellnestam — Wed, 06 Sep 2006 08:09:27 +0000

Hey Adrian,

First I´d just like to say I really enjoyed reading your posts. It seems to me you’ve run into
every single problem I have lately. Issues with TDD, continous integration etc.

I´ve run into this particular problem not too long ago. Hope I can help.

When doing TDD you’re forced to do a lot of injection.

In one place or another. Usually at ‘the top’ of an application, the main-class or whatever,
there is bound to be a lot of factories etc.

However you could avoid this by using an ‘inversion of control’-framework. Like Pico or Spring.
I’ve only used Spring and it’s ApplicationContext to achieve this. But I’ve seem other use Pico.

This allows you to configure your way out of the problem. Or rather externalize your configuration.
As a side effect it is easier to get that 100% test coverage. That alone isn’t telling you a lot about your application thou. But it was one of your concerns right?

Note: When you’re TDDing and getting the very loose coupling every one is longing for ;-) you must be aware that integration tests and acceptance tests are an absolute necessity. Since this is the only way to really test your
configuration. But as far as I could tell from earlier posts. You already knew this.

Hope this helps.

Cheers,
Ola Ellnestam

P.S Feel free to contact me if you’d like. D.S

By: Adrian Sutton

Adrian Sutton — Mon, 08 May 2006 02:58:01 +0000

The advantage of 100% code coverage is that you can quickly glance at a report and know that you haven’t slacked off. TDD requires an awful lot of discipline and if you slack off it really comes back to bite you hard. Having even a semi-effective smoke test like 100% code coverage is really beneficial so that you have reasonable confidence that you missed something. Testing until you’re not afraid anymore is a reasonable rule as well, but it is a compromise and not one you should be willing to take unless you’ve investigated the alternatives.

We’re currently in the process of investigating alternatives and looking at how we can test things that we usually just say - oh that can’t be tested. It wasn’t long ago that we were saying we couldn’t test GUI code, but now we have atomic tests for GUI classes that have already caught bugs that would have been hard to track down. Before that we used to think it will excessively difficult to test HTML, now we have libraries that make that job much easier and we’ve caught a lot of bugs because of that.

If you just accept the things that you can’t test, you never make progress on solving them - if you keep looking for better ways to do things and new approaches to testing you start to find solutions to these problems and your codebase improves. I’m pretty sure I don’t want to have a custom classloader involved when running my tests, but if someone has developed one that’s simple to use and they’ve had good results, I’d certainly investigate it further. Mostly though I posted this in the hope that someone would have already encountered these areas and come up with ways to test them.

By: Charlie Groves

Charlie Groves — Mon, 08 May 2006 01:46:49 +0000

If I swapped out my classloader, I feel like it’d enter into the state of things I have to consider when trying to fix something. The point of atomic testing for me is to get the level of complexity down to a point where I can understand all of it one chunk. A possibly broken(at least definitely different from the one in the actual execution environ) classloader adds another variable to consider and pushes me past the level of complexity I can comofortably understand.

I’m not sure why you’re worried about a ’slipppery slope’ of less than 100% coverage. Test until you’re no longer afraid and then test no longer. Adding tests just to hit a line coverage number just increases noise in the system. Substantially reworking the test environment adds noise to the ‘why is this test broken’ process. You have to have some trust in your teammates and yourself that you’ll test the sticky bits so you don’t have to be slavishly devoted to code coverage.