Testing Your Setup Code
In order to write atomic tests for classes I've gotten into the habit of injecting dependencies instead of creating them inside the class. This makes it simple to mock the dependencies but means that those dependencies are now externally visible instead of being neatly encapsulated and it means that somewhere in your code, there has to be a place that actually creates all these instances and passes them in.
I have no idea how to test that bit of code.
This pops up in a couple of places I've been working on lately, firstly the main method of a program. How do you atomically test a main method that by definition has to setup all the different objects and initialize everything. I suppose I could use a factory pattern but it seems like overkill. How are people testing these things?
The other problem area is in creating dialogs. I can separate out all the business logic and atomically test that, I can even use JFCUnit to test the GUI code for accessibility etc, but at some point my business code has to create the dialog and show it - how do I test that? I'd really like to keep my business code with 100% code coverage to avoid the slippery slope of just saying "oh I can't test that" (which we've been sliding down a bit). I do need to make sure my atomic tests run really fast though.
Maybe I just have to live with no atomic tests for that bit of my code and leave the testing to integration tests but it opens up a whole in our TDD atomic test coverage which I'd like to avoid if I can.
Has anyone found a way to play with class loaders so that a call to new MyObject() actually creates a mock object/stub instead? Is that so ridiculously complex to do that you'd have no confidence in the resulting tests anyway?
May 8th, 2006 at 11:46 am
If I swapped out my classloader, I feel like it’d enter into the state of things I have to consider when trying to fix something. The point of atomic testing for me is to get the level of complexity down to a point where I can understand all of it one chunk. A possibly broken(at least definitely different from the one in the actual execution environ) classloader adds another variable to consider and pushes me past the level of complexity I can comofortably understand.
I’m not sure why you’re worried about a ’slipppery slope’ of less than 100% coverage. Test until you’re no longer afraid and then test no longer. Adding tests just to hit a line coverage number just increases noise in the system. Substantially reworking the test environment adds noise to the ‘why is this test broken’ process. You have to have some trust in your teammates and yourself that you’ll test the sticky bits so you don’t have to be slavishly devoted to code coverage.
May 8th, 2006 at 12:58 pm
The advantage of 100% code coverage is that you can quickly glance at a report and know that you haven’t slacked off. TDD requires an awful lot of discipline and if you slack off it really comes back to bite you hard. Having even a semi-effective smoke test like 100% code coverage is really beneficial so that you have reasonable confidence that you missed something. Testing until you’re not afraid anymore is a reasonable rule as well, but it is a compromise and not one you should be willing to take unless you’ve investigated the alternatives.
We’re currently in the process of investigating alternatives and looking at how we can test things that we usually just say - oh that can’t be tested. It wasn’t long ago that we were saying we couldn’t test GUI code, but now we have atomic tests for GUI classes that have already caught bugs that would have been hard to track down. Before that we used to think it will excessively difficult to test HTML, now we have libraries that make that job much easier and we’ve caught a lot of bugs because of that.
If you just accept the things that you can’t test, you never make progress on solving them - if you keep looking for better ways to do things and new approaches to testing you start to find solutions to these problems and your codebase improves. I’m pretty sure I don’t want to have a custom classloader involved when running my tests, but if someone has developed one that’s simple to use and they’ve had good results, I’d certainly investigate it further. Mostly though I posted this in the hope that someone would have already encountered these areas and come up with ways to test them.
September 6th, 2006 at 6:09 pm
Hey Adrian,
First I´d just like to say I really enjoyed reading your posts. It seems to me you’ve run into
every single problem I have lately. Issues with TDD, continous integration etc.
I´ve run into this particular problem not too long ago. Hope I can help.
When doing TDD you’re forced to do a lot of injection.
In one place or another. Usually at ‘the top’ of an application, the main-class or whatever,
there is bound to be a lot of factories etc.
However you could avoid this by using an ‘inversion of control’-framework. Like Pico or Spring.
I’ve only used Spring and it’s ApplicationContext to achieve this. But I’ve seem other use Pico.
This allows you to configure your way out of the problem. Or rather externalize your configuration.
As a side effect it is easier to get that 100% test coverage. That alone isn’t telling you a lot about your application thou. But it was one of your concerns right?
Note: When you’re TDDing and getting the very loose coupling every one is longing for ;-) you must be aware that integration tests and acceptance tests are an absolute necessity. Since this is the only way to really test your
configuration. But as far as I could tell from earlier posts. You already knew this.
Hope this helps.
Cheers,
Ola Ellnestam
P.S Feel free to contact me if you’d like. D.S
September 7th, 2006 at 6:51 am
Olla,
Thanks for that. I can see that an inversion of control library would avoid the problem in your code, but the problem is really just hidden in some third party library which is still a critical part of your app. At least you’re not going to go changing that code so you don’t need the tests as a saftey harness as much but it would still be nice to have a way to effectively test this stuff.
Regarding integration and acceptance tests, I tend to agree - I really like having a very high level of coverage from tests that exercise the whole segment of code, if not end-to-end. The trouble is, these tests are slow and cause your build time to sky rocket. There’s seems to be a lot of people recommending you avoid integration and acceptance tests because of this. I’m not sure what the right answer is, but I’m hoping to think about it some more in a new post in the next couple of days.
September 7th, 2006 at 9:45 pm
[...] At least in my mind, there seems to be a clash of aims in XP. You want to make sure that you have complete confidence in your tests so that you can go faster and reduce the cost of change. To achieve this you write lots and lots of tests - until your fear of something breaking turns to boredom from writing tests you know will pass. Most of those tests are atomic and test a particular component, but fear lies in the gaps between components too so you regularly get recommendations like Ola Ellnestam's on my previous post, Testing Your Setup Code: [...]