Symphonious

The basic idea of unit tests is that you write them once and run them a lot - you don’t check in your changes unless all the unit tests pass.  This is a great idea.

There is one problem with it - it prevents unit tests from being used as a specification of work to be done later.  I don’t mean it prevents test driven development where the tests are written then the code is to make them pass is added and then the tests and code are checked in.  I mean where a particularly complex piece of functionality is going to be added and the best way to ensure that all the use cases are covered is to write unit tests for them all and let them act as the specification (note: this doesn’t work well if you have a non-technical client, but if you are working on off the shelf software it can be very useful).  You need to plan the set of features to include in a release ahead of time so you wind up with a whole heap of new unit tests which are acting as this specification that at some point in the future you will make pass.  At the moment though, those tests are okay to fail.

How do we preserve the ability of unit tests to ensure no regressions while still being able to use them as a to do list?

There are a few options.  The most obvious is to split the tests up into stuff that’s done and stuff that’s not done and make them separately runnable.  The standard build process only runs the "stuff that’s done" set of tests and the others can be ignored until those features are to be implemented.

This however has the disadvantage that if some of the to do tests already pass (the functionality is partially present or is fully available from buggy pre-existing code etc) or if over time some of the tests begin passing because of other changes, it would be nice to ensure that progress is not lost inadvertently.

The solution I’m trying out is to create a list of tests that are okay to have fail for whatever reason and a custom ant task that reads that list and the JUnit reports and fails the build if any of the tests that should pass didn’t but lets it pass even if some of the to do tests fail.  The ant task then tasks the next step and automatically identifies any to do tests that now pass and removes them from the list of tests that can fail.

With that setup once a test passes once it is never again allowed to fail.  Every change is then a step forward.  You can even use the number of tests remaining in the to do list as a rough progress meter if the functionality takes a number of steps to implement or has multiple developers working on it.

It should be interesting to see how this pans out anyway…

I’ve spent a large chunk of this week investigating what the best way to set up continuous integration builds is for our product.  Being a cross platform application we need a continuous integration set up that builds on four different OSs constantly (Windows, Mac OS X, Solaris and Linux).  Ideally of course it would do so on all variants of those OSs and at some point that might be possible using vmware of some kind.  For now though, just building and running the automated tests on some version of the OSs we support is a huge improvement.  Fortunately, our build process already uses ant and requires just a single target be achieved.

I’ve previously tried setting this up using AntHill with no success.  My mind is a little vague on the details of how it went wrong but it did go very, very wrong.  I’ve also tried writing my own system with cron, scheduled tasks, shell scripting and ant and I think it’s still running but it doesn’t give any useful feedback so it’s as good as useless.

This time round I looked at hudson straight off the bat and found it couldn’t figure out CVS for a variety of reasons.  I tried to investigate the problems a little further but couldn’t checkout the source code from CVS - most likely the server was having a bad day.

I looked at Damage Control and it’s releases seemed to have issues with files being missing (including the executable itself in one build).

Finally I tried CruiseControl despite the fact that I don’t like its design much.  It appears to be working so I guess that’s what I’m going with.  So far I’ve only got it running on Windows though I had it running on OS X for a bit.  The next step will be to set it up dedicated build machines running the various OSs and get it running on that.  Then I need to come up with a way to combine the results from each of the build servers into a single dashboard display so that the build results are more immediately obvious.

Once all that’s running I’m hoping it will be easy to add our other products into the build servers and having everything building away happily.  Overall I’m amazed at how many problems I’ve had with the various continuous integration tools that are out there.  I’m sure it shouldn’t be as difficult as it has been…

As an end to this rambling monologue - any recommendations of good resources for continuous integration, unit testing and general code quality improvement techniques?  I’m thinking I should pick up a copy of "Pragmatic Project Automation" and probably also "Pragmatic Unit Testing".  Any others?  Any other techniques I should be looking at either before, after or during the process of setting up continuous integration?

I noticed today that The Kleptones have released an mp3 version of their broadcast on "The Rinse" entitled "From Detroit To J.A."   It’s not as good as "A Night At The Hip Hopera" but it is definitely enjoyable.  It involves a lot of Motown material but unfortunately they too often took the average lyrics instead of the sensational music.

My pick of the series would have to be "Really Rappin’ Something" mostly for it’s driving baritone sax line.  I must admit though that the use of the music from "Ben" (early Michael Jackson) as the backing for "Revolverlution" was an inspired choice.  I’m not sure where the words over the top came from but they are also brilliant - "The Revolution Will Be Televised".

Ben Fowler points to AppFuse, a quick way to set up J2EE projects with more buzz words than you can shake a stick at.  Specifically it sets up a Tomcat/MySQL apps that:

Uses Ant, XDoclet, Spring, Hibernate (or iBATIS), JUnit, Cactus, StrutsTestCase, Canoo’s WebTest, Struts Menu, Display Tag Library, OSCache, JSTL and Struts (or Spring MVC)

That’s a huge number of different technologies.  It’s not necessarily bad to use all those technologies together (in fact it’s probably very good) but it’s important to be aware that pragmatic programmers don’t use wizards they don’t understand.

So unless you are already very familiar with how to set up a project using all of those technologies and understand exactly what AppFuse is doing for you, don’t use it.  At some point the code and configuration files that AppFuse sets up are going to become code that’s a part of your project and they will be yours to maintain.  If you don’t understand what they do, how are you going to fix problems that show up or make changes later?

Having said that, for seasoned veterans who understand all of those technologies well, AppFuse should be a big time saver.  I can see a lot of beginners getting into very deep water with it though.

It’s funny, last night I read this reuters article about Microsoft beginning an initiative to improve their interoperability and thought wow that’s excellent!  I bet Slashdot completely misses the point of the message and rants on about how Bill Gates also mentioned that opensource doesn’t necessarily lead to interoperability.  Sure enough, Bill Gates Claims Linux Has Poor Interoperability.  Sigh.

Gates of course is right - there are plenty of open source products out there that aren’t interoperable in one way or another.   Further, while Linux systems generally adhere to standard network protocols well, they have an abysmal track record of interoperability on the same system.  The best example of this would be the lack of robust drag and drop support.

Where predefined standards exist, open source software generally follows them very well.  Where predefined standards don’t exist, open source software typically struggles to agree on a new standard.

Far more important than the typical marketing debates going on between the Microsoft and Linux camps (both of which are happily spreading falsehoods about the other) is the fact that Microsoft is going to focus on improving their interoperability.  That should be applauded, not criticized.  Sure it came along with a shot at Linux but that’s what happens when there’s competition.  The response from Linux advocates should be "we applaud Microsoft’s decision to improve interoperability and are proud that the determined efforts of open source projects have driven them to this point".  In other words, applaud the good thing that Microsoft is doing and then tag along your own marketing hype with that response.  Don’t just run around like a spoiled child who has just been rebuked.