Symphonious » Code and Geek Stuff

Ant Dependencies Running Twice

Adrian Sutton — Fri, 10 May 2013 08:42:25 +0000

Most people believe that ant will only ever execute a target once per execution of ant. So if we have targets A, B, C and D, where A depends on B and C and C and D both depend on D.

When we run ‘ant A’, we expect each task to execute once, D first, then B and C, ending with A. Since there is no dependency between B and C, they may execute in any order, so we might also get D, C, B, A. All pretty straight forward.

Similarly, if we run ‘ant B’ we expect (and get) D then B and ‘ant C’ gives D then C. Most people expect that if you run ‘ant B C’, the execution path would be D, B, C but that’s not what happens. Instead we get D, B, D, C. Each target passed to the command line creates a new dependency chain and runs all targets in that chain.

The other case that catches people out is when using antcall. The antcall task actually sets up essentially a new execution of ant (by default inheriting any properties that have already been set). It never inherits the dependency chain or list of targets that have already been executed. So if our targets were:

Then the execution order for ‘ant A’ would be A,D,B,D,C.

The cost of running targets twice is reduced because most ant tasks do nothing if their inputs haven’t changed (so javac only compiles files once), but for a large project there can still be a significant cost in checking over all the files to determine if anything needs to be done. So whenever possible, use actual dependencies to build up a chain of targets instead of antcall or specifying multiple targets on the command line.

Doctypes, Compatibility Modes, Charsets and Fonts

Adrian Sutton — Fri, 26 Apr 2013 01:08:22 +0000

This information is all covered in much more detail elsewhere on the web but for my own future reference, here’s a primer on doctypes, compatibility modes, charsets and fonts which was required to explain why certain Chinese characters weren’t showing up in IE8. Of course the best answer is that you need to have the East Asian Font pack installed and then it just works (usually) but this tends to be useful background and saves “server side” folks from a number of gotchas.

Doctypes and Compatibility Modes

IE 7 and above has an insane array of compatibility modes which are out to get you. The most common gotcha is that it will use compatibility mode (emulating IE7) if the website is in the “intranet zone”. There’s an option to disable this somewhere in the preferences dialogs. You wind up in the intranet zone if you’re accessing a site via any domain name that doesn’t look like a real one (e.g. http://dog/ is in the intranet zone).
If you can avoid falling into compatibility mode, any pages including the DOCTYPE as or will render in standards compliant mode (where the world is as sane as it gets in web development). Go with the shorter version unless you have a reason not to.
http://hsivonen.iki.fi/doctype/ is the gold standard for information about browser modes.

Character Sets and Fonts

If you’re tracking down problems with foreign languages there are two major categories of problems – encoding corruption (where characters come out garbled or as ?) and missing glyphs in fonts (where characters come out as little boxes).
Corruption is fixed by specifying the same character encoding everywhere. It is a security issue if any webpage is missing a meta tag defining the character set (smallest variant is ). It must be the first tag in the of the document.
Little square boxes mean that either the font currently in use doesn’t have a glyph for that particular character and the font fallback routine was unable to find any font on the system which supports that character.
Browsers have a default stylesheet which is automatically applied to every page which commonly sets a specific font-family and font-size for text input elements, so adding the style body { font-family: 'Arial Unicode MS' } may get some Asian characters working in the main content but not in text boxes unless you also add input { font-family: inherit; }.

The security issue mentioned above is that any page which doesn’t define a character set but includes any form of user supplied content is vulnerable to a cross site scripting injection attack – even if the user supplied content is escaped properly, because the content may include a character that causes the browser to incorrectly switch to UCS-7 or other weird character sets and drastically change the meaning of the content on the page (hence the user content is no longer correctly escaped). There have been steps taken by modern browsers to remove this risk (including removing support for UCS-7 I believe) but its good practice to specify your charset explicitly anyway.

Demystifying Doubles: Consistent Inaccuracy

Adrian Sutton — Wed, 29 Aug 2012 03:36:19 +0000

Of all the data types, double is probably one of the most misunderstood. A huge amount of folk lore has been built up around it to help protect developers from falling into its many pitfalls. Lately I’ve done a lot of work replacing usage of BigDecimal with double and learnt a lot about where those pitfalls are and how the folk lore can be misleading.

The great challenge with double is that it has a degree of inaccuracy because of the way the number is actually stored. For example, the number 2983792734924.2293 actually winds up being stored as 2983792734924.2295, a tiny difference but with far reaching consequences. All of the folk lore around doubles is designed to deal with this potential for inaccuracy.

One piece of folk lore is that you should never do a strict comparison of double values (e.g. with == or !=). Due to the inaccuracy of doubles you should always compare them with a level of tolerance:

abs(doubleA - doubleB) < tolerance

It’s important to realise however, that this inaccuracy isn’t random, it’s perfectly consistent and predictable. The number 2983792734924.2293 always comes out as 2983792734924.2295. The same applies if you parse it from a string: parseDouble(“2983792734924.2293″) always comes out as 2983792734924.2295. As a result the following are consistently true:

2983792734924.2293 == 2983792734924.2293
parseDouble("2983792734924.2293") == 2983792734924.2293
parseDouble("2983792734924.2293") == parseDouble("2983792734924.2293")

Introducing tolerance to the comparison can lead to subtle bugs. For example, if we were updating our model with user input we might have something like:

// In our model:
public void setAmount(double amount) {
  this.amount = amount;
  fireChangeEvent();
}
// In our UI code:
model.setAmount(parseDouble(widget.getText()));

So far so good, but what if we want to avoid firing the change event if the value hadn’t really changed? We might change the model code to:

public void setAmount(double amount) {
  if (abs(amount - this.amount) > 0.01) {
    this.amount = amount;
    fireChangeEvent();
  }
}

It follows the double folk lore, but also introduces a bug. Let’s say the user first enters the amount ’1′. Then they change the amount to ’1.001′. The second amount is within the tolerance so it’s not considered an actual change. The user is now looking at a text field that says 1.0001, but the model thinks the value is still 1. You can make the tolerance smaller but that just means the user has to enter more zeros to trigger the problem.

The right answer is to take advantage of the fact that inaccuracy with doubles is consistent and do the comparison using strict comparison:

public void setAmount(double amount) {
  if (about != this.amount) {
    this.amount = amount;
    fireChangeEvent();
  }
}

So when doesn’t this work? When the two values are calculated in different ways. So:

(1491896367462.1147 * 29.837927349242293) / 2.3982983923
= 18561116318075.207
1491896367462.1147 * (29.837927349242293 / 2.3982983923)
= 18561116318075.203

The only difference between the two equations is the order of evaluation – they should give the same answer, but because doubles have a degree of inaccuracy the intermediate result is rounded to something that will fit in a double, so the order of evaluation matters because it affects what rounding is done to the intermediate result. Repeating the exact same calculation will always give the exact same result, but if there’s any chance that the values were calculated in different ways, comparisons must be done with tolerance.

So the folk lore about needing to compare doubles with some tolerance is perfectly valid, good advice, but be aware that it introduces its own inaccuracy which may cause problems. You should consider what the impact of falsely deciding that the values are different vs falsely deciding that the values are the same; which side should you err on?

Sometimes you really do want a strict comparison, even between doubles.

Continuous Integration Requires Integrating Continuously

Adrian Sutton — Tue, 31 Jul 2012 03:31:19 +0000

Sarah Goff-Dupont has a post on the Atlassian blog about how Atlassian is using feature branches and still doing continuous integration:

In the not-so-recent past, continuous integration in the context of story branching was considered so impractical as to be outright incompatible. Who has time to manually configure a CI scheme for dozens of active branches that will only live a couple of days? Nobody –that’s who. And story branch-loving teams would frequently encounter *ahem* “surprises” when merging work onto the main code line –”master”, in Git-speak. But recent versions of Bamboo have essentially removed that overhead and allowed these two frenenmies to become genuinely harmonious partners in our development lifecycle.

Unfortunately, what Sarah describes is not continuous integration. At best, it’s regular integration but really it’s just automated testing. The key principal of continuous integration is that all developers are merging all changes, all the time. That way you see the messy combinatorial effects of concurrent changes and can fix them before you waste a lot of time going down incompatible paths. Anything that reduces the frequency of doing that integration is a step away from continuous integration, regardless of how often automated tests are run.

Let’s take a look at the basic process they’re using:

Next, we go into development and testing cycles on the story branch. The developer is making local commits on her workstation. [...] When she reaches a point where she wants to run tests against her work, the developer pushes those local commits up to the repository –still on the story branch, of course. Bamboo sees that there’s been a change, and starts building the corresponding Plan branch/es.

This is not continuous integration. The developer’s changes are being kept away from the main team and not integrated while they’re being developed, while they sitting just in the local git repository and while they’re in the story branch. Even though Bamboo runs tests against the code, there has been no integration done here. This is just automated testing, not continuous integration.

It seems that Atlassian have discovered this lack of integration and have developed ways of combating it:

To help find conflict before merging to master, the developer can choose to have Bamboo merge master into her story branch each time she pushes changes to the repo, then run through the CI Plan against the merged code. If the Plan passes, Bamboo will commit the merged code to her branch.

Finally we get some integration happening, but only one way. Since everyone’s off working on feature branches this is really only integrating with completed stories. All the work going on in parallel on other stories is still not integrated so we have no idea if they will work when combined.

Some teams are taking this workflow a step further and requiring developers to merge to an integration branch during development, before finally merging the completed story to master. This practice ferrets out integration issues even faster. Developers get to mash their in-progress stories together several times a day, rather than waiting until one has merged a completed story to master (which may happen only every 2-3 days) and the other has pulled code from master onto their story branch.

The key lesson here is that if we integrate more often, it’s easier to ferret out integration issues. It’s like continuous integration but with the extra fun of managing branches…

Of course, Atlassian aren’t alone in the use of feature branches – it’s become all the rage since DVCS supposedly made branching cheap and merging painless. Unfortunately while DVCS has eliminated many of the basic mechanics of merging, they haven’t done anything to reduce the risk of different developers push the codebase in incompatible directions at a structural level. The only way to avoid that is to integrate all changes as often as possible – that’s continuous integration.

The only way that feature branching and continuous integration can be at all compatible is if the build system automatically builds and tests every possible combination of branches, all the time – a massive waste of build system resources. Anything less leaves potential combinations untested until they are finally merged on to master and all the problems are discovered.

Or you know, we could just do continuous integration…

Golden Rule of Dependency Management

Adrian Sutton — Tue, 19 Jun 2012 01:54:48 +0000

There’s a huge amount of complaining and problem solving for various dependency management solutions (particularly maven, but ivy and friends certainly aren’t immune). Problems range from having optional dependencies unnecessarily included to having duplicate classes or class conflicts and the solutions tend to be complex and involve a lot of trade offs. All these problems stem from breaking the golden rule of dependency management:

Own your own repository
– Sutton’s golden rule of dependency management

The vast majority, if not all, problems with dependency management comes from having incorrect, conflicting or imprecise meta-data in the repository of dependencies. Maintaining a public repository of perfectly accurate, precise and flexible dependency metadata is next to impossible – there are just too many libraries and the interrelationships are too complex. Fortunately, even extremely large companies only use a tiny subset of these libraries. With the scope reduced it’s much easier to ensure the metadata is correct and consistent.

Any time you need to introduce a new dependency, very carefully review the metadata associated it and correct any errors or inconsistencies before importing it into the repository you administer and control.

You don’t let anyone commit to your source code repository, don’t let anyone commit to your dependency repository either.

Minimise Configuration

Adrian Sutton — Tue, 05 Jun 2012 05:27:47 +0000

Having configuration differences between development and production is largely unavoidable, but it’s important to keep the number of differences to a minimum to avoid unexpected bugs appearing in production that don’t occur in development mode. More than that though, it’s important to minimise configuration.

Configuration is Code

Often things are put in configuration files so that they are easy to change later, often without a full redeployment of the software, but that also implies it can be changed without going through the full testing that a release would usually be subject to. It’s rare for automated tests to cover more than one possible setting for these configuration options so changing them in production is equivalent to deploying completely untested code. What’s worse, it’s quick and easy to do with no audit trail of what happened and when.

These settings should be avoided altogether and the values hard coded in the built product and deployed along with it. That may be as simple as embedding the existing config file in the built software – the aim isn’t necessarily to mix configuration and code, only to remove the options to change the configuration outside of the normal development and deployment process.

For settings that are designed to be changed by the user as a normal part of using the software, make sure that the full range of values is tested, just like any other system input should be.

Soft Coding

The most extreme form of configuration is “soft coding” where the system becomes so configurable its essentially a poor-man’s programming language. The Daily WTF has an excellent explanation of soft coding and how even the best of intentions can lead to it.

Dealing with Production Problems

One of the biggest reason people make options configurable by administrators is so that they can quickly reconfigure things to resolve issues in production. Sadly, the worst time you could be making untested changes in production is when things are already behaving in unexpected ways. Instead, make the build, test and deployment process for your software so fast and so seamless that you can make the change in code, fully test it and get it out to production fast enough to resolve the issue. Doing so isn’t easy, but it delivers immense value beyond just resolving production issues. It allows you to more reliably and more rapidly deploy any code change into production. That means spending less time wrangling releases and more time improving the product.

Default to Development Settings

Adrian Sutton — Mon, 28 May 2012 05:55:00 +0000

Most systems have multiple configuration profiles – one for production, one for development and often other profiles for staging, testing etc. Minimising differences between these configurations is critical but there are inevitably some things that just have to be different. This then leaves the question, what should the default settings be?

There are three main options:

Default to production values
Default to development values
Refuse to start unless an environment has been explicitly chosen

My preference is to default to development values. Why?

Development values should be “safe” in terms of any external integrations. So a developer isn’t going to accidentally start sending real buy or sell instructions to your stock broker.

There are more developers than production systems. If you default to production systems, every developer needs to remember to switch to development mode whenever they setup a new checkout. Defaulting to development mode means it just works for the most common case.

Checking authentication credentials for external systems into your source control system is generally considered a bad security practice, so the default values are unlikely to actually work in production anyway.

The down side with defaulting to development is it’s possible to accidentally deploy to production with development values causing an outage. This can be pretty easily prevented with automated deployments or using tools like RPM where files can be marked as config and thus avoid overwriting them when doing updates.

Refusing to start is the worst of all worlds – every developer has to specify a configuration mode and you still risk production outages by not specifying a mode.

Juries and Complex Subjects

Adrian Sutton — Tue, 22 May 2012 23:35:09 +0000

There is a lot of discussion at the moment about the Oracle v Google trial today, mostly centring around how impossible it is for the jury to understand the very technical concepts involved in the case. As Daring Fireball puts it:

How could a randomly-selected jury possibly decide this? No knock intended against the jurors themselves — and it sounds like they’re doing their best to make an informed decision. But there’s a difference between a jury of your citizen peers and a jury of your technical peers.

Indeed, the situation is fairly ridiculous, however it’s not at all unique to technology. Juries are expected to evaluate extremely complex laws in most cases and contracts between parties can be just as confusing as any technical issue. Even when the law is relatively clear the elements of a case can be extremely complex – trying to understand exactly what forensic evidence does and doesn’t show for example.

The challenges for the jury in this trial are obvious to people with technical background, but the challenges for the jury in other types of cases is likely to be just as obvious to experts in those fields. Of course if we tried to create a jury of peers from the area of expertise required, we’d just create the extremely difficult challenge of deciding who is qualified and who isn’t.

It’s not a perfect system, but it is the best one we have.

Joho the Blog » Will tablets always make us non-social consumers?

Adrian Sutton — Mon, 21 May 2012 22:55:31 +0000

I thus think (= hope) that it’s a mistake to extrapolate from today’s crappy input systems on tablets to a future of tablet-based couch potatoes still watching Hollywood crap. We’re one innovation away from lowering the creativity hurdle on tablets. Maybe it’ll be a truly responsive keyboard. Or something that translates sub-vocalizations into text (because I’m too embarrassed to dictate into my table while in public places). Or, well, something. via Joho the Blog » Will tablets always make us non-social consumers?.

I suspect that the idea that input systems on tablets are crappy will rapidly become a tell-tale sign of age. Feature phones have “crappy” input systems and yet with learning, and some rather unfortunate adaptations of language, people quite happily chat away via SMS. The same process will happen with tablets – people will simply learn to type on on-screen keyboards fast enough that they don’t consider them crappy. It’s also important to remember that the vast majority of people can’t type particularly fast on a full size desktop keyboard so the bar for satisfaction is significantly lower than geeks would expect.

So Long EditLive! and Thanks For All The Fish (with rounded corners and drop shadows)

Adrian Sutton — Thu, 17 May 2012 10:20:24 +0000

Since the very early days of my blogging, I’ve integrated a copy of EditLive! to make the editing pleasant and more powerful. For many, many years there was simply no way I could bring myself to use anything else. Lately though, Apple have been making Java applets less and less appealing while browsers have been continuously improving their content editable suport and JavaScript editors have gotten better at working around the remaining quirks and smoothing off the rough edges on the editing experience. The final straw for me though was that the current early access release of EditLive! 8.0, which appears to include a lot of fixes for the latest OS X, breaks backwards compatibility with a number of APIs I use to integrate it into this blog.

So the time has finally come to switch to TinyMCE. I still have a custom plugin to fix some of the incredibly poor configuration choices made by default in WordPress (P tags are important remember!) but it’s significantly less custom code and complexity. There are also a bunch of new features that have built into WordPress which I’d never enabled in my EditLive! integration which are nice to have. The biggest win however is that I can now write blog posts on my iPad, so hopefully I can share a few more links and quick thoughts without having to drag out my laptop.

It is sad to retire EditLive! though. I worked to build it almost from scratch for nearly 10 years and for most if not all of that time it was the absolute cream of the crop for in browser editing with features and usability that were simply unmatched. I have some slight consolation that I’ve also been able to spend some time contributing to TinyMCE and I know the guys behind it are top notch so I’m happy to pass the baton of favourite editor over. I will miss the integrated image editor and its rounded corners and drop shadows…