Mostly just so that I can find the recordings more easily later, here’s the recordings of DevCon VI talks I gave in Bogotá.

With the merge fast approaching, all beacon chain nodes will need to run their own local execution layer client (e.g. Besu or Geth). For validators who have just been using Infura for eth1 data so far, that means some extra setup and potentially more hardware (mostly disk). To avoid this, some people are considering going the other direction and not running their own beacon node either, just running a validator client against a public beacon node API like Infura. While this is possible, it’s extremely likely to result in reduced rewards for a couple of reasons.

Yesterday I spun up two new nodes on Prater, and sent deposits for 5000 validators to run on each. Due to the restrictions on how many new validators can activate per epoch (currently 4 for Prater), the number of active validators are currently gradually increasing giving a really nice insight into the cost of adding new validators.

This is most clearly seen by graphing the number of active validators and CPU usage on a single chart.  Here the green is CPU usage and the yellow the number of active validators.

For many years I’ve been using the free tier of Google Workspace (previously Google Apps for Your Domain) to host my email. While I almost never use the actual web interface itself having google handle the IMAP hosting and spam filtering was an awful lot easier than hosting it myself. Of course Google are now ending their free tier and requiring all users to pay – fair enough, but the cost per user is just too much for me to justify it.

The merge is at the point where real interop testing can begin with testnets spun up involving multiple clients. This video shows a testnet running locally with four nodes:

• Teku & Geth
• Teku & Besu
• Lighthouse & Geth
• Lighthouse & Besu

Each node has 1/4 of the validators and the network transitioned through from the initial phase0, to Altair and then then merge after which the two chains merge into one.  Also shows sending transactions with MetaMask post-merge and how the user experience is largely unchained. Ethereum keeps on being Ethereum but now with PoS instead of PoW.

One of the most commonly overlooked steps when setting up a beacon node is to setup port forwarding on your router to ensure that other nodes can connect in to yours. This is often overlooked, or not done correctly, because there are so many different routers, all with different, fiddly interfaces that providing simple instructions is essentially impossible and because your node will generally work even without it by connecting outbound to other peers even though none can connect inbound.

I got the chance to talk with Pooja Ranjan and Alita Moore from the Ethereum Cat Herders as part of their PEEPanEIP series last week: PEEPanEIP #38: Altair in Teku with Adrian Sutton. It was a chance to talk through the actual implementation details required to support the Altair fork.

Mostly listing here for my own record, but if you haven’t seen it and you care at all about Ethereum hopefully it will be interesting.

Note: This post is deliberately high level. It doesn’t attempt to be an economic proof or go into the gory details of exactly how the difficulty adjustment works. The aim is just to see the high level effects of how the system pans out without getting lost in the nitty gritty details of what is ultimately a very complex system.

Pretty much every time the word “miner” is mentioned in an Ethereum discussion someone will claim that miners are paid too much and miners will respond saying they’re struggling to survive. Turns out, both can be simultaneously true and in fact it’s pretty much the expected case.

If you’re getting into staking ETH on the beacon chain, it’s important to know what you’re actually signing up for. There’s lots of articles about how easy it is and how awesome, intelligent and good looking stakers are, so here’s a few hard truths you should make sure you’re ok with…

Short Term Stuff

Most people know this stuff so I’m just going to skim over it.

No Withdrawals Yet

You can’t withdraw staked ETH or rewards yet. There isn’t a fixed timeline on when you will be able to.

Through the beacon chain testnets people have been gradually developing better ways to evaluate how well their validators are working. Jim McDonald made a big leap forward defining attestation effectiveness which focuses on the inclusion distance for attestations. That revealed a bunch of opportunities for clients to improve the attestation inclusion distance and attestation effectiveness scores improved quite dramatically until now scores are generally at or close to 100% pretty consistently.

But then beaconcha.in added a view of how much was earned for each attestation. Suddenly people were left wondering why some of their “rewards” were negative even though the attestation was included quickly. It became obvious that inclusion distance was only one part of attestation rewards.

Teku requires a password file for each individual keystore, with the same relative path as the keystore but a .txtextension. If all the keystores have the same password, you can copy a single password.txtfile for all the keystores with the command below:

for keyFile in keys/*; do cp password.txt "secrets/$(echo "$keyFile" | sed -e 's/keys\/\(.*\)\.json/\1/').txt"; done

Note that the keys directory is specified twice in that command so if the keys are in a different directory name, make sure to update both places.

Last week I put together a video explaining attestation inclusion and delays in ETH2. This is one of the keys to maximising your validator rewards and the health of the network.

We run a bunch of hosts on EC2 and while most just have the default DNS name, they all have a Name tag to identify their purpose. While there’s lots of automation setup via ansible, it is often still necessary to SSH directly into a particular box, especially while debugging issues.

I find it annoying to have to go log into the AWS console just to find the DNS name of the particular box I want so I’ve written a little script that searches for hosts based on the name tag and can then SSH into them.

It’s like something out of a Sherlock novel – the doors are all locked, windows barred and yet somehow the jewels have been stolen. Or in this case, every block created forms a single chain and yet somehow the validator client has wound up on the wrong fork. Fortunately Sherlock, or rather Cem Ozer, was on the case. This is the story behind those very annoying “Produced invalid attestation” errors that Teku would sometimes generate.

Stumbled across ArchUnit today which looks useful. In particular I think there’s power in being able to assert that certain packages really never depend on each other. Although gradle/maven modules would probably be a better level to assert at. It’s depressingly common for code bases to be split into separate modules with the intention that they be a clear separation of concerns only for a web of dependencies to be added because someone wanted to reuse some class and didn’t refactor to a common module.

Occasionally the term “weak subjectivity period” pops up in Eth2 discussions. It’s a weird concept that you can usually just watch fly by and not miss too much. But when you’re talking about how to sync an existing Eth2 chain it becomes quite important. Probably the best resource for it is Vitalik’s post: Proof of Stake: How I Learned to Love Weak Subjectivity I’ve struggled to get my head around it and why it matters so am writing up my current understanding. There is almost certainly at least one mistake in here somewhere…

One subtle detail in the way Ethereum works is that there is a difference between Ommers and Non-Canonical Blocks. It’s common for people to use the term Ommer for both of these and most of the time the difference doesn’t matter but sometimes it does.

So what is a non-canonical block? Non-canonical blocks are ones which a client imports but which don’t wind up on the canonical chain. Maybe they were on the canonical chain for a while and then a re-org switched to a different chain or maybe they were imported to a fork and spent their entire life languishing there. Either way, they don’t form part of the current consensus chain and have absolutely no effect on the world state. It’s like their transactions have never been executed and no one gets any form of miner reward for them. Non-canonical blocks must be entirely valid blocks that could form part of the canonical chain, but weren’t because we found a better chain.

I’ve been upgrading Moolah after a long period of neglect. One of the tasks is to update it to WebPack 4. Mostly this went smoothly by just creating a new project with vue-cli and bringing its generated build setup over to replace the old one. Then a bunch of tweaking.

One thing that did bite however, was tests using inject-loader started failing as soon as I added vuetify-loader to the project with:

Continuing our adventures in setting up a private beacon chain… Previously we got an Eth 1 chain with the deposit contract and successfully sent a deposit to register a new validator. So far though, we have no way of telling if it actually worked. What we need is an actual beacon chain client. Enter Artemis…

Step 3 – Add a Beacon Chain Client

We’ll be using Artemis, partly because I’m incredibly biased and partly because it happens to support the exact version of the deposit contract we’ve deployed (what a coincidence!).

I’ve started a little side-project to setup a “private beacon chain”. The aim is to better understand how the beacon chain works and start to discover some of the things still required to be built or fixed in clients before it can officially launch.

So what is a private beacon chain? It’s intended to be an entirely self-contained, runs-on-my-laptop instance of the beacon chain, run as a small-scale simulation of how the real beacon chain will be fired up.

If you’re switching from a CLA model to a DCO model you may want to add a Signed-off-by line to every existing commit so that you can run automated DCO checks over the entire repository. Doing this obviously assumes that your CLA sets things up so that you are actually in a position to provide a DCO sign-off.

Once you’re happy with all the legalities, the change is a single command using git filter-branch:

There’s a fun little gotcha introduced in Java 10 which causes trouble for anyone wanting to support earlier JVMs.  If you take the code:

import java.nio.ByteBuffer;

public class ProblemCode {
  public static void main(String[] args) {
    final ByteBuffer buffer = ByteBuffer.allocate(10);
    buffer.position(10);
    System.out.println("Yay it worked!");
  }
}

If you compile it on Java 8 it will run on any Java from 8 and above. If you compile it on Java 10 or above it will only work on Java 10 and above, even if you specify -target 8 -source 8.

I’ve had the opportunity to do some proof of concept development of the Ethereum state-rent proposal that Alexey Akhunov has been leading on the Pantheon code base. The proposal evolved as the work continued so the actual implementation is now a lot simpler than described in that PDF.

Note that the aim is to explore what it takes to implement the proposal, not to create production ready code.  The current work is all available on on my state-rent branch.

This week, the work I’ve been doing for the past 6 months, and that PegaSys has been working on for the past 18 months or so was released into the world. Specifically we’ve released Pantheon 0.8.1, our new MainNet compatible, Apache 2 licensed, Java-based Ethereum client. And it’s open source.

I’m pretty excited about it on a few fronts. Firstly I think it’s a pretty important thing for the Ethereum community. To be a healthy ecosystem, Ethereum needs to have diversity in its clients to avoid a bug in one client taking out or accidentally hard forking the entire network. Currently though, Geth and Parity dominate the Ethereum client landscape. Pantheon clearly won’t change that in the short term, but it is backed by significant engineering resources to help it keep up with the ever changing Ethereum landscape and be a dependable option.

There’s an exceptionally valuable set of ethereum reference tests that are run by most or all of the different Ethereum clients to ensure they actually implement the specifications in a compatible way. They’re one of the most valuable resources for anyone developing an Ethereum client.

The Aleth project maintains the official test client called testeth but it’s a little cryptic to work out how to actually run things with it and then use that to debug failures happening in the client you’re actually developing. So this is what I’ve found useful:

This is the second article on things I found particularly interesting in the Ethereum yellow paper.  The first is “What’s on the Blockchain?” and the same disclaimers apply: I’m no expert and you should go verify any claims I’m making before depending on them. Comments and corrections are most welcome either via email or @ajsutton on twitter.

One of the little details in the way Ethereum works is the idea of a “nonce” attached to each account and transaction. It’s a small but important detail.

Hapi 17 was released a while back with quite a few nice improvement – mostly centered around using promises instead of callbacks and thus fitting in with async/await well.  However, it’s a very significant change right across it’s APIs so it took quite a while for the various plugins that Moolah uses to be updated.

Eventually though they were all ready and then the actual upgrade of Moolah code was pretty straight forward.  Moolah had been using async handlers via a plugin all along so there were minor changes to how they were specified but very few changes to the control flow they then used.

I dropped out of the habit of writing about Moolah but it’s been ticking along doing what I need it to for a fair while with a few additions here and there. Recently I spent a chunk of time adding the ability to ‘earmark’ money for a particular use.

My main motivation is that we’re planning a family trip to Germany at the end of the year and I want to be able to set aside money through the year as we save for it. Various things will need to be paid upfront and those expense need to come out of the earmarked funds. I could track that with a new account for the savings, but since I’m not setting up a new bank account the balances wouldn’t match up and I’d spend a lot of time transferring money in and out of the holiday account.

I ran into an issue with Vuetify recently where the 1.0 betas were constantly utilising a CPU core when run in Safari. It turned out, I wasn’t alone and it had been reported as bug 2773.

There wasn’t a lot to go on – the problem only happened in Safari and the timeline showed it was constantly invalidating and recalculating the layout, apparently without any JavaScript executing.

First step was to create a simple reproduction case, which turned out to be a template with just:

The internet is full of forum posts with various questions, tips and suggestions for IPv6 on EdgeRouters as people struggle to get it working. The challenge is that the fix you actually need depends on which version of EdgeOS you’re running (ubnt are continuing to flesh out the IPv6 configuration support) and the setup of your ISP.

So here’s the magic steps I needed for an ERX running EdgeOS 1.9.7+hotfix.3 with SkyMesh NBN over HFC (cable).

As Vue gradually spreads through our JavaScript, it’s reached the point where we need to support internationalisation. We already have a system for localising numbers and dates that we can reuse easily enough with Vue filters. We also have an existing system for translations, but it’s tied into the mustache templates we’ve used to date so we’ll need to adapt or change it somehow to work with Vue.

In mustache we simply wrap any strings to be translated with {{#i18n}}…{{/i18n}} markers and a webpack loader takes care of providing a translation function as the i18n property of the state for the compiled template.

moment is one of those javascript libraries that just breaks my heart. It does so much right – nearly every date function you’d want right at your finger tips, no major gotchas causing bugs, clean API and good docs. I’ve used moment in some pretty hard code, very date/time orientated apps and been very happy.

But it breaks my heart because when you pull it in via webpack, it winds up being a giant monolithic blob. I only need a couple of date functions but with moment I wind up getting every possible date function you can imagine plus translations into a huge range of languages I don’t need. If I use moment, about 80% of Moolah’s client javascript is moment.

Ultimately the point of Moolah isn’t just to record a bunch of transactions, it’s to provide some insight into how your finances are going. The key question being how much more or less are we spending than we earn? I use a few simple bits of analysis to answer that question in Moolah, the first of which is income vs expenses over the past x months.

The simple way to do that is to show totals based on calendar month (month to date for the current month), but since my salary is paid monthly that doesn’t provide a very useful view of the current month since no income has arrived yet.

Moolah just reached it’s first key milestone – it’s reached “data parity” with the old version. Basically it has enough functionality that it’s viable to import the data from the old version and get the same balances and totals. Using real data that’s been generated over the last 5 years has revealed a few important things.

The good:

• The balances add up so things really are working
• It really does load way faster
• The old version allowed some data inconsistencies that the new version will prevent

The bad:

Moolah has two separate Vuex modules, transactions and accounts. That’s a fairly reasonably logical separation, but in the new Moolah, it’s not a complete separation. Moolah only ever loads a subset of transactions – typically the most recent transactions from the currently selected account.  As a result, accounts have their own current balance property because they can’t always calculate it off of their transactions.

That means that sometimes the transaction Vuex module needs to make changes to state owned by the account module which is a rather unpleasant breach of module separation. While we ensured transaction balances remained consistent inside each mutation, we can’t do that for account balances because Vuex (presumably deliberately) makes it impossible for a mutation to access or change state outside of the module itself. Instead, I’ve used a simple Vuex plugin that watches the balance on the most recent transaction and notifies the account module when it changes:

Previously on the Moolah Diaries I had plans to recalculate the current balance at each transaction as part of any Vuex action that changed a transaction. Tracking balances is a good example of an invariant – at the completion of any atomic change, the balance for a transaction should be the initial balance plus the sum of all transaction amounts prior to the current transaction.

The other invariant around transactions is that they should be sorted in reverse chronological order. To make the order completely deterministic, transactions with the same dates are then sorted by ID. This isn’t strictly necessary, but it avoids having transactions jump around unexpectedly.

Experience at LMAX has very clearly demonstrated the benefits of good test isolation, so one of the first things I added to Moolah was multi-tenant support. Each account is a completely isolated little world which is perfect for testing.

Given that I’ve completely punted on storing passwords by delegating authentication to Google (and potentially other places in the future since it’s using bell to handle third party authentication), there’s actually no user creation step at all which makes it even easier. As a result it’s not just acceptance tests that are benefiting from the isolation but also the database tests which no longer need to delete everything in the database to give themselves a clear workspace.

Most of the work I’ve done on Moolah so far has been on the server side – primarily fairly boring setup work and understanding the best way to use any number of libraries that were new to me sensibly. The most interesting work however has been on the front-end. I’ve been using Vue.js and Vuetify for the UI after Vue’s success in the day job. The Moolah UI has much more data-interdependence between components than what we’ve needed at work though so I’ve introduced Vuex to manage the state in a centralised and more managed way.

I’ve previously written about how Angular 2 didn’t work out for us, our second attempt was to use Vue.js which has much far more successful. The biggest difference with Angular is that it’s a much smaller library. It has far less functionality included in the box but a wide range of plugins to flesh out functionality as needed. That avoided the two big issues we had with Angular 2:

• Build times were faster because there were way fewer dependencies to install, manage and bundle together. Not using Typescript may have also helped but I never had a clear indication of how much that affected build time.
• It integrated much more easily with our existing system because it didn’t try to do everything. We just kept using the bits of our system that worked well.

We’ve been so happy with how Vue.js fits in for us, we’re now in the process of replacing the things we had built in Angular 2 with Vue.js versions.

It’s always important to have some basic principles in mind before you start writing code for a project. That way you have something to guide your technology choices, development practices and general approach to the code. They’re not immovable – many of them won’t survive the first encounter with actual code – but identifying your principles helps to make a code base much more coherent.

Naturally for the Moolah rebuild I just jumped in and started writing code without much thought, let alone guiding principles but a few have emerged either as lessons from the first version, personal philosophy or early experiences.

When I moved back to Australia about 5 years ago, I suddenly had a much more complex task of tracking our money. I was consulting, so had to set aside money to pay tax at the end of the year, we were buying our first house and having just moved countries needed to buy pretty much everything so budgeting was critical. I found a reasonable looking money tracking application and started recording all our transactions there and using it to manage our money.

At LMAX we value simplicity very highly and since most of the UI we need for the exchange is fairly straight forward settings and reports we’ve historically kept away from JavaScript frameworks. Instead we’ve stuck with jQuery and bootstrap along with some very simple common utilities and patterns we’ve built ourselves. Mostly this has worked very well for us.

Sometimes though we have more complex UIs where things dynamically change or state is more complex. In those cases things start to breakdown and get very messy. The simplicity of our libraries winds up causing complexity in our code instead of avoiding it. We needed something better.

With Promise/A+ spreading through the world of JavaScript at a rapid pace, there’s one little detail that makes them very hard to unit test: any chained actions (via .then()) are only called when the execution stack contains only platform code. That’s a good thing in the real world but it makes unit testing much more complex because resolving a promise isn’t enough – you also need to empty the execution stack.

I’ve been gradually tidying up the build process for UI stuff at LMAX. We had been using a mix of requires and browserify – both pretty sub-optimally configured. Obviously when you have too many ways of doing things the answer is to introduce another new way so I’ve converted everything over to webpack.

Webpack is most often used as part of a node or gulp/grunt build process but our overall build process is controlled by Buck so I’ve had to work it into that setup. I was also keen to minimise the amount of code that had to be changed to support the new build process.

Buck is a weird but very fast build tool that happens to be rather opaque about where it actually puts the things you build with it. They wind up somewhere under the buck-out folder but there’s no guarantee where and everything under there is considered buck’s private little scratch pad.

So how do you get build results out so they can be used? For things with built-in support like Java libraries you can use ‘buck publish’ to push them out to a repo but that doesn’t work for things you’ve built with a custom genrule. In those cases you could use an additional genrule build target to actually publish but it would only run when one of it’s dependencies have changed. Sometimes that’s an excellent feature but it’s not always what you want.

Symlinks have been causing me grief lately.  Our build tool, buck, loves creating symlinks but publishes corrupt cache artefacts for any build rule that includes a symlink amongst it’s output.

We also wind up calling out to npm to manage JavaScript dependencies and it has an annoying (for us) habit of resolving symlinks when processing files and then failing to find required libraries because the node_modules folder was back where the symlink was, not with the original file. Mostly this problem is caused by buck creating so many symlinks.

I have three Benq GW2765 monitors which periodically report “No Signal Detected” for DisplayPort even when the computer it’s attached to recognises the monitor is present (displaying it in the monitors/displays list etc). Changing  the DisplayPort cable or plugging it into a different computer doesn’t help (I tried with both Mac OS X and Linux/Fedora machines), but HDMI and D-Sub connections work perfectly (but can’t support the full screen resolution). I can even disconnect a cable from a working monitor, plug it into a non-working monitor and it will continue to complain about no signal, but plug the cable back into the working monitor and it carries on working fine.

After any major Fedora upgrade my system loses the proprietary nvidia drivers that make X actually work (I’ve never successfully gotten the nouveau drivers to handle my card and multi-monitor setup) so the system reboots and simply presents an “oops, something went wrong” screen.

The issue seems to be that the nvidia driver doesn’t recompile for the new kernel, despite the fact that I’m using akmod packages which should in theory automatically recompile for new kernels.

One reason that automated UI tests can be unreliable is that they tend to be sensitive to what else is on screen at the time and even things like the current screen size. Developers running the tests locally also find it annoying to have windows opening and closing on their machine while the test runs and are unable to do anything else because their clicking might interfere with the test.

When an automated UI test fails, it can be hard to tell exactly what went wrong just from the failure message. The failure message typically just says that some element the test was looking for wasn’t found, but it doesn’t tell you what was there.  Was there an error message displayed instead? Was the operation still executing? Did something completely unexpected happen instead?

To answer those questions our DSL automatically captures a screenshot when any UI operation fails and we include a link to it in the failure message. That way when someone reviews the test result they can see exactly what was on screen which typically makes it straight forward to identify what went wrong and fix it.

So I can find this again easily in the future, to avoid introducing merge commits git pull –rebase is a great idea and you can make it the default behaviour with:

git config --global pull.rebase true

Found on Coderwall which has some further details.

Once an application goes live, it is absolutely essential that any future changes are able to be work with the existing data in production, typically by migrating it as changes are required. That existing data and the migrations applied to it are often the riskiest and least tested functions in the system. Mistakes in a migration will at best cause a multi-hour outage while backups are restored and more likely will subtly corrupt data, producing incorrect results that may go unnoticed for long periods making it impossible to roll back. At LMAX we reduce that risk by running compatibility tests.

Just like production code, you should assume things are going to go wrong in your tests and when it does you want good logging to help track down what happened and why. So just like production code, you should use a logging framework within your DSL, use meaningful log levels and think about what info you’d need in the logs if something went wrong (and what you don’t). There’s also a few things we’ve found very useful specifically in our test logging.

With Selenium 1, JavaScript alert and confirmation dialogs were intercepted by the Selenium JavaScript library so they never appeared on-screen and were accessed using selenium.isAlertPresent(), selenium.isConfirmationPresent(), selenium.chooseOkOnNextConfirmation() and similar methods.

With Selenium 2 aka WebDriver, the dialogs do appear on screen and you access them with webDriver.switchTo().alert() which returns an Alert instance for further interactions.

However, when you use WebDriverBackedSelenium to mix Selenium 1 and WebDriver APIs – for example, during migrations from one to the other – the alerts don’t appear on screen and webDriver.switchTo().alert() always throws a NoAlertPresentException. This makes migration problematic because selenium.chooseOkOnNextConfirmation() doesn’t have any effect if the next confirmation is triggered by an action performed directly through Web Driver.

Today LMAX Exchange has released ElementSpecification, a very small library we built to make working with selectors in selenium/WebDriver tests easier. It has three main aims:

• Make it easier to understand selectors by using a very English-like syntax
• Avoid common pitfalls when writing selectors that lead to either brittle or intermittent tests
• Strongly discourage writing overly complicated selectors.

Essentially, we use ElementSpecification anywhere that we would have written CSS or XPath selectors by hand. ElementSpecification will automatically select the most appropriate format to use – choosing between a simple ID selector, CSS or XPath.

Probably the best thing I’ve discovered with my recent playing is Travis CI. I’ve known about it for quite some time, even played with it for simple projects but never with anything with any real complexity. Given this project uses rails which wants a database for pretty much every test and that database has to be postgres because I’m using it’s jsonb support, plus capybara and phantomjs for good measure, this certainly isn’t a simple project to test.

Given our DSL makes heavy use of aliases, we often have to provide a way to include the real name or ID as part of some string. For example, an audit record for a new account might be:

Created account 127322 with username someUser123.

But in our acceptance test we’d create the user with:

registrationAPI.createUser("someUser");

someUser is just an alias, the DSL creates a unique username to use and the system assigns a unique account ID that the DSL keeps track of for us. So to write a test that the new account audit record is written correctly, we need a way to build the expected audit string.

I have a dual monitor setup with Linux where one display is connected via DisplayPort and the other via DVI, both from an Nvidia graphics card. When the screen is locked both displays go black, the DVI monitor says it’s entering power save mode and the DisplayPort monitor says “No input” at which point both monitors turn back on displaying the unlock screen.

Playing with xset dpms force standby|suspend|off gave a variety of effects, sometimes the DisplayPort stayed off but the DVI turned back on, sometimes the DisplayPort went black but didn’t turn off etc.

At LMAX, all our acceptance tests are written using a high level DSL. This gives us two key advantages:

• Tests can focus on what they’re testing with the details and mechanics hidden behind the DSL
• When things change we can update the DSL implementation to match, avoiding the need to change each test that happens to touch on the area.

The DSL that LMAX uses is probably not what most people think of when hearing the term DSL – it doesn’t attempt to read like plain English, just simplifies things down significantly. We’ve actually open sourced the simple little library that is the entrance-way to what we think of as the DSL – creatively named simple-dsl. It’s essentially the glue between what we write in an acceptance test and the plain-java implementation behind that.

The Google Testing Blog has an article “Just Say No to More End-to-End Tests” which winds up being a rather depressing evaluation of the testing capabilities, culture and infrastructure at Google. For example:

Let’s assume the team already has some fantastic test infrastructure in place. Every night:

1. The latest version of the service is built. 
2. This version is then deployed to the team's testing environment. 
3. All end-to-end tests then run against this testing environment. 
4. An email report summarizing the test results is sent to the team.

If your idea of fantastic test infrastructure starts with the words “every night” and ends with an email being sent you’re doomed. Bryan Pendleton does a good job of analysing and correcting the details so I won’t recover that ground. Instead, let me provide a view of what reasonable test infrastructure looks like.

It’s not uncommon to want to find a particular line in a file and print just a part of it.  For example, if we wanted to find the total memory on a machine we could use:

grep '^MemTotal:' /proc/meminfo | 
  sed -e 's/^MemTotal:\s*\([0-9]*\).*/\1/'

Which does the job but duplicates a bunch of the matching regex. If you’re using a reasonably recent gnu grep, you can use it’s support for perl regex syntax and the \K operator:

Avdi Grimm in Patterns are for People:

Patterns aren’t tools for programming computers; they are tools for programming people. As such, to say that “patterns are a language smell” makes very little sense. Patterns are a tool for augmenting language: our language, the language we use to talk to each other about a problem and its solutions; the language we use to daydream new machines in our minds before committing them to code. Patterns aren’t a language smell; rather, patterns are by definition, [optional] language features. There’s so much we can learn if only we stop making everything into a battle between “new” tools or approaches and “old” ones. Patterns are relevant and useful in all forms of programming and we shouldn’t discard them just because they’re not “cool” anymore. Sadly we seem to throw out nearly all our experience and learning each time a new approach comes along rather than learning from and taking advantage of both.

Jeffrey Ventrella in The Case for Slow Programming:

Venture-backed software development here in the San Francisco Bay area is on a fever-pitch fast-track. Money dynamics puts unnatural demands on a process that would be best left to the natural circadian rhythms of design evolution. Fast is not always better. In fact, slower sometimes actually means faster – when all is said and done.

Jeffrey’s right in suggesting that we sometimes need to go slower to go faster, unfortunately he makes the mistake of believing that committing and releasing in very short cycles is the cause of these problems:

Lawrence Kesteloot has an excellent post Java for Everything.

About a year ago, though, I started to form a strange idea: That Java is the right language for all jobs. (I pause here while you vomit in your mouth.) This rests on the argument that what you perceive to be true does not match reality, and that’s never a popular approach, but let me explain anyway.

There are two key realisations that are vital to understanding why this argument has merit and the first one is right there in the introduction: what you perceive to be true does not match reality. As Lawrence notes:

When generating a JavaScript file dynamically it’s not uncommon to have to embed an arbitrary string into the resulting code so it can be operated on. For example:

function createCode(inputValue) {
  return "function getValue() { return '" + inputValue + "'; }"
}

This simplistic version works great for simple strings:

createCode("Hello world!");
// Gives: function getValue() { return 'Hello world!'; }

But breaks as soon as inputValue contains a special character, e.g.

When you have automated testing setup with a big display board to provide clear feedback, it’s tempting to try and use it for things it wasn’t intended for. One example of that is as a kind of reminder system – you identify a problem somewhere that can’t be addressed immediately but needs to be handled at some point in the future. It’s tempting to add a test that begins failing after a certain date (or the inverse, whitelisting errors until a certain date). Now the build is green and there’s no risk of you forgetting to address the problem because it will fail again in the future. Perfect right?

Every web based product is adding “responsive design” to their feature lists. Unfortunately in many cases that responsive design is actually making their product much harder to use on a variety of screen sizes instead of easier.

The problem is that common CSS libraries and grid systems, including the extremely popular bootstrap, imply that a design can be made responsive using fixed cut-off points and the design just automatically adjusts. In reality making a design responsive requires tailoring the cut off points so that the design adjusts at the points where it stops working well.

I wanted to create a CSV file showing the number of JUnit tests in our codebase vs the number of Spock tests over time. I can count the number of tests, along with the revision pretty easily with:

git svn find-rev HEAD
find src/test -name '*Test.java' | wc -l
find src/test -name '*Spec.groovy' | wc -l

But that outputs the results on three separate lines and I really want them on one line with some separator (comma, space or tab). The simplest way to achieve that is:

Marco Cecconi – I don’t love the single responsibility principle:

The principle is arbitrary in itself. What makes one and only Reason To Change™ always, unequivocally better than two Reasons To Change™? The number one sounds great, but I’m a strong advocate of simple things and sometimes a class with more reasons to change is the simplest thing. I agree that we mustn’t do mega-classes that try to do too many different things, but why should a class have one single Reason To Change™? I flatly disagree with the statement. It’s not supported by any fact. I could say “three” reasons and be just as arbitrary. Interesting critique of the single responsibility pattern. The quote above is perhaps one of the weaker arguments against the single responsibility pattern but it struck a nerve with me because all too often we assume that because a rule sounds good it has some validity. In reality, picking one responsibility for a class is quite arbitrary and, as the article argues, deciding what a responsibility is is even more arbitrary.

Previously in the Testing@LMAX series I’ve mentioned the way we’ve provided isolation between tests, allowing us to run them in parallel. That isolation extends all the way up to supporting a multi-tenancy module called venues which allows us to essentially run multiple, functionally separate exchanges on a single deployment of the LMAX Exchange.

We use the isolation of venues to reduce the amount of hardware we need to run our three separate liquidity pools (LMAX Professional, LMAX Institutional and LMAX Interbank), but that’s not all. We actually use the isolation venues provide to extend our testing all the way into production.

One of the most common reasons people avoid writing end-to-end acceptance tests is how difficult it is to make them run fast. Primary amongst this is the time required to start up the entire service and shut it down again. At LMAX with the full exchange consisting of a large number of different services, multiple databases and other components start up is far too slow to be done for each test so our acceptance tests are designed to run against the same server instance and not interfere with each other.

LMAX has invested quite a lot of time into building a suite of automated tests to verify the behaviour of our exchange. While the majority of those tests are unit or integration tests that run extremely fast, in order for us to have confidence that the whole package fits together in the right way we have a lot of end-to-end acceptance tests as well.

These tests deliver a huge amount of confidence and are thus highly valuable to us, but they come at a significant cost because end-to-end tests are relatively time consuming to run. To minimise the feedback cycle we want to run these tests in parallel as much as possible.

The key to making continuous integration work well is to ensure that the build stays green – ensuring that the team always knows that if something doesn’t work it was their change that broke it. However, in any environment with a build pipeline beyond a simple commit build, for example integration, acceptance or deployment tests, sometimes things will break.

When that happens, there is always a natural tendency to commit an additional change that will fix it. That’s almost always a mistake.

Here’s a fun one, we have some code generated using javassist that looks a bit like:

public void doStuff(byte actionId) {<br />  switch (actionId) {<br />    case 0: doOneThing(); break;<br />    case 1: doSomethingElse(); break;<br />    default: <br />      throw new IllegalArgumentException("Unknown action: " + actionId);<br />  }<br />}

This works perfectly fine on Java 6 and Java 7 but fails on Java 8.  It turns out the problematic part is "Unknown action: " + actionId. When run on Java 8 that throws “java.io.IOException: invalid constant type: 15” in javassist.bytecode.ConstPool.readOne.

Esoteric Curio – 折伏 – as it relates to coding

Our job as a team is to reinforce each other and make each other more productive. While I attempt to conquer some of the engineering tasks as well as legal tasks, and human resource tasks I face, I need concentration and it world certainly help to not be interrupted. I used to think I needed those times of required solitute pretty much all the time. It turns out that I have added far more productivity by enabling my teams than I have personally lost due to interruptions, even when it was inconvenient and frustrating. So much so that Ive learned to cherish those interruptions in the hope that, on serendipitous occasion, they turn into a swift, sprititual kick to the head. After all, sometimes all you need for a fresh persective is to turn out the light in the next room; and, of course, to not have avoided the circumstance that brought you to do it. Too often we confuse personal productivity, or even just the impression of it, for team productivity. So often though having individuals slow down is the key to making the team as a whole speed up.  See also Go Faster By Not Working and The Myth of Measurable Productivity.

Geekdom is not a club; it’s a destination, open to anyone who wants to put in the time and effort to travel there…

…dive headfirst into the things that interest you. Soak up every experience. Lose yourself in the pursuit of knowledge. When you finally come up for air, you’ll find that the long road to geekdom no longer stretches out before you. No one can deny you entry. You’re already home.

As a follow up to the earlier link regarding the performance of animations in CSS vs JavaScript, Christian Heilmann – Myth busting mythbusted:

Jack is doing a great job arguing his point that CSS animations are not always better than JavaScript animations. The issue is that all this does is debunking a blanket statement that was flawed from the very beginning and distilled down to a sound bite. An argument like “CSS animations are better than JavaScript animations for performance” is not a technical argument. It is damage control. You need to know a lot to make a JavaScript animation perform well, and you can do a lot of damage. If you use a CSS animation the only source of error is the browser. Thus, you prevent a lot of people writing even more badly optimised code for the web. Some good points that provide balance and perspective to the way web standards evolve and how to approach web development.

Myth Busting: CSS Animations vs. JavaScript:

As someone who’s fascinated (bordering on obsessed, actually) with animation and performance, I eagerly jumped on the CSS bandwagon. I didn’t get far, though, before I started uncovering a bunch of major problems that nobody was talking about. I was shocked.

This article is meant to raise awareness about some of the more significant shortcomings of CSS-based animation so that you can avoid the headaches I encountered, and make a more informed decision about when to use JS and when to use CSS for animation.

Chris Poole – Small things add up:

By migrating to the new domain, end users now save roughly 100 KB upstream per page load, which at 500 million pageviews per month adds up to 46 terabytes per month in savings for our users. I find this unreal. Just by moving images to a domain that doesn’t have cookies. Impressive, especially given that users upload data significantly slower than they download it and HTTP 1.1 headers are sent uncompressed. Important to note however:

There’s lots of projects these days moving from one language to the next. Whether that’s a good idea or not varies but let’s accept that it was the right choice and the world is a better place for it. One thing really bugs me: inevitably justifications of how successful that move has been includes a claim that the number of lines of code were so significantly reduced.

We rewrote 1.5 million lines of Java in just 6,000 lines of haskell!

Password managers are like exercise, some people are really into and and you know its good for you but it always seems like too much effort.

A few times I’ve actually gotten around to trying out password managers and I went pretty far down the rabbit hole with LastPass at one point, but its never stuck with me before. LastPass was ok, but just seemed clunky and instead of making life easier always seemed to make things take just a little bit more effort.

If you ever copy or rsync your home directory to another drive on Linux (especially Fedora 19), you may find that Google Chrome starts to crash whenever you type punctuation, including space characters, into HTML form fields like input or textarea.

You may also find that ‘useradd’ complains that it can’t create home directory.

It turns out that this is because of SELinux having incorrectly file labels for the files in the new home directory. Simply run:

Couple of very interesting articles today on architecture of CSS. Firstly Thierry Koblentz questions the benefits of separation of concerns – advocating a coding style where most style rules map a single classname to a single CSS property.

In complete contrast Ben Darlow argues the importance of semantic meaning, pushing back against techniques like OOCSS and BEM which move away from semantic class names in the name of modularity – generally resulting in long strings of class names on elements.

If you have:

• A javascript project using require.js
• Which is optimised using r.js
• Includes Rickshaw or any other library that uses the prototype class system

Then you are very likely here because you’re trying to enable JavaScript optimization (minification) and finding that suddenly stuff breaks.

The problem is that the prototype class system allows overriding methods to call the original method by adding a $super as the first parameter to the function. From the prototype docs:

One of the biggest challenges with selecting a web host is that it’s very difficult to determine the quality of a provider without actually setting everything up and seeing how it goes for a while. Generally that means that you either wind up avoiding the very low cost providers out of fear they won’t be reliable and possibly paying too much, or spending a lot of time and effort setting up with a provider only to then discover they’re unreliable or the performance isn’t as good as you expected.

I’ve recently embarked on a fairly complex new application, a large part of which is a webapp written in JavaScript. The application uses require.js to handle modules and loading of dependencies and we want to be able to unit test our JavaScript.

In order to test specific pieces of the application, we want to be able to inject stubs or mocks into the module being tested. For example, if we had a module:

According to the ant documentation:

It is possible to include the same file more than once by using different prefixes, it is not possible to import the same file more than once.

Unfortunately, it turns out this isn’t _quite _true. In most cases, ant will indeed ignore a request to import a file for the second time, so:

<import>
   <file name="utils.xml"/>
   <file name="utils.xml"/>
</import>

will only import utils.xml once. This is true even if there’s a chain of files being imported (so A imports B and C, then B imports C as well, C will only be imported once).

Most people believe that ant will only ever execute a target once per execution of ant. So if we have targets A, B, C and D, where A depends on B and C and C and D both depend on D.

<target name="A" dependencies="B, C"/>
<target name="B" dependencies="D"/>
<target name="C" dependencies="D"/>
<target name="D"/>

When we run ‘ant A’, we expect each task to execute once, D first, then B  and C, ending with A. Since there is no dependency between B and C, they may execute in any order, so we might also get D, C, B, A. All pretty straight forward.

This information is all covered in much more detail elsewhere on the web but for my own future reference, here’s a primer on doctypes, compatibility modes, charsets and fonts which was required to explain why certain Chinese characters weren’t showing up in IE8. Of course the best answer is that you need to have the East Asian Font pack installed and then it just works (usually) but this tends to be useful background and saves “server side” folks from a number of gotchas.

Of all the data types, double is probably one of the most misunderstood. A huge amount of folk lore has been built up around it to help protect developers from falling into its many pitfalls.  Lately I’ve done a lot of work replacing usage of BigDecimal with double and learnt a lot about where those pitfalls are and how the folk lore can be misleading.

The great challenge with double is that it has a degree of inaccuracy because of the way the number is actually stored.  For example, the number 2983792734924.2293 actually winds up being stored as 2983792734924.2295, a tiny difference but with far reaching consequences. All of the folk lore around doubles is designed to deal with this potential for inaccuracy.

Sarah Goff-Dupont has a post on the Atlassian blog about how Atlassian is using feature branches and still doing continuous integration:

In the not-so-recent past, continuous integration in the context of story branching was considered so impractical as to be outright incompatible.  Who has time to manually configure a CI scheme for dozens of active branches that will only live a couple of days?  Nobody –that’s who.  And story branch-loving teams would frequently encounter *ahem* “surprises” when merging work onto the main code line –”master”, in Git-speak.  But recent versions of Bamboo have essentially removed that overhead and allowed these two frenenmies to become genuinely harmonious partners in our development lifecycle.

There’s a huge amount of complaining and problem solving for various dependency management solutions (particularly maven, but ivy and friends certainly aren’t immune). Problems range from having optional dependencies unnecessarily included  to having duplicate classes or class conflicts and the solutions tend to be complex and involve a lot of trade offs. All these problems stem from breaking the golden rule of dependency management:

Own your own repository

— Sutton's golden rule of dependency management

Having configuration differences between development and production is largely unavoidable, but it’s important to keep the number of differences to a minimum to avoid unexpected bugs appearing in production that don’t occur in development mode. More than that though, it’s important to minimise configuration.

Configuration is Code

Often things are put in configuration files so that they are easy to change later, often without a full redeployment of the software, but that also implies it can be changed without going through the full testing that a release would usually be subject to. It’s rare for automated tests to cover more than one possible setting for these configuration options so changing them in production is equivalent to deploying completely untested code. What’s worse, it’s quick and easy to do with no audit trail of what happened and when.

Most systems have multiple configuration profiles – one for production, one for development and often other profiles for staging, testing etc.  Minimising differences between these configurations is critical but there are inevitably some things that just have to be different. This then leaves the question, what should the default settings be?

There are three main options:

• Default to production values
• Default to development values
• Refuse to start unless an environment has been explicitly chosen

My preference is to default to development values. Why?

There is a lot of discussion at the moment about the Oracle v Google trial today, mostly centring around how impossible it is for the jury to understand the very technical concepts involved in the case.  As Daring Fireball puts it:

How could a randomly-selected jury possibly decide this? No knock intended against the jurors themselves — and it sounds like they’re doing their best to make an informed decision. But there’s a difference between a jury of your citizen peers and a jury of your technical peers.

I thus think (= hope) that it’s a mistake to extrapolate from today’s crappy input systems on tablets to a future of tablet-based couch potatoes still watching Hollywood crap. We’re one innovation away from lowering the creativity hurdle on tablets. Maybe it’ll be a truly responsive keyboard. Or something that translates sub-vocalizations into text (because I’m too embarrassed to dictate into my table while in public places). Or, well, something. via Joho the Blog » Will tablets always make us non-social consumers?. I suspect that the idea that input systems on tablets are crappy will rapidly become a tell-tale sign of age. Feature phones have “crappy” input systems and yet with learning, and some rather unfortunate adaptations of language, people quite happily chat away via SMS. The same process will happen with tablets – people will simply learn to type on on-screen keyboards fast enough that they don’t consider them crappy. It’s also important to remember that the vast majority of people can’t type particularly fast on a full size desktop keyboard so the bar for satisfaction is significantly lower than geeks would expect.

It’s widely recognised that adding people to a team often doesn’t result in increases in velocity and can even reduce the velocity of the team due to the extra communication overhead. Instead, teams look to increasing levels of automation and tools to improve their productivity without adding extra people.

Where many teams struggle however is in finding the right balance of building out the product versus “sharpening the saw” by improving their tools and automating processes. With the same people attempting to do both jobs that contention is unavoidable.

We all know by now that continuous integration is part of good software development – check in regularly and have a suite of automated tests run to confirm that everything is working as expected. If a test fails, jump on it quickly and get the build back to green. Simple right?

But what happens when something goes wrong in a way that can’t be fixed quickly? For example, the build server has a hardware fault or runs out of disk space. You can’t just rollback the faulty change, its going to take time to get the build back to green. As your CI system grows, it may take time just to understand what went wrong. If your team is small they may all be taken up fixing the problem, but if the team is larger a pair focusses on fixing the build as quickly as possible and the other developers carry on working. Now you have two problems.

SVN tree conflicts can be insanely annoying – especially when they occur because you’ve deleted something locally and then an incoming change also deletes it. They can also be hard to resolve from most IDEs.

The trick is to resolve them from the command line.  Edit/create/delete the files to get things into the state they should be and then from the command line run:

svn resolve --accept working <path>

No more tree conflict.

I keep forgetting this so for the record, you can use SSH to round trip data out to a remote server and back to your own box.  This is most useful for adding latency and bandwidth limitations to a connection when doing website testing.  The trick is to use both local and remote port forwards:

ssh -L 9090:localhost:9091 -R 9091:localhost:80 remote.server.com

The -L argument starts listening on port 9090 locally and sends all that traffic to port 9091 on the remote server (the domain name, localhost, is resolved by the remote server so it refers to the remote server, not your local machine).  Then the -R argument listens on port 9091 of the remote server and forwards all that traffic back to your machine’s port 80 (here localhost is resolved on your local machine).

HTML 5 Boilerplate reminded me of an old-school tool which is extremely useful for concatenating JS and CSS files on the fly – server side includes:

<FilesMatch ".*\.combined\.(js|css)$">
  Options +Includes
  SetOutputFilter INCLUDES
</FilesMatch>

Then you have a main scripts.combined.js (or css) which contains:

<!--#include file="libs/backbone-min.js" -->
<!--#include file="libs/underscore-min.js" -->
<!--#include file="libs/jquery-1.7.1.min.js" -->

Plus any other files you need, in the order you specify. This works great for development mode so you can change a file on the fly and just refresh the browser without running any kind of build script. When it comes time to push to production, it’s easy for a build script to process the file ahead of time and be confident that you’ll get exactly the same result.

This evening I stumbled across two interesting posts about alternate layouts for pairing rather than sitting side by side. Firstly Tom Dale talking about Tilde’s corner desk pairing setup (and some of the software setup they use) and that was inspired by Josh Susser’s face to face pairing setup at Pivotal Labs.

Both approaches require more floor space which makes them difficult to setup but I would expect the face to face pairing to be a heck of a lot better if done well. I’ve always preferred having separate screens in mirror configuration as well as separate keyboards and mice to allow the developers to sit a little further apart to be comfortable and to be able to look straight ahead at the screen. That said, I quite like having a second monitor for spreading out windows as we have at LMAX so it’s not clear cut which is better.

Yan Pritzker recently posted “5 ways to get insane productivity boosts for coders” which provides some good tips on how to improve your usage of tools when doing technical work. To summarise:

• Never look when you can search
• Don’t repeat yourself (by setting up shortcuts and command line aliases)
• Learn a scripting language
• Learn an editor (and use that editor everywhere)
• Learn regular expressions

However, nearly all of these tips really boil down to how to be more productive at writing text and the mechanics of writing code – editing actual source code files, jumping to different places in those files, executing commands more efficiently etc. Are these really the tasks that consume the vast majority of a developers time?

A couple of little patterns for writing long-poll based JavaScript applications have evolved in some internal apps I’ve been working on lately so I thought I’d document them for posterity.

Simple Long Poll Loop

Peter Lawrey posted an example of using the Exchanger class from core Java to implement a background logging implementation. He briefly compared it to the LMAX disruptor and since someone requested it, I thought it might be interesting to show a similar implementation using the disruptor.

Firstly, let’s revisit the very high level differences between the exchanger and the disruptor. Peter notes:

This approach has similar principles to the Disruptor. No GC using recycled, pre-allocated buffers and lock free operations (The Exchanger not completely lock free and doesn’t busy wait, but it could)

Martin Fowler has posted an excellent overview of the LMAX architecture which helps put the use-cases and key considerations that led to the LMAX disruptor pattern in context.

Good to see some focus being put on the programming model that LMAX uses as well. While convention suggests that to get the best performance you have to make heavy use of concurrency and multithreading, the measurements LMAX did showed that the cost of contention and communication between threads was too great. As such, the programming model at LMAX is largely single-threaded, achieving the best possible performance while avoiding the complexities of multithreaded programming.

The LMAX disruptor is an ultra-high performance, low-latency message exchange between threads. It’s a bit like a queue on steroids (but quite a lot of steroids) and is one of the key innovations used to make the LMAX exchange run so fast. There is a rapidly growing set of information about what the disruptor is, why it’s important and how it works – a good place to start is the list of articles and for the on-going stuff, follow LMAX Blogs. For really detailed stuff, there’s also the white paper (PDF).

As my colleague and favorite debating opponent Danny Yates noted:

We got into a bit of a debate at work recently. It went a bit like this:

“Gah! Why do we have this interface when there is only a single implementation?”

(The stock answer to this goes:) “Because we need the interface in order to mock this class in our tests.”

“Oh no you don’t, you can use the FingleWidget [insert appropriate technology of your mocking framework of choice here – e.g. JMock ClassImposteriser]! I’m smarter than you!”

Joyously, it looks like iOS 5 fixes the issues with contentEditable not working correctly in Mobile WebKit. A patch has landed in TinyMCE to enable the editor to work there. Great news.

Hopefully the changes to WebKit can flow through to Android as well.

With Agile and Continuous Integration becoming popular, a significant number of development teams are using information radiators – mostly focussed on showing the latest build status. Information radiators are a very powerful way to aid communication and subtly adjust a team’s behaviour and attitude. The catch however is that it can be quite hard to control how you subtly adjust the team’s attitude because the pressure to change is so subtle.

Robert Burrell Donkin:

ThoughtWorks are opening their new office right in the center of Manchester with a conference. The cost is £75 and places limited to 50.

I’m particularly looking forward to Dave Farley on continuous delivery…

It’s nice to read things like this and be able to think – meh, I already have to listen to him everyday at work. :) Basically, I’m really enjoying constantly having the feeling that I’m the dumbest person in the room because there’s always a ton of learning that comes out of that experience.

The current competition in the smart phone space between Android and iPhone is both fascinating to watch and provides for some quite vigorous and entertaining debates. I find one pair of arguments particularly entertaining:

1. Apple is abusing their monopoly with iPhone
2. Android has more market share than iPhone and is selling at a faster rate

It’s interesting how often the same person will attempt to leverage both of these arguments (or likely better worded variants thereof) without ever realising that by definition, they can’t both be true. If Android is outselling iPhone, then it would be impossible for Apple to have a monopoly and thus can’t abuse it1{#footlink1:1305386215210.footnote}.

One of the things I really enjoy about the software development teams I’ve worked with is their relentless focus on improving their overall productivity – producing more business value in less time and with less cost each every iteration. It’s largely this drive for productivity that drives developers to focus on quality – buggy code is hard to maintain and delivers little or no business value. Similarly it makes developers really care about the tools they use – the programming language, the OS, the IDE or editor, continuous integration, source control and all the other tools or systems they have to work with. When it comes to tools, nothing is ever good enough and we always want more. So long as that desire for perfection is balanced by careful, realistic prioritization it’s a fantastic thing to have in a team.

The last couple of weekends I’ve been fiddling with a skunk-works project to improve one of the internal web apps at work. Since it’s an internal tool and the entire development team run on Linux I’ve had the luxury of completely ignoring IE support. Previously I’ve felt liberated when I could drop support for IE6 but web development is a whole new world when you’re prepared to drop IE altogether – even IE9 wouldn’t have some of the things I’ve been happily using.

For the past three months I’ve been settling into my new role with LMAX developing a high-performance financial “exchange”1{#footlink1:1301173295622.footnote}, hence the silence on the blog lately. One of the things I’ve found extremely impressive about LMAX is the impressive acceptance test coverage and the relative ease with which they are maintained. Normally as a set of acceptance tests gets to be extremely large they can become an impediment to being agile and changing the system since a small change to the UI can affect a large number of tests.

i18n trap #1: I should have a simple way to map an Enum to a presentable string.

This seems eminently reasonable at first glance – there are many cases where you have one of a set of things and need to inform the user which one you have. Unfortunately human language, unlike their computer language counter-parts, are context-dependent. So that single Enum value in code could well be presented with many different words when presented as part of different sentences or in different contexts.

You may have heard something about Chrome dropping support for H.264 from it’s Video tag implementation in favour of WebM. The reactions are unsurprisingly strong and partisan. From my point of view I see that:

• From a normal user’s point of view there is no discernable difference between a video that is played using the embed tag and one that uses the video tag. Advanced users may notice the performance difference or some of the particular behaviours of either Flash or the video tag but ultimately, both view the video perfectly successfully1{#footlink1:1295131564706.footnote}.
• Essentially the only users who can play video via the video tag but not Flash are iOS users.
• Before Google’s announcement, 99% of the web could view a video encoded in H.264 due to it’s support in Flash. About 40-50% could view one in WebM.
• Since every version of Chrome includes Flash, Google’s announcement did nothing to change those figures.
• There are already a variety of excellent libraries to allow the use of both the video tag with a Flash fallback.
• In order to maximise their potential audience, and thus business, content producers are going to supply video in the most widely supported video format.
• Since supporting two video formats simultaneously is expensive in terms of storage and bandwidth, content providers will avoid using a second format unless it is the only reasonable way to address a significant part of the market.

As a result, the only company who can exert any serious leverage on web video formats is Adobe. If they ship a version of Flash that supports WebM, suddenly it has traction very nearly equal to that of H.264 (but H.264 would still have the advantage of also working on iOS devices). For WebM to have the advantage Flash would have to support WebM and disable support for H.264.

CSS3 introduces the very handy columns support which is implemented in WebKit and Firefox browsers going back a fair way and degrades gracefully in all browsers that don’t support it (which appears to include IE9 sadly). The downside however is that support for the corresponding column-break property to control where column breaks are allowed to occur doesn’t appear to exist despite documentation suggesting it should. So for example1{#footlink1:1293713493905.footnote}:

<div style=”column-count: 3”>
  <div class=”item”>
    <h1>Heading</h1>
    <p>Content</p>
  </div>

  <div class=”item”>
    <h1>Heading</h1>
    <p>Content</p>
  </div>
</div>

Since column hights are always equal, the two items above will be split across the three columns. To keep each item together in a column, we should be able to use column-break-inside:

Many browsers include user preferences to select preferred colours for text, backgrounds and links. This is intended as an accessibility feature so that people with limited vision, dyslexia or other disabilities can choose a colour scheme that makes it easier for them to read web pages. In theory it’s a great idea. In practice however it seems to be nearly useless.

There are two “modes” these colour preferences can be used in:

In ARIA, there are two ways that you can attach a label to an element, aria-label and aria-labelledby. In theory, you should use aria-labelledby if the text is visually on-screen somewhere and this form is preferable. You should only use aria-label when it’s not possible to have the label visible on screen. This makes a lot of sense because generally sighted users like to be able to see the label as well and it prevents the visual and non-visual cues from getting out of sync.

I think Otis Gospodnetić has it spot on:

IMHO, commit messages are an important piece of the communication puzzle, and I feel they are often neglected or completely not even seen that way. Commit messages (that are distributed/delivered to co-developers via email or some other mechanism) help a developer keep others in the loop. They help co-developers stay up to date mostly passively, without having to proactively look for information about recent changes.

I have a bunch of JavaScript unit tests and for some of them I need to trigger a real keypress – one that activates the browser’s default functionality, not just triggers the JavaScript listeners. To do that, I have a simple signed applet that uses the AWT Robot to programmatically trigger the keypresses. Mostly, this works brilliantly.

The problem I’m having is that about 1 time in 10, the focus goes AWOL and the key events generated by the robot simply aren’t delivered anywhere. It’s generally not reliable, though I have got a setup now where I can make it happen within a few minutes of running the tests on a loop.

So Apple have announced that they will be contributing large swaths of code to the OpenJDK project and that from Java 7 onwards, Java will be a separate download from Oracle, much like Flash is now a separate download from Adobe. This really shouldn’t be unexpected for anyone who was paying attention to what was going on rather than just running around thinking the sky was falling.

This is fantastic news for Java developers of all types. Mac Java developers have been asking for Java to be separated from the OS for many, many years so that multiple versions of Java are more manageable and especially to decouple Java releases from the OS release timeline.

As part of exploring JavaScript performance, I started wondering if the particular method of looping made a difference to performance. After all, one of the most often repeated bits of advice is that JavaScript is much faster if you loop backwards through an array instead of forwards and the slow part of my code was a loop. Using jsperf.com I could set up a simple test case pretty quickly and easily: for vs forEach.

I’ve been working to build a better filter for pasting from Microsoft Word into TinyMCE and if you know anything about filtering what Word calls HTML, you know there’s an awful lot of work to be done. As such, performance becomes a real issue – ideally we’d like to be able to filter reasonable sized documents (10-20 pages) in under the half a second that users generally call snappy.

Since filtering Word HTML is also inherently complex and we want to ensure the code is maintainable, I started out by developing the code in the most maintainable way possible, without regard to performance. There’s also a lot of tests to ensure correctness, and allow me to optimise later without fear of breaking things. With that done, it’s time to set up some performance tests and see what kind of performance behaviour it has.

Over the weekend I went to JSConf EU and every time I met someone and told them I was working on TinyMCE the conversation rapidly veered off into why contentEditable is too hard/buggy to work with and how it would be so much better if it was all built on top of canvas or just used the DOM with a custom selection API. This matched the thinking throughout the conference that it doesn’t really matter how messed up browsers are because JavaScript can be used to work around it.

Windows uses a particularly ugly format for putting HTML on the clipboard, called CF_HTML. Basically, it adds a header to the content with pretty useless information essentially declaring how many bytes in the header (and yes you have to pad the byte counts since they themselves are included in the count).

The problem between Java and Firefox is that Java always sets the ‘StartHTML’ and ‘EndHTML’ fields to -1 to indicate there is no context and Firefox will refuse to paste HTML from the clipboard if StartHTML or EndHTML is set to -1. As such, if you copy HTML from Java it will be impossible to paste into Firefox. It works perfectly with Word and IE.

I quite enjoyed reading Joe Duffy’s piece: The ‘premature optimization is evil’ myth, highlighting just how misused that Knuth quote often is. There’s a lot of good stuff that should always be considered, especially around picking the right data structure – mostly to improve maintenance more so than performance.

The most important point I find in the piece is to understand the order of magnitude that matters:

First and foremost, you really ought to understand what order of magnitude matters for each line of code you write.

The issue of P tags vs BR tags comes up very often and since my response to it has become a series of 5 articles, this post is here to provide a convenient index to point to in future.

General Primer

• Why P Tags are Your Friends
• P Tags in Word

Solutions for Specific Situations

• P Tags and Single Line Fields
• P tags and Flash
• The Email and P Myth

What Is contentEditable?

JavaScript editors are all fundamentally based on a technology called contentEditable which makes a part of the web page become editable by the user. While there is a lot of functionality that’s usually bundled under the “contentEditable” title, there are a few core parts which are essential for a browser to support correctly1{#footlink1:1278939007857.footnote}:

• Selection – allowing the user to position a text caret in the content and select various parts of it.
• Content entry and deletion – allowing the user to type basic text and to delete selected content.

Without these two key components of contentEditable it’s impossible to create a high-quality, WYSIWYG editor with JavaScript. Despite some variations in how things work, this functionality is available on all common desktop browsers and has been for quite a few years.

There has been a fair bit of discussion on the WHATWG HTML 5 list around whether or not browsers should ‘sniff’ for video types instead of relying on the HTTP Content-Type header. There are some reasonable arguments on both sides, sniffing is unreliable and can introduce security problems, but a lot of servers default to incorrect Content-Type headers and it’s too hard to configure them.

Having watched that debate go on for a week or more, I found it quite interesting and timely to see a support case come in specifically complaining that we’d gotten one of our Content-Type headers wrong. It turns out that we’d been serving up a JavaScript file as text/html but since it was included via a script tag, browsers always treated it as text/javascript anyway and everything worked. The catch is, Tivoli Access Manager was then set up between the client and the server of this JavaScript and it actively filters all HTML pages. Naturally, filtering JavaScript as if it was HTML tends to break stuff.

One of the greatest frustrations for anyone who develops an HTML editor is the constant supply of people who are convinced they want to use BR tags instead of P tags. Most of these are just people who don’t want “double spacing” and they’re happy once you give them the magical CSS incantation:

p {
  margin-top: 0;
  margin-bottom: 0;
}

The other group however are people writing HTML emails who insist that P tags are completely incompatible with some email clients and cause rendering problems. At one point it seems that Yahoo! Mail really did strip them entirely but now it just applies a different default CSS styling (ironically, the magical CSS above that removes the extra spacing). So if you naively use P without specifying the padding you want, some clients will display a blank line between paragraphs, others, notably Yahoo!, will push all the lines immediately under each other. The solution is of course the opposite magical CSS incantation:

It’s not often I say things like this, but oddly today’s problems are Firefox exclusive. It turns out that prior to Firefox 3.5 Range.extractContents is subtly but, for my uses, devastatingly broken. Basically, on any browser except those earlier versions of FireFox the test below should pass:

test('range.extractContents', 2, function() {
  var item1, item2, list, r, fragment, test;
  test = document.createElement('div'); 
  test.id = 'test';
  test.innerHTML = '<ul><li id="item1">Item 1</li><li id="item2">Item 2</li></ul>';
  document.body.appendChild(test);

  item1 = document.getElementById('item1');
  item2 = document.getElementById('item2');
  list = item1.parentNode;

  r = document.createRange();
  r.setStart(list, 0);
  r.setEnd(list, 1);
  fragment = r.extractContents();
  list.appendChild(fragment);

  ok(item1 === document.getElementById('item1'), 'Item 1 must not be cloned');
  ok(item2 === document.getElementById('item2'), 'Item 2 must not be cloned');
}

Basically, extractContents should remove the nodes within the range from the document and give them to you in a DocumentFragment (so you can put them somewhere else usually). The key requirement is that they be the same nodes unless only a part of the node is within the range, at which point the node should be shallow cloned and split.

Note to self: if you ever want to get a simple list of people who have committed to a git repository, the command you want is:

git log --format='%aN <%ae>' | sort -u | uniq

Or for just the e-mail address:

git log --format='%ae' | sort -u | uniq

Or for just the name:

git log --format='%aN' | sort -u | uniq

This is in the help manual but it’s way down the bottom so takes ages to read through to find it.

Problem

If you’re attempting to use the Java AWT Robot to help automate tests, you may find it convenient to log in via remote desktop to watch the tests execute and verify they work well. You may also find that they work fine while you’re watching them but then inexplicably start failing whenever you aren’t. Basically your test is an angel from Dr Who.

What’s most likely happening is that when you disconnect from the remote desktop session, the console on the Windows box is left in a locked state and you need to enter the user password before you can resume interacting with programs on the main display. Since the AWT Robot is doing a very effective job of acting like a user, it also can’t interact with the programs you’re intending and instead is interacting, probably quite ineffectively, with the login dialog box.

Once upon a time, the state of the art for building software projects was to have a continuous integration server so your builds were repeatable, reliable and performed in a controlled environment. As a bonus they also ran a whole suite of automated tests so you knew that what came out was at least reasonable quality.

Those days have gone.

These days, it’s far more common to have a build and test farm churning away to handle the volume of tests, various different modules and projects that are all built separately and of course the requirement to run the tests on a huge variety of different platforms and environments. VMWare has been a big help in this area for quite some time, effectively letting teams build a private cloud before it was cool to call it a private cloud. Now with the growing availability of public cloud instances, it makes a lot of sense to stop buying and maintaining hardware and simply move the whole system into the public cloud.

I’ve always known you could do a lot of programmatic stuff with Amazon but I’ve simultaneously been too lazy to actually look into it in detail and never really needed to. However, I’ve stumbled across instructions for auto-mounting an EBS volume which turns out to be very handy, and leads on to the particularly awesome right_aws ruby gem.

Lots of useful looking stuff for working with EC2, S3 and EBS volumes that I can see me taking a lot more advantage of in the future.

One of the common examples of the limits of Maven and other “strict” build tools was how difficult is to concatenate CSS files together as part of the build. It’s nice to be able to split CSS files up to categorize things but the extra HTTP requests really slow down page loads, so the build should concatenate them back together. What I’ve come to realise though, is that building a custom concatenation script is just another instance where people are reinventing the wheel in build systems, and getting less power as a result.

If you have a method like:

String getParameter(String name, String defaultValue)

where defaultValue is returned if the parameter isn’t specified, it can be challenging to get the right behavior in JMock without just hard coding what the defaultValue should be. Fortunately, a custom match which doubles as an Action can solve this pretty easily:

import org.hamcrest.*;
import org.jmock.api.*;

public class CapturingMatcher<T> extends BaseMatcher<T> implements Action {
	public T captured;

	public boolean matches(Object o) {
		try {
			captured = (T)o;
			return true;
		} catch (ClassCastException e) {
			return false;
		}
	}

	public void describeTo(Description description) {
		description.appendText("captured value ");
		description.appendValue(captured);
	}

	public Object invoke(Invocation invocation) throws Throwable {
		return captured;
	}
}

It can then be used like:

There’s been a lot of new innovations going on in programming languages and development techniques in general lately, but there’s a really key missing element: metrics.

Seeing posts like Stephan Schmidt’s Go Ahead: Next Generation Java Programming Style and the response from Cedric Otaku really should make any professional software engineer concerned about the future of their craft. Not because either of them are necessarily right or wrong, but because two highly skilled engineers are engaged in debate based purely on pure personal preference. There’s barely even any anecdotal evidence provided, let alone actual measurements to see what impact on productivity, quality or other desirable qualities the proposed approaches have.

There’s a huge amount of new technology pouring out into the world these days – lots of it focussed around the web and consumers and the key term that keeps being talked about it ubiquity. Flash brought video to the web because it was ubiquitous, HTML5 will kill flash once it’s ubiquitous, mobile web access is becoming ubiquitous etc. The most striking thing in all of this is just how quickly these new technologies are actually finding their way into real world usage.

Emmanuel Lécharny – Pick good names for your methods/data structures…

5 years later, when I come back into this crap, I have *no* bloody idea about what is what. Is Twix for the frontend or the backend ?

When you pick a name, and when you think it’s funny, just think about those, and probably you, who will not have fun at all when it’ll be time to fix some code in this area, with no clue about what Twix and Snickers are…

Here’s an important lesson for people combining ant scripts – the way basedir is calculated is very unlikely to be what you expect. In particular, if you combine the task with the task you’re probably in for a surprise.

I learnt this important life lesson when the improved build scripts I’d been working on failed on the build server even though it worked perfectly on my machine. The difference is that the build server is running cruise control and it has a wrapper ant script which checks out a fresh copy of the project then uses the task to build it. The ant task was:

Thanks to a tweet from @bdelacretaz I discovered Apache Pivot today. It does indeed have a nice website, and the library itself looks great. It’s pitched as an RIA framework and mostly seems focussed on browser based deployment via applets. However, pivot apps can also be deployed as standalone applications which is where I think it’s most promising.

The killer feature as far as I can see, is the much richer set of components that are available, combined with a better set of customisation points such as effects, transitions, web queries and data binding. There are libraries to do most or all of this with Swing, but since it wasn’t designed for it up front, they tend to feel pretty clunky and “bolted-on”. Hopefully Pivot can avoid that.

Dylan complained about the lack of testability with file system APIs, which reminded me of a neat looking library I stumbled across while looking at build systems: EntityFS. Among a bunch of other useful sounding things it provides:

File system APIs are backend-independent. File systems can be created in RAM memory, on Zip files, on file system directories, etcetera Sadly it’s under the LGPL which makes it difficult if you happen to sell to big enterprises that are still somewhat scared of GPL type licenses, but they’re nowhere near as common as they used to be. I should note though that Holocene Software are offering commercial licenses for an undisclosed price.

Recently, the subject of build tools and systems has come up again at Ephox and it appears the topic is rising up again around the internet. As part of this I’ve been reading up on and playing with a bunch of build tools to get a feel for their benefits and limitations, so it seemed worthwhile writing up what I find as I go along.

The Projects

Which build tool suits best clearly depends on the type of project you’re working with. I’m currently playing with three quite different projects:

If you’ve tried to set up Alfresco 3.2 and use the WCM component, you’ve probably run into one or more1{#footlink1:1260264856408.footnote} of these problems:

1. When you click “Preview Website” you get an unable to connect message.

   • You need to start the virtualization server that provides the previews or configure it’s IP correctly.

2. When you click “Preview Website” you get a blank page and the browser is “connecting” forever.

   • You did the right thing and entered a root password other than ‘admin’ in the installer and now you’re being punished for it. Go and update the virtualization server configuration and give it the admin password you entered.

3. When you click “Preview Website” you get a response immediately that just reports “Virtual website not found”.

Having become quite interested in languages that support deployment on both Java and .NET, then diving into Scala to find it’s a good language, the obvious next step is to try using Scala on .NET. I found a little time over the weekend to try that out with good success. Notably, I haven’t tried this in a real project yet so there may yet be some significant catches, but so far it looks really quite promising.

When I was in Australia visiting Ephox’s awesome engineering team recently, it was suggested that we should be seriously looking at using Scala for some or all of the server side development we have coming up. The key benefit that was described to me was based on productivity – I certainly think that has merit but I’m somewhat more interested in it being a potential solution to the new cross platform problem1{#footlink1:1255615173985.footnote}. Options like JRuby and Jython are also a possibility in this space, but Scala fits more nicely into a Java programmers mind-set and is designed with the JVM in mind, so while it pushes a lot of boundaries, ultimately it fits with the JVM and Java very nicely2{#footlink2:1255615348893.footnote}.

When I first started professional programming, Java was reasonably young and was being hailed as a work of genius with it’s cross platform deployment capabilities. Programming was going to be revolutionised forever and all programs would run on any OS1{#footlink1:1253859827521.footnote}. Then Java wound up being most popular on the server side and liberated server code from the choice of OS more than the desktop side2{#footlink2:1253860194622.footnote}.

Server OS’s started to compete mostly on how well they were a server rather than what programs ran on them because what really mattered was that you could run the Java stack. In other words, Java became the platform that people deployed their applications to. It shouldn’t come as a surprise that a single platform didn’t stand unchallenged for long. .NET sprang up as the big enterprise competitor to Java and all of a sudden you had to choose sides again.

The relentless addition of cool features in HTML5 is getting a lot of web developers excited and there’s really no doubt that it will be a huge step forward for the web. The more I follow the WhatWG mailing list though, the more I think we’re long past the era where it was considered safe to visit web pages. I’m not talking about browser security holes which have been around for a fair while and certainly do pose a risk, I’m talking about the things that are actually by design.

I keep seeing otherwise intelligent people1{#footlink1:1251967506123.footnote} encounter problems with software but never actually report them to anyone who could fix them. For commercial software opening a support ticket takes just a few minutes and usually gets the issue resolved pretty quickly so you can stop wasting time suffering through it or complaining about it. For open source software it’s often a bit more difficult – you need to join the users mailing list and ask questions there, follow up if you don’t get a response and maybe even put together a solid bug report. Still nothing ominous.

Every so often while using NetSuite, it pops up and asks me to fill in a quick little survey – basically how likely are you to recommend NetSuite and why? This is annoying when your in a rush but on balance not a bad way for them to ensure their customers are happy.

There’s just one catch – every single time I tell them that I’d never recommend NetSuite and that they should improve product quality, specifically they should handle escaping XML tags correctly. As a company that makes HTML editors, we quite often send and receive emails that talk about HTML tags rather than just including them. Unfortunately, NetSuite for the past 5 years or more has happily messed these emails up and we’ve had to come up with complicated ways to get the original email text back out. We’ve reported this to them on multiple occasions and at one point had to actually put in a XSS exploit to get them to understand how big a problem it was. Over time they’ve shifted where the correct HTML will appear in the system, but never actually fixed it entirely.

Back in 2007 I looked at EC2 for a web server and while it wound up being feasible it had a number of drawbacks:

Those familiar with EC2 won’t be surprised to hear that we won’t be going with the service for three reasons:

1. It’s at least as expensive as the dedicated server we’d need.
2. The filesystem gets reset everytime the server reboots (S3 provides a REST API to store and retrieve data, not a filesystem)
3. The server gets a new IP address every time it reboots.

Since then Amazon have rolled out new services that solve problems 2 and 3 and reserved instances to help with 1. What surprises me after a couple of years running a single EC2 instance with an app that’s using S3 for storage though is just how stable it has been.

Remember back when the iPhone first came out and Steve Jobs proudly announced that the SDK for it was “the web”? Apparently history really does repeat itself because now Google is trying the exact same thing with Chrome OS:

The software architecture is simple — Google Chrome running within a new windowing system on top of a Linux kernel. For application developers, the web is the platform. All web-based applications will automatically work and new applications can be written using your favorite web technologies. The decision was extremely unpopular with the iPhone but no doubt it will be extremely popular with Google because it matches people’s expectations. Interestingly, the most common reason people give for the iPhone’s success is now the App Store. I guess we’ll get a chance to see if they’re right or not. I don’t see the web as the only API working even though it would work very well for a large percentage of computer usage. Twitter would be a particularly good example of why – you can use Twitter via the web and certainly some people do, but the standalone clients are by far and away the best way to use it. The same generally applies to instant messaging as well.

I was reminded on the weekend of how much I like working with parser generators – they’re just so pure and clean. You really feel like you’re working with a grammar and all those CS lectures come flooding back. Writing code to parse the same content by hand just never has that feel. Plus they create incredibly accurate parsers in very little time at all.

The development I seem to do these days tends to run at the extremes of reliability – either it has to be fully tested, nice, clean, production ready code, or it’s complete throw away code where development time is the only consideration.

The advantage of doing this rapid fire development is that you wind up with proof of concept code for most situations you’re every likely to run into. The disadvantage is that the code is rubbish and probably no use to you at all. That bit me once again today. I wanted to quickly whip up a plugin that makes pasting plain text into EditLive! work the way I want it to. I have a plugin that filters pasted content to wrap it in a blockquote for my blog so clearly I could just reuse that as a starting point. Sadly, I cheated:

Deployment ruins everything. So many cool technologies that let you develop more rapidly and do awesomely cool stuff fall down at the last hurdle of deployment. Sometimes it’s because they haven’t thought it through properly so it’s just plain too hard, but often it’s just that it’s too hard to convince people that it won’t be another headache for them.

The latest in my deployment-caused frustrations is CouchDB. I have a few use cases that I think CouchDB would be perfect for and it would save me heaps of development effort and headaches. The trouble is, while CouchDB may be of the web, it really isn’t of the enterprise IT architecture.

I’ve come to the conclusion that PHP libraries are simply designed to eat up RAM and do their very best to never spit it back out. There seems to be an assumption that everything will be done in RAM and then at the last possible moment dump everything out to the browser.

Sadly, this doesn’t work if what you’re building in RAM happens to be a zip file containing a whole heap of images. There are a few zip libraries around for PHP but none of them can directly stream the created zip file back out to the browser. Most of them create the entire zip file in RAM and then tell you to just ‘echo zip->file();’ which is just plain crazy. Others can “stream” but only to disk where they have random access.

Apparently my bashrc is append-only:

export gucis=z1419539@gucis.cit.gu.edu.au:

For those who didn’t study IT at Griffith Uni, gucis is the student UNIX server. This particular shortcut was so that I could scp files over there more easily. Actually, given that gucis had open access to the Internet and my dorm room didn’t1{#footlink1:1243014118649.footnote}, it was far more often transferring files from gucis back to my local machine. I finally removed those shortcuts today.

The otherwise mostly link bait suggestion from Matt Asay that Apple and Google should drop their browsers in favour of FireFox triggered an interesting reaction for me. The argument goes along the lines that IE is dominating the browser landscape and FireFox is clearly the second place runner, so everyone who wants to bring down IE should just pool resources behind FireFox. My immediate reaction was to wonder why people saw FireFox as the best horse to back.

I’m stumped so I’m throwing this one out there in the hope that someone might know the answer. I have a JSP component that builds a URL to create a new content item, in a specific site area with a specific authoring template. It works great on Portal 6.0.1.3 and Portal 6.1.0.0 but breaks on 6.1.0.1 and 6.1.0.2. When you go to the URL, it correctly starts creating content but instead of skipping the stage where it asks for an authoring template it just gives a blank list to choose from. If you omit the authoring template from the URL it will correctly list all templates and go on to create the content.

Note to self:

That magical file that controls just about everything you ever want to control about IBM LWCM (at least so far as things that are controlled from the file system rather than the web interface) is under the profile directory at /wcm/shared/app/config/wcmservices/WCMConfigService.properties

This includes:

• configuring backwards compatibility for WCM 6.0 -> 6.1 migrations. Primarily the don’t expire content immediately when no expire date is set.
• setting up the SMTP server properties so you can get e-mail notifications in workflow
  • Side note: These get annoying really quickly if you’re not careful.
• A bunch of stuff on caching that looks cool. I wonder what the changes I just made will do…

Also, why did it take me this long to add fancy list bullet styles to my blog stylesheet? Mixing the blue and green is probably a bit much – I should create a blue tick or green arrow, but oh well, I’m drunk with power.

The conversation that has sprung up around how the use of distributed version control, Github in particular, affects community is a refreshing change in the blogosphere1. It’s people collectively thinking things through rather than just reacting in uproar or following the latest meme.

The latest installment is from Ben Hyde, git: Balene for Knowledge2. It’s definitely worth reading in its entirety, but let me pull out a couple of key points:

A few handy links about JavaScript techniques.

• Loading Scripts Without Blocking – Steve Souders
• Coupling Asynchronous Scripts – Steve Souders
• Why I Don’t Love JavaScript’s Module Pattern – Jonathan Snook

I find the last one on JavaScript’s module pattern most interesting. It’s main reason for existing is to add the concept of private variables to a language which was designed without any access controls. It reminds me a lot of the argument around typing in languages. Many people think having the compiler enforce variable types leads to fewer bugs, in this case the compiler is enforcing access controls.

I spent the weekend discovering just how poorly thought out and poorly supported closed captioning is in online video. Some highlights (based on the SRT format because that was the only one I could find any information about syntax for):

• Definitive guides to the format either don’t exist or are so poorly used that they don’t appear in the first 3-4 pages of a Google search.
• Timestamps use a non-standard format. The inventor was apparently French so instead of the usual hh:mm:ss.ttt the period is replaced with a comma. I sympathise with everything being US-centric, but the number of people who were making this mistake on forums etc and the number of players who support period but not comma (and visa-versa) is scary.
• Only DOS line endings are supported. \r\n is fine as part of the standard, but why can’t players be a little liberal in what they accept? It’s not like \n or \r by itself within the caption is going to work anyway…
• No standard font, so no way to know if the caption will fit or not.
• Absolutely zero feedback from any player I found about why it was ignoring the captions.
• Really, really poor tooling support for creating captions.

So, coming around to the real point of this post, I wound up writing my own tool to help take a script, synchronize it to the spoken voice and output the right format. Worked a treat, and now I have working captions on my video. I suspect it would be useful to others and there’s certainly plenty to do to improve it that would be great if others jumped in and helped with.

Shelley Powers:

Weblogging, where no one really knows how many people are following you, most people don’t care, we can actually communicate complete thoughts, and do what we want with our URLs. Twitter is like haikus, an interesting application of extreme limits that inspires creativity and a different way of thinking. Entirely useful, fascinating pursuit, good luck to you all. Haikus have their strong points, but let’s face it even the best haikus don’t come close to telling the same kind of story as a novel, or even a ballad. Even a great long string of haikus wouldn’t.

When installing most if not all versions of IBM Portal on Linux, to get the installer to run you need to install a couple of extra packages:

• openmotif
• compat-libstdc++

Depending on the options you chose when first installing Linux there may be more – the easiest way to find them is to know that the graphical installer sends it’s error messages to /tmp/wpinstalllog.txt. It’s usually fairly easy to match the error messages up to the package you need to install to fix it.

The Australian brings news of the growing battle for mobile banking leadership among Australian banks:

Brisbane-based Suncorp launched the first mobile browser-based banking service and last week made it compatible with iPhone and Google Android handsets.

The Commonwealth Bank has similarly updated its mobile service, which will work on any internet-enabled mobile phone, and has additional functionality for the iPhone.

People have been talking about the coming mobile revolution for a long time. In fact, as the article mentions, the Commonwealth Bank had previously tried to jump on the mobile banking wagon as early as 1999 via a WAP interface. So what’s changed and what does this have to do with accessibility?

I’ve been fiddling with my blog again. I’ve enabled a couple of plugins – one to allow comment preview and one to add OpenID authentication when commenting. OpenID authenticated comments skip the moderation step. All working quite nicely – the OpenID plugin even lets me run my own OpenID server instead of using Verisign.

The problem is, to set up the OpenID server, the plugin adds two meta tags to each page:

I’ve done a fair bit of train travel today to get oitto a meeting. The trip was easy enough to pln online and even buy the tickets in advance but there were a few simple failings that made the trip more difficult than it should have been. Firstly the website conveniently allows you to reserve seats online but it makes you reserve seats in both directions or not at all. If you’re not sure when you’ll be returning you’re left with pot luck as to whether you get a table and power point on that busy morning train. Oh well. Then you get a handy little urinary for the trip showing which stations you change at when you arrive at each and when you depart again. Sadly all the signs at the station show the final destination of the train instead of the stop you want. So the 8:40 train to Banbury you’re looking for doesn’t exist. You need to go to each platform to fin the more detailed sign that lists all the stops. The key lesson is to make sure that you sync up the information you’re providing rather than just showing the minimum required information. In this case the final destination actually really important to know purely because that’s how the train was most commonly identified. Still, I made it just with a little extra excercise going up and down the overpass stairs.

The upgrade process from Portal 6.0 to 6.1 is really overly complex for a simple little one server deployment like I’m managing. To save myself that pain I’ve manually migrated a bunch of components and written a script to bring over all the content.

Firstly, it’s been a lot of fun playing with Java’s ObjectOutputStream and designing stuff to work well with that. Nothing earth-shatteringly new, but just a few little tricks like when you want to serialize a list of objects:

A little while back Charles Nutter posted about his progress at getting the JRuby bug list under control:

Roughly around September, we crossed an important threshold: 500 open bugs. They dated as far back as 2006 (we moved to Codehaus in 2006), and across the entire release sequence of JRuby; there were even some bug reports against pre-1.0 versions. From September until about mid-January, we worked to keep the number of open bugs below 500. Sometimes it would peek above, but we were generally successful.

This one should be obvious but well, it wasn’t… When you upgrade to the latest Debian stable (Lenny at time of writing which was released 14 Feb 2009), it will upgrade PAM and a few other really important login-type modules.  At the time it will tell you that you have to restart any services that use PAM or they mightn’t authenticate properly and offer to restart a number of services for you so everything seems happy.

One of the great challenges of writing an HTML editor is discovering and smoothing over all the weird complexities in HTML. There are just some areas of HTML (and CSS) that are brain-dead stupid and you have to wonder how on earth it ever came to be like that. I suspect most of those brain-dead areas are involved with alignment or tables. This of course means that aligning tables is particularly stupid.

Why is it that Tim Bray’s perfectly correct Atom feed is being converted into broken RSS2 by the time it goes through NewsGator and comes out in NetNewsWire?  The reason I ask is that whenever Tim posts photos in a blog entry they wind up returning 404s from his direct feed, but work perfectly when they come through Planet Intertwingly.  It’s crazy.

The actual feed content contains (excerpted):

<feed
xml:base='http://www.tbray.org/ongoing/ongoing.atom'
…
<entry xml:base='When/200x/2009/01/22/'>
…
<img src="PS082906.png" alt="Fog" />

When I view the XML source in NetNewsWire though, it’s been corrupted into an RSS2 feed and the image tag is now:

I’ve had a couple of articles open to remind me to blog about them for a while now – one from Tim Bray’s Android Diary and a response to it from Nick Kew. The key part of Tim’s post:

At a deep level, debugging with print statements is superior to all other approaches. Which is good, because we seem to be stuck with it. Firstly, I’m very strongly against debugging via System.err.println – you should have a logging system in place instead and just add extra messages to that. That way if you accidentally check them in, at least they go to the logs and are hidden at DEBUG level rather than annoying everyone all the time by going to System.err.

Shelley Powers on Burning Bird’s RealTech:

We’ll see a significant reduction in MSIE corporate users, as many will get laid off. Fair cop…

Geir turns out to be the latest in a very long and prestigious line of folk to bring up a rather common fallacy that I find amusing:

I’d love to see a common UI for things like this, just like cars have a common UI for the basics Cars do have some things that are very much similar – they have a steering wheel and at least an accelerator and brake. That’s good and while you could define that as the basics as Geir was probably thinking, that’s about the same complexity as turning on a computer and maybe opening a program or two.

One of the hardest types of bugs to track down is a deadlock situation. They are very time dependant which makes them intermitten, specific to a particular computer and configuration and generally impossible to reproduce in a debugger. Fortunately, at least in Java, it’s fairly easy to spot most of the situations where a deadlock is possible:

1. Calls to SwingUtilities.invokeAndWait
2. Calls to Object.wait()
3. synchronized blocks

There might be a few others depending on the libraries you’re using, but starting with those three, in that order, is very likely to lead you to at least one point in the deadlock.  Just put an old fashioned System.err.println before and after each of those calls and you’ll quite quickly see where things are waiting forever.

For future reference, James Tauber has a very useful article on using the HSL color scheme to create shiny gradient buttons.  It’s worth knowing that in Java HSL is actually called HSB (brightness instead of luminosity) and there are various methods on the java.awt.Color class for converting between HSB and RGB as well as creating color objects directly from HSB.

After my amazingly successful use of mod_proxy to provide clean URLs in an IWWCM instance, it’s been added to my bag of useful tricks to know about.  When you realize you can proxy differently based on the current virtual host it’s a very powerful solution.

My latest use for it was to add name based virtual host support to two completely separate virtual machines.  One machine runs IBM WCM and the other runs Quickr. Both use the same port, and in the future there will be more VMs with different versions as well, so while it would be possible to assign different port numbers, I’d prefer to not have to remember which VM is using which port etc.  The firewall however can only forward connections on a given port to one VM.

There’s a lot of talk around at the moment about how to allow cross-domain AJAX requests without compromising security. What I don’t get, is why this whole thing is an issue. What’s the difference between the two examples below:

Option 1: Browser connects to site A which loads JavaScript and initiates a request via AJAX to site B.

Option 2: Browser connects to site A which loads JavaScript and initiates a request via AJAX back to site A. The server at site A proxies the AJAX request to site B.

Dear lazyweb (actually, just more intelligent web, I’ve done a lot of searching on this already…),

I have a nice install of WebSphere Application Server and IBM Portal 6.0.1.3 up and running with the WCM component configured. This is great, really happy.

I’d like to add Quickr (later also Connections and maybe even Sametime) but all the Quickr installs I can find want to deploy a second instance of WAS which isn’t ideal. Is it possible to just add Quickr to an existing install or is it possible to install it separately and move it into an existing install?

Since the search function for Apple’s mailing lists is pretty much useless, I’m making a note of the Reading PICT Images thread which contains numerous options for rendering PICT images in Java on OS X.

So here’s a surprise – FireFox 3.0.1 can’t handle RSS/Atom feeds that have a port number in the URL. The problem stems from the fact that it uses a custom feed: protocol instead of just working off of the mime-type of the content.  So if you visit http://www.example.com:10038/ and it provides an RSS autodiscovery link that points to /feed/, FireFox will try to go to feed:http//www.example.com:10038/feed/.  Note the missing colon after http.

I’ve always liked the Atom spec. It’s neat and tidy with strict rules about what’s valid and what’s not with all those rough corners and incompatibilities of RSS sorted out (well, mostly). If I run into one of the silly sites that offer both RSS and Atom I pick Atom just because it feels right even though both would work perfectly well for me. So it came as quite a surprise to me to discover a major weakness in the Atom spec – it’s a right pain to generate. Let me explain…

Working in the world of editors there are a range of blog posts that float back up to the surface every so often, generally because they discuss an age old concern that just keeps resurfacing.  Recently a post from Moxiecode resurfaced complaining about the lack of focus on editing support in browsers. There’s been a few such posts in the past that I’ve seen and while the world of contenteditable support has definitely improved lately, it’s still one of the weakest areas of modern browsers.

Sylvain Wallez:

If your application uses java.net.URL, and chances it does are very high, and you are using Sun’s JVM (since 1.4.2), you should set the sun.net.client.defaultConnectTimeout and sun.net.client.defaultReadTimeout system properties to a reasonable value. Otherwise, if a remote site hangs, your application or server will also hang. Useful to know…

Dear Lazyweb,

Since this worked so well to find a great article on HTTP caching… Does anyone know of a good introductory article for how to get MathML to display in Web Pages across multiple browsers etc.

My primary recommendation unfortunately will be to convert it to an image, but I’d like to provide instructions for the folk who want to maintain accessibility for their equations as well. Related and also interesting is anything discussing mime-types, XHTML and how to solve the IE problem.

Dear Lazy Web,

Can anyone point me to a simple introduction to HTTP caching? I know quite a lot about it and can just dive into the relevant standards, but I’m writing an article that will be read by people who have never heard of HTTP cache control headers before and I’d like to give them a good starting point.

Thanks,

Lazy author, er believer in reuse…

Answer

This article on MNot has an excellent overview of the whole issue.

From CMSWire:

Optaros has announced the launch of a new user interface framework for Alfresco. DoCASU 1.0 is an open source initiative, bringing AJAX and Rich Internet Application features to the Alfresco Enterprise CMS platform UI. Docasu “is not intended to replace the already existing Alfresco web user interface,” but rather is aimed at those who require a simpler and more user-friendly, yet highly configurable interface solution. Sounds like a sensationally good idea to me. It’s surprisingly difficult to get a team together that has the back-end skills required in terms of scalability, content management etc and the front end skills to create an intuitive UI for users. Or in graph form and Ephox specific:

I had two of our keen young developers (Dylan and Suneth) email me overnight to ask my CTO-ish opinion of trends in the mobile space and how they might apply to Ephox. It’s a very good question – with the advent of BlackBerrys first and now even more so with the iPhone, mobile internet is finally moving from “the future” to “the now”, even if it’s not evenly distributed yet. Of course, Ephox is squarely placed in the enterprise content creation business so no matter how popular the mobile world becomes we’re very unlikely to bring out a mobile phone game or a tip calculator. So here’s my take one where the mobile world is with regard to enterprise content creation.

Mike Gunderloy provides an overview of the terms and conditions from three of the popular online office applications and questions who owns your documents? The more important point that comes out of it though is who is backing up your documents? When people move data into “the cloud” the often forget that ultimately having backups is their problem and they should only trust themselves to do it.

One thing that’s clearly missing is any sort of backup guarantee. While you may feel more secure storing your documents on Google’s or Zoho’s or Adobe’s servers than your own, that security is not something that you’re promised. Any of the three can lose your documents or terminate your ability to get to them at any time for pretty much any reason, and you’re out of luck. That’s precisely why I ensure that any data in a hosted solution of any kind is also backed up locally. I’ve already been through the experience of hosts going broke, or just plain stuffing up their backups and having to restore from my local copy and in time everyone will.

It’s really amazing how many web sites have broken shopping carts in one form or another. It’s the ultimate form of stealing defeat from the jaws of victory. The favorite is always shopping carts that time out. Nothing like throwing your customers out of the store after they’ve decided to purchase from you.

British Airways seem to have perfected the art of displaying an error page just when you were pulling out your credit card. Bonus points for reporting that their systems aren’t responding as if that actually means something to the user who just got a response from their systems – the error page.

Got a couple of good comments on my last post about NewsGator Syncing that I thought were worth following up on. Firstly, Greg Reinacker points to the article I had in mind about how NewsGator polls the feeds, and Andy pointed me to this forum posting about it which shows how to see why feeds aren’t updating.

When I go and look at my feeds I find a whole bunch reported as having errors from the source of the feed – obviously why they’re not updating. There’s just one problem, even Andy’s comments are marked as broken:

I love the fact that I can read my feeds in NetNewsWire and on my iPhone seamlessly, but there’s one really annoying aspect that’s almost driving me to turn off syncing for a large number of feeds: NewsGator is days or weeks out of date for many feeds.

When syncing is enabled in NetNewsWire it no longer downloads feeds directly, but instead gets them from NewsGator which is how all the syncing magic works. This leads to much faster sync times but also means that you can’t actually refresh your feeds to find what’s new. All the refresh button does now is check NewsGator and there’s no way to check directly with the site itself.

I’ve seen in a few places people getting excited about the upcoming support for Google Gears in WordPress as a way to locally cache common files so they don’t have to be downloaded repeatedly. For instance, this article from Geniosity Musings:

But, some of the new features (and features I’ve just started using now that I use the Visual Editor) just aren’t as cool thanks to the not-so-great internet speeds in South Africa.

Kevin Gamble echoes an increasingly common theme at the moment, complaining that some US ISPs are trialling metered internet plans instead of unlimited data:

This is serious stuff. This is an both an economic and freedom issue. Changing the way the Internet works means people will be less likely to share and to try innovative things. If you don’t think this will impact the quality of your Internet experience you are dead wrong. It will make a massive difference in changing people’s online behavior. Here’s the thing though – metered internet plans are not a new idea. They’re not even unusual, they’re just a way of life for a huge number of people, like say pretty much anyone in Australia. You don’t have a civil right to unmetered internet access any more than you do to unmetered electricity.

More a bookmark for myself than anything – the W3C has published a preliminary guide to differences between HTML 5 and HTML 4. Quite useful for anyone planning to update their products for HTML 5.

There’s already a lot of stuff written on the internet about how GoDaddy SSL certificates aren’t recognized by Mac but are by Windows, all of it pointing to “a configuration problem”. I’m not sure how we got such special treatment but none of the instructions I’ve seen work in our particular case.

In case you’re not familiar with it, the problem is that on Mac OS X connecting to the site displays a dialog saying that the certificate could not be validated for an unknown reason. Thanks a heap for that OS X… On Windows it works just fine. The problem turns out to be that the server isn’t configured to provide the full issuing certificate chain all the way back to the root SSL certificate (which is in OS X’s set of trusted roots by default). All around the internet you’ll get instructions saying to make sure that the SSLCertificateChainFile is set to point at gd_bundle.crt (available from GoDaddy’s repository). This doesn’t work with our certificate, not sure why.

Doug’s discovered a way to improve the effectiveness of simian to avoid adding more duplication to a code base:

The solution is very simple. The simian-report is a an xml file, so I wrote a SAX2 DefaultHandler that was able to parse the number of duplications at the different threshold levels. Putting this into a trivial ant task then gave us a task to help make things no worse even at levels below what the simian-check was doing! Within the first week, the new legacy-check was breaking the build (where the simian-check would never have) and focusing the teams attention on how to make things better. The solution is simple and really cool. I have no doubt that it will have a big impact on the amount of duplication and the overall quality of our code. The trouble is, Simian’s normal reporting is really, really lousy and Doug’s extra check highlights just how useless Simian is at telling you where the duplication is instead of just detecting that there is duplication.

Back as far as Raskin’s The Humane Interface, and quite possibly before, modes in user interfaces have been frowned upon. Despite that, huge amounts of software ships with a simple mode and an advanced mode. The theory being that when users get started they use the simple mode which makes the simple tasks they want to do really straight forward. Later when they want to do more than the basics, they’ll be more familiar with the software and thus be able to handle the advanced mode.

David Welton described a frustration he had with FireFox’s spell checker which piqued my interest:

I write most of my email (in gmail) and submit most web site content in English, however, a significant portion is also done in Italian. I leave the spell checker in English, because Italian is, in general, quite easy to spell, so that even as a native speaker, a helping hand is occasionally welcome. However, it isn’t as if I write Italian perfectly either, so the help there would be nice as well. I find it quite annoying to go change the language in the spell checker option each time, especially when, as an example, I’m responding to email and do 2 in English, one in Italian, another in English, and so on. On the face of it, identifying what language an author is using looks a lot like a natural language processing problem and thus requires a PhD and many years of research to tackle. Looking a little closer though, you begin to realize that the problem domain is dramatically reduced in this case:

Kevin Gamble pointed me towards the Sun Wiki Publisher for publishing documents to MediaWiki servers straight from OpenOffice/StarOffice. The key problem with these types of integrations is that wiki markup simply can’t handle anywhere near the same level of expressiveness as even HTML, let alone a word processor document. Hence the description mentions:

All important text attributes such as headings, hyperlinks, lists and simple tables are supported. Even images are supported as long as they have already been uploaded to the wiki site. An automatic upload of images is currently not supported.

Jeff Atwood:

Some of the largest sites on the internet — sites you probably interact with on a daily basis — are written in PHP. If PHP sucks so profoundly, why is it powering so much of the internet?

The only conclusion I can draw is that building a compelling application is far more important than choice of language. While PHP wouldn’t be my choice, and if pressed, I might argue that it should never be the choice for any rational human being sitting in front of a computer, I can’t argue with the results.

Nick Bradbury:

Those of us who believe that desktop software is still relevant in a browser-based world should be up in arms about how hard it is to install software (on Windows, at least – it’s easier on the Mac). Multiple security warnings, required OS updates, and tech-heavy language make downloading and installing software too scary a prospect for non-technical users.

Desktop software on all platforms can and should be really easy to install. All it requires is for the user to click a link, click “Run” and the application should be up and running. In future they should be able to access it directly from their hard drive (start menu, Applications folder, command line, whatever is appropriate for the platform). Anything more complex is purely because the developer chose to make it more complex.

Automator is a cool little app that comes with OS X that makes it much, much easier to programmatically control the various applications on your computer (all the good ones anyway). It’s all done by dragging and dropping actions into a workflow – the output of one leads to the input of the next. In Leopard, Automator got a big upgrade so it can now handle loops and variables but most of the time you don’t need them.

I’ve recently noticed that having auto-save functionality no longer makes me feel “safe”. Previously, if a program automatically saved every 5 minutes or so I was completely confident that I could never lose more than 5 minutes work which seemed good enough. Now however, I want more than auto-save, I want version history.

If I can’t go back and find a previous version then I can’t be confident that the next time I save it won’t corrupt things and I’ll lose everything. Odd really.

We’ve found a cheap little NetGear router that can roughly load balance and fail-over between our two internet connections to hopefully get a little bit more speed. Of course, the simplest thing to do is to set it up so that from the inside it looks just like the old modem and then on the WAN side set it up to look like a client using that old modem as it’s router. Of course, that means that the inside and outside of the router are in the same subnet (192.168.0.0) in this case and the new router’s internal IP is also the IP of the WAN router it forwards on to.

For some time, we’ve had an Arabic translation in EditLive! and the editor pane itself fully supports bidirectional text, but we’ve never updated the UI to flip over to a mirror image when a right to left language is in use. Java actually has pretty reasonable support for this via the ComponentOrientation property in AWT (and inherited through to Swing) but there are a couple of annoying limitations:

1. It applies on a per component basis, so there isn’t a single place you can apply the orientation and be done with it.
2. Various components still don’t flip, eg JSplitPane.
3. The Aqua L&F on OS X has some annoying bugs when in right to left mode.

For the first issue, you can use applyComponentOrientation which will automatically traverse the component hierarchy and make the changes but it still misses things like combo box renderers, dialogs or components that are generated later. For an application that loads its UI incrementally instead of blocking the swing thread while it loads in one go this is a significant annoyance. Even beyond that, all the dialogs need to be updated as well. While it is good that you can override the orientation on a per-component basis, it seems more intelligent to have a simple way to change the default value in one place without all the work.

So the iPhone roadmap looks very promising. The enterprise functionality is really impressive and places the iPhone extremely well as a mobile device for corporations. The SDK has a lot of power and seems to have access to pretty much everything you’d need (there’s already an SDK for access to the dock connector). Even things like instant messaging and VOIP will be allowed, though obviously the carriers don’t want to deal with all the traffic from VOIP so it’s just wireless but that seems quite reasonable to me. I’m not sure I’d want to try VOIP over an edge network anyway…

All the hubbub about flash support on the iPhone highlights an interesting “gotcha” that many people fall into: there’s a world of difference between having something running and having something you can actually ship. The funny thing about software development is that it is generally much quicker and easier to solve all the “hard” problems and create the software you need than it is to polish off all the little loose ends that turn code into an actual product. It’s also surprising how many ideas that sound really good on paper turn out to actually make things worse.

Great notes on some common mistakes Java developers make from Christopher Owen at Atlassian. I know I’ve explained most of these to a number of new developers – well worth a read even if you’ve been coding in Java for a number of years you may easily have not learned these important lessons.

Ephox folk – we learn a lot of interesting tidbits like this both on coding and on XP and development processes and tools – could we take a little time to share them with the world and each other more often?  Ta.

After much hassle with getting proof of address and what not, I’ve finally managed to get a mobile broadband deal through 3 in the UK and I’m very impressed. It helps that my house is within a HSDPA area so we’re getting the fastest possible speeds but even when that drops out and we just get 3G connectivity it’s quite usable. At £15/month for 3Gb/month it’s actually quite affordable too. I can see it getting a lot of use on the train and places like that so I don’t wind up wasting huge amounts of time travelling.

Some time ago, Jeff Atwood posed the question: How Should We Teach Computer Science? In the article, he laments the fact that formal education doesn’t teach students about deployment issues or source control:

 And yet, as Greg notes, existing software engineering textbooks give this crucial topic only cursory treatment. Along the same lines, a few weeks ago, a younger coworker noted to me in passing that he never learned anything about source control in any of his computer science classes. How could that be? Source control is the very bedrock of software engineering.

My previous post, Atom is the New JCR, sparked some interesting conversations, firstly from Lars Trieloff via private e-mail (with a option of spilling over into a pub if we ever wind up in the same town…) and now from Dan Diephouse. Both Lars and Dan made a really good point – that JCR needs to find ways to work together well.

I completely agree that JCR isn’t very worthwhile as something that will break down content silos. It does have value as an API to work with data though. Atom is quite limited in the granularity it can work with data (which coincidentally is one of the reasons Web3S exists as well). And you still need to store your data somewhere.

So if I build a social network on Ning I can add forums which are kind of cool except that noone actually knows that new stuff has been posted and don’t bother checking back in. It doesn’t seem to matter what options you provide – RSS feeds, offers to email notifications of new threads etc – people drift away from forums very quickly once their question has been answered. On the other hand, mailing lists tend to be harder to get people to use in the first place because you have to subscribe, but then they tend to stick around longer because they’ve already subscribed. If in that time you manage to teach them a few things they didn’t know they needed to know they hang around permanently and the community grows.

A couple of days ago we had an issue where a new test I’d checked in passed locally but failed on the build machine and I didn’t get a build failure notification. Everyone who did get a build notification thought I was working on it and the build stayed broken overnight. Clearly, we need a really obvious way of reporting build failures that everyone notices and continues to notice until the build is fixed again.

When the Java Content Repository (JCR) standard first came out it was supposed to bring in a new era of compatibility between content repositories and put an end to the content silo. There was, and still is, a lot of talk about it and just about everyone added JCR compliance to their marketing materials. Unfortunately, that’s mostly where things stopped – the implementation work that followed was generally done was buggy or incomplete and the only viable JCR implementations that I’ve seen have come out of Day Software, who lead the JCR spec effort. There are a few CMSs around that do have good JCR support – Alfresco for example – but they’re few and far between and even with that, there isn’t a lot of people taking advantage of that support and the standardization of the repository interface.

Michael Krigsman sticks it to Nick Carr and includes an interesting assertion: that how good software is can be decided by how much revenue it drives:

Nick, please let the market decide whether enterprise software is “good” or not. There’s a simple metric for measuring this: it’s called revenue. Just for kicks, compare the revenue of enterprise companies, such SAP or Oracle, to consumer-oriented firms such as Twitter (click to follow me).

The conversation around enterprise software goes on, with a couple of good responses to my last post that I want to highlight. Firstly, ddoctor (aka Dylan Just who recently started working here at Ephox) in the comments:

I’m thinking of making this one of my career goals – making enterprise software not suck.

Then you’re very much in the right place – that’s what we do…. He goes on to give some very good advice on designing good UIs, but it misses a key point that I was trying to make in my last post:

I original skipped over Robert Scoble’s post, Why Enterprise Software Isn’t Sexy, it just seemed too obvious to be worth reading in much detail. I’ve been working on software that sells to enterprise customers for the past 6 years or so and no one cares about it, but release a poor version of that software for the consumer space and everyone goes ga-ga over it. EditLive! and eWebEdit Pro have been bringing WYSIWYG editing to the browser for years and no one cared because they were sold to the enterprise, but when Google put out Google Docs everyone went crazy about it, even though it has half the functionality and twice the bugs.

I’ve been keeping an eye on OpenSocial since it’s initial annoucement with some interest but also a healthy dose of skepticism. I’m still wondering why anyone would want to support open social. It doesn’t give you any integration between systems – all it provides is potential access to the OpenSocial widgets that 3rd party developers make.

Now, if we look at the 3rd party widgets from FaceBook you’d be doing your users a great favor by not supporting OpenSocial. I’m yet to find any FaceBook “application” (I use the term with great disdain) that doesn’t do whatever it takes to sign up new users, usability and user experience be damned.

Maybe I’m just a spoil sport but I think project code names are the most ridiculous concept. I’ve never seen a code name help clarify things – they only ever cause confusion. Compare:

Hey Joe, how’s the schedule looking for futzbist?

with:

Hey Joe, how’s the schedule looking for the next EditLive! release?

The argument is of course that everyone gets to know the project code name so it’s a nice shorthand for the project – but for how long do they remember them? How far back can you name Java, Ubuntu, Debian or even OS X releases? Did Puma come before or after Cheetah? What was the Tiger release? And what’s in the stray alley cat release?

Is it just me (quite possible) or is the standard JSON validator for JavaScript unable to handle quotes (") in strings? Specifically, shouldn’t the JSON below be considered valid?

{“name”:"""}

It throws a parseJSON error when run through the validator but executes perfectly as JavaScript and seems to match the specification for strings in JSON. It’s frustrating because even though the data is coming straight from the server so it’s as trustworthy as it can get, I still think it’s incredibly poor form to execute data without first validating it.

The rumors of Java 5 being horribly broken beyond all usability on Leopard are, quite frankly, bullshit. It’s faster, has better integration with the OS, the Aqua L&F is significantly improved, it has full support for 64 bit and a huge raft of bug fixes and miscellaneous improvements. Everyone’s pointing to an uninformed rant on JavaLobby which as it’s key example actually highlights a major improvement to the Aqua L&F – the JOptionPane icons should use the application icon, not some obscure artwork that’s not used anywhere else in the system. The new dialogs actually allow you to look more like native app, not less. It’s even explained in the release notes – heck, the old implementation was probably reported as a bug against Tiger.

My first thought when I heard that the Leopard blue screens being were by Unsanity’s APE was “and that’s why only fools use hacks like that”. So it was with some surprise that I read John Gruber’s article on the issue this morning and discovered that the Logitech driver installs APE. Moments before I’d installed the Logitech driver…

Turns out my decision to do an archive and install to get rid of left over cruft from all the software that I’ve tried was a pretty wise one – I’ve had the Logitech drivers installed for quite some time, so most likely I would have run into the “blue screen of foolishness”.

I was silly enough to open my work email this morning, only to discover that the Apple Java-Dev list had broken out into the age old Java on OS X argument. First up here’s what people have reported1:

• Java 6 is not included with Leopard.
• The previous Java 6 DP which was pulled from ADC a while back does not run on Leopard.
• Upgrading to Leopard from a system with the Java 6 DP installed can cause some frustrating issues with switching Java versions.
• Apparently Java 5 is much faster on Leopard.
• Java 5 looks different with quite a few tweaks to the Aqua L&F.
• Some fonts don’t look right in Java 5 on Leopard because it uses the Sun 2D graphics pipeline instead of the Quartz pipeline. The Sun 2D pipeline doesn’t support sub-pixel antialiasing. You can override the default and there’s a few other conditions that trigger the Quartz pipeline to be used by default.
• Java 5 supports 64bit on Intel Core 2s (but not PPC). There seemed to be some problem with it when using the Java tools in /usr/bin though – can’t say I followed that discussion too carefully.
• Lots of documentation is coming, but not much is available yet.

Pretty sure that’s the Java on Leopard wrap up, the other 150 emails to the list were just the usual gnashing of teeth about Apple abandoning Java and how Apple will lose so much business if they don’t get Java 6 out yesterday etc etc etc. Of course, Apple’s doing better than it ever has before and JavaOne was full of people using Macs – without Java 6 – so it would seem it’s all just talk and insignificant numbers of people are actually leaving OS X. As a fresh twist, this time round people are talking of porting OpenJDK to OS X themselves and finally freeing themselves from the evil clutches of Apple! Apparently no one has told them that Java 6 isn’t available from OpenJDK either – it will become Java 7 and is quite some way from that yet. I think it’s a safe bet that Apple will have Java 6 out long before even a reasonable uncertified port of OpenJDK is available for OS X.

I’m writing up three proposals for the O’Reilly Web 2.0 conference. I think the content is compelling enough but I’m not overly hopeful that any will be accepted because it turns out I really struggle with writing session abstracts. This is actually the first conference I’ve submitted proposals too so hopefully I’ll get better (and faster!) at it with a bit more practice. In a lot of ways I’m actually more interested in just getting practice at submitting proposals than actual speaking at the conference – the timing of the conference is bad, but can be worked around.

If you’re using the scp or ssh tasks with ant, you may run into a problem where part way during the upload or never disconnecting after the command completes for the ssh task. There are a couple of possible causes:

1. The scp problem is almost certainly caused by using ant 1.7.0 or below and jsch 0.1.30 or above. You could upgrade to the latest nightly of ant1 but it’s probably easier to just drop back to jsch 0.1.29 which is what ant was developed against and works nicely. Bug 41090 contains the gory details.
2. If the command you’re executing with the ssh task starts a background service or otherwise leaves a process running, that may be the cause of the problem. You can add ‘shopt -s huponexit’ to your /etc/profile, .bashrc or somewhere like that. I must admit, I’m somewhat vague on the exact details of what that does but the basic idea seems to be to signal any background processes that bash is exiting and then not wait for them to complete (which allows your ssh connection to close). If you’re starting a server they’ll probably ignore the hup signal it sends and if not, use the nohup command.

Hopefully that will be the last I’ll see of that issue.

The realization that there is valuable information in users attention data is a wonderful thing – it leads to so many really useful features like Amazon’s recommendation system. I’ve seen a lot of really good uses of this kind of data where systems use fuzzy logic to improve a users experience or make recommendations of things they’d like. It appears that Microsoft has noticed this trend as well, but somehow I think they missed the point:

Some interesting responses to my complaint about feed readers stripping CSS:

• Nick Bradbury – Response: On Stripping Styles For Security
• Sam Ruby - Stripping Styles

There’s a common misperception that my complaint was about all styles but in fact I was just referring to inline styles on the basis that they are actually part of the content, not just presentation. Sam Ruby points to a feed from Wikipedia that is exactly the use case I had in mind. Many of the comments however want to strip styles to preserve a uniform look in a “river of news” type of reader, for example Nick’s comment:

A while back people discovered that many RSS readers, and all online RSS aggregators didn’t sandbox content from different sites and malicious HTML could cause cross site scripting (XSS) attacks and general nastiness. As a result most feed readers filter HTML through a seriously restrictive white list, including removing all CSS information. I’ve reached the point where I’ve simply had enough of this. CSS is a vital part of the internet and if feeds are going to be useful, we need them to work with CSS properly. So let’s take a look at what’s really going wrong:

Here’s a really simple golden rule for anyone thinking of adding auto update to their products – never ever include any user identifiable information_._ There’s simply no reason you need to know who is checking for updates, you only need to know what version they have. Given the infrastructure of the internet you will wind up getting their IP address, your policy should be that these aren’t stored.

It comes as no surprise to me that the WordPress mob broke this rule with their new auto update – they always seemed shifty to me. Tell me why exactly you need the URL of the blog to determine if a new version is available? Exactly what use to you is blog.ephox.intra going to be? Oh well, I’m already removing all the pointless blog entries they spam the dashboard with and those weird technorati partner parameters so I guess I’ll be asking for updates from wordpress.com or something too….

It’s a common question – what’s the point of social networks? The most common answer is basically none. Most social networks provide yet another way to get in touch and keep in touch with people which is great but lets face it, there are about a billion different ways to communicate and leaving messages on someone’s wall only looks good compared to sending smoke signals. Some people might argue that it lets you map out and visualize your social network but seriously I know who my friends are, why don’t you? The address book has been around a long time and it still works seriously well.

 My open tabs in NetNewsWire have exploded in the last couple of days with really good articles about driving wiki adoption and generally making wiki’s work. First up Making Wikis Work is a pretty good overview of all that is wrong with wikis. It’s odd to think that a technology as young as wikis has legacy cruft but they do.

In particular, WikiWords are no longer required or a good idea, use an editor that makes creating links easy or use a simple shortcut for creating links (the square bracket notation was easy for most people to learn, but you need to provide a completely GUI alternative as well).

Yesterday afternoon Suneth and I took on a research project to see how feasible it was to keep server caches up to date by using XMPP to notify the other servers in the cluster of a change. Imagine a web server with some latency between it and the resources it’s serving (eg: it’s using S3), to speed up performance you’d want to cache the recently used or most commonly used resources locally on the server, but if you need to scale up to a cluster of servers and the resources are being changed, that cache becomes a problem.

Mindy McAdams gives advice about how students should test their online page designs, the trouble is the statistics she’s looking at are lying to her.

You can see that although the screen resolutions larger than 1024 x 768 add up to more people (4,512 vs. 3,524), the single most common resolution in use (among people who read this blog, that is) is 1024 x 768. You can also see that the number of people viewing the site at the old standard, 800 x 600, is quite small.

A while back I took Paul Stamatiou (and by proxy, VMWare) to task about their claim that Cocoa makes them so much more efficient. My take was that it was a Cocoa vs Carbon argument and VMWare employees came rushing to explain that it was actually a Cocoa vs Qt argument. Kudos to them for being in touch enough to join the debate, I had to log a support case with Parallels to get their side. Unfortunately, the point stands that users shouldn’t care what framework an application is used in – I certainly had no idea Parallels used Qt.

There’s a constant argument over whether data should be structured or unstructured in content management and knowledge management systems. The key advantage of structured data is that it’s easier to process and manage – the system can manipulate and report on the data far more accurately. The downside is that it’s more difficult and frustrating for users to be limited to the specified structure so less data tends to get captured and it can be more difficult to get adoption.

If you use sessions to track logins the session key acts as a password equivalent while the session is active. So if anyone can intercept that session key they can masquerade as the logged in user without knowing their actual password. Hence, sessions time out to improve security by giving only a small window that the session key can be used in.  This of course drives users crazy because they have to login again and again.

Flow|State has an excellent semi-rant about how poor the user experience is when using OpenID – both signing up and logging in. In particularly the question of what happens to all your accounts when your OpenID provider disappears is a particularly good one.

It so happens that I was looking into this just today since I needed a user friendly but secure authentication mechanism. OpenID seemed like a natural choice since I was effectively starting from scratch anyway, why not use a standard? The main problem I had with OpenID didn’t really come through clearly in the Flow|State article though – OpenID requires users to log in twice. The first log in requires them to enter a URI, the second log in requires them to enter their password (or in some cases a username and a password). It’s bad enough that most URIs are much longer than most usernames or even email addresses, but there’s actually a page reload between the URi and the password. When was the last time you saw a webapp display the username and passwords fields on separate pages?

I’ve struggled with Lucene before and failed to configure it properly resulting in absolutely horrendous search results so a recent need to implement search functionality wasn’t something I particularly wanted to take on. In fact, I was prepared to do almost anything to avoid delving back into Lucene filters and parsers and tokenizers and “stuff”. This tends to be problematic given that Lucene is the dominate search library – so popular in fact that it’s been ported to other languages.

I’ve been setting up a site for my wife and I to help us keep in touch when we move over to the UK next year and to let us share photos with friends and family etc. Of course this leads to the fun of picking a domain name that makes sense for us and is easy to remember. Going through the list of names we thought up, “thesuttons” was the only one that we liked and was available – but only in a few TLDs.

Just as I’m catching up on Amazon’s web services, they introduce another one – this time aimed at payment processing. My first impression though is pretty underwhelming. The one thing that FPS seems to have going for it is that it is extremely flexible. Most processing systems focus on moving a specific amount of money from a credit card to the sellers account. FPS provides options for combining micro-payments, direct debit (and proprietary Amazon funds) as well as recurring payments etc. In other words, FPS provides a ready made billing department rather than just an order processing system.

The Ephox Weblog pointed me to James Robertson’s comments on Seth Gottlieb’s article, “Homebrew CMS” all of which is good reading. The key part for me is:

Editing environment. If the authors can’t easily and efficiently get their words onto the site, you’re toast. There’s a huge amount that goes into a good editing tool, including table support, CSS, images, spell checking, and clean cut-and-pasting from Word. Even if you chose to use one of the commercial editing tools (a good idea!), it still needs to be tightly integrated into the CMS.

One of the common first steps for scaling up an application is to move the database off to a dedicated server – often followed by having multiple application server instances to handle requests. With a standard SQL database that’s pretty straight forward, with data stored in Amazon S3 that’s not always as simple.

S3 obviously provides a network API, but it doesn’t necessarily provide all the functionality you need from your data layer. For instance if you need to update search indexes you need a central server to track the changes and update the indexes. You may also need synchronization above what S3 provides etc. Whatever the reason, you need to provide a server to handle those data layer tasks and then pass the storage off to S3.

I like the idea of resources being addressable by a simple URL, but I’m having some difficulty reconciling that with resources that are versioned. Getting at and working with the latest version of a document with REST APIs is all pretty straight forward, but how do you retrieve the document history or a specific version of the document? I’m sure this is something that people have already worked out, but all my searching for discussions of it leads to people talking about versioning the API so that things don’t break when you change what operations are available or the data format returned, rather than versioning the resources themselves.

$ unzip ~ephox/vmware-debian-etch-r0-mini.zip

bash: unzip: command not found

Might have been just a little bit picky when I first installed that box….

So we’ve got a new internal system that we’ve built on top of JCR. Currently we’re using Jackrabbit as the repository, but eventually it will be ported over to something like IBM Portal or something like that. Unfortunately, right now we’re deploying the app to a pretty limited server – both in terms of CPU and RAM.

It turns out that using Jackrabbit with the Derby persistence manager in that kind of situation is a horrible, horrible idea. Everything works great on systems with modest amounts of CPU and RAM but once we deploy to that poor little virtual server in the sky page load times skyrocket and the whole thing becomes unusable.

Ken Coar complains about some of the changes in FireFox 2.0 and mentions:

My basic plaint is as usual: when changing the user interface, don’t violate the Principle of Least Astonishment and force the change on the user. Make it the default, perhaps, but always provide a preference option that lets the user keep the old UI behaviour. The user should be in charge of changing his work habits, not the software.

If ever there was a problem that just wouldn’t die it has to be inter-project dependency management. If your code depends on external libraries, it’s pretty simple to pick whichever solution you prefer – either grabbing a version from a repository or checking jars into your source control. However, if you depend on a project that you control, it gets so much messier.

If the projects are small, it’s probably a good idea to just set up the build system to build them all together – effectively making them separate projects even if for development purposes you can just build this bit or that bit and utilize precompiled versions of the dependencies.

That permanent generation had better provide some pretty damn amazing optimizations or I’m going to add it’s inventor to my list of people to track down and torture at next JavaOne. It turns out you can only reload a Tomcat webapp so many times before the PermGen space fills up and everything dies. That’s annoying enough in development but it means I can’t upload a new WAR file to our test server regularly without also bouncing Tomcat. Right now our continuous integration server (Bob the Builder) is doing just that after every commit so the server isn’t staying up all that long.

It took forever, but I’ve finally how to stop Tomcat adding a Pragma: no-cache header to any resources in a secure context. You need to set disableProxyCaching to false, so if you’re using basic authentication you need a valve like:

<Valve className="org.apache.catalina.authenticator.BasicAuthenticator"
  disableProxyCaching="false" />

That needs to go within the definition for your web-app. So, within the META-INF (some sources say within WEB-INF?) create a context.xml file like:

<Context>
<Valve className="org.apache.catalina.authenticator.BasicAuthenticator"
  disableProxyCaching="false" />
</Context>

I found deploying as a WAR didn’t work (it seemed to ignore the context.xml), but deploying the exploded files did work and that’s fine by me.

I’ve spent a fair bit of time writing articles for LiveWorks! lately – posting weekly isn’t as easy as it seems, particularly when you have three weeks leave coming up. Most of the articles include some kind of code, be it JavaScript or Java and an explanation of what the code does, why it does it and how you might use those techniques in other ways. As I go through writing the article trying to explain my code it really highlights how nonsensical my original design decisions are. Even when I’ve gone back over code and carefully refactored it to be clear, when I come to write the article I almost always rearrange things again.

I discovered one of our client’s company blogs1 tonight because they mentioned they’d just upgraded to EditLive! 6. Reading through the backlog I discovered they’ve found a clever way to reduce the impact of form spam that I hadn’t come across before2 – only accept the submission if it uses multipart/form-encoding not just a plain post. Like most techniques it won’t work if everyone does it but it’s another interesting way to differentiate.

One of the challenging practices in XP is daily deployment – it requires your development team to have a very low defect rate and completely automated build and deployment tools. In an off the shelf software scenario it has the additional challenge that you can’t actually get your customers to upgrade every day.

At Ephox we’ve adhered to this practice pretty well in the past by automatically deploying builds to all our internal systems, including our corporate wiki, website CMS and even this blog. Of course, that’s still just a small subset of the kind of environments that we’re actually used in and it’s not actually getting out to real customers. To take the next logical step, we need to make those builds available to any customers who feel like checking out what’s new – maybe not every day, but between all our customers hopefully regularly enough to give us good feedback.

It turns out the root cause of the problems I was seeing with Safari on Windows is that it simply doesn’t support signed applets. That’s pretty sad really. Apple seem to have implemented their own Java plugin instead of using the one that Sun provides (they’re using the Sun JRE via their plugin) and of course it’s missing most of the features you’d want from a Java plugin.

I’ll report it to Apple officially just as soon as someone fixes the bug reporter – it’s currently down. Perhaps there are too many people logging bugs against the new Safari and Leopard builds….

So Apple have released Safari for Windows – really no idea why, but they have. Sadly, just like on Mac it has terrible JavaScript and Java debugging support. In fact, it’s worse on Windows than on Mac – anyone know how to turn the debug menu on? Anyone know how to get a Java console to appear?

I’d really, really like to make things work with it but all I can do at the moment is randomly guess at the source of problems.

Probably the most commented on experience from Mac OS 9 was the fact that once one application crashed, they were all in trouble; generally, you were in for a complete system reboot. At last OS X came along and solved all that, finally providing protected memory and the ability to force quit one application without affecting the others. People rejoiced as they left their computers running for days on end without needing reboots and uptimes soared.

Opensource projects tend to have a well-earned reputation for being poorly documented. Originally this was caused by the fact that most of the people who knew the codebase were developers who didn’t have the skills or interest in writing the documentation. With opensource becoming more mainstream and more people getting involved I think that reason, while still having an effect, has become less of a problem. Despite that the quality of opensource documentation hasn’t improved – if anything it seems to be getting worse. More and more projects don’t even have half-decent reference information and just a few scraps of information on where to start.

Where is it? Clearly I’m missing something here. There’s a wiki with some good getting started overview stuff and some other chicken scratching but I’m yet to find an actual reference telling me what’s actually allowed in struts.xml. Shouldn’t that be pointed to in a big neon sign? The struts.xml page on the wiki isn’t exactly comprehensive and while the DTD is listed on the examples page it’s not exactly commented..

Oh and did I mention that wiki’s are a horrible way to write documentation?

Robert Scoble posts about saving Pandora:

…Tom talking about how the business of Internet Radio is under pressure due to coming changes in how the music industry wants Internet Radio stations to pay for the distribution of music (basically the costs will triple, if the proposed changes go into effect). After we talk about the challenges that Pandora’s business faces if the fee changes go through Tom gives me a preview of their new Sprint/mobile service.

So for the record, if you have a textarea in Safari that is set to display: none; You can’t set it’s content via JavaScript unless you show it first. So:

edCanvas.style.display = "inline";
edCanvas.innerText = body;
edCanvas.style.display = "none";

Really annoying. For those Ephox people using the latest version of our EditLive! WordPress plugin in Safari, that’s where your content went. Pull an update from subversion soon and the issue should be fixed.

I’ve spent the day doing some work with our Alfresco integration to make it easier to install and a bit more maintainable. Fortunately the Alfresco team were kind enough to point me at their Module Management Tool which while being a bit rough around the edges is really quite cool. Basically, it takes the alfresco.war and a zip file containing the module you want to install and shoves the module into the war file, ready to be deployed. What’s particularly cool is that it actually has version management so it knows if it’s previously installed a module or if the installed module is more recent than the one you’re installing etc.

With a trip to the US coming up, I needed to set up a way to access my work email from my OS X laptop. Last time I went I just used Mail.app to access the Exchange server via IMAP and it worked ok, but obviously had no calendaring ability. This time round I really need the calendar to work as well so I fired up Entourage.

Going through the set up was a bit tedious and I thought it pretty dodgy that it wanted the Outlook Web Access (OWA) URL – obviously it goes via that instead of the normal exchange protocols. As it turns out though, that’s the way that the Exchange server hosting company recommends you access your email. In fact, the instructions for setting up Outlook on Windows include a series of complex maneuvers to get it to use OWA too. Indeed, Entourage seems just as capable a client as Outlook at least form early impressions. The only thing I miss is the SpamBayes integration and that’s not really a feature of Outlook.

Back in the days prior to OS X, I used to run Linux on the desktop (dual booting with OS 9) and used Window Maker. Since then I’ve used Linux a fair bit but only on servers – I have all the UNIXy goodness I need in OS X. I’ve just looked into setting up another Linux desktop system to see how viable it is and found that Window Maker still hasn’t made it to a 1.0 release – in fact, it hasn’t seen an update since 2005.

It seems every time I do some work integrating EditLive! I find a new technique that makes it easier, simpler and more future proof. It’s not so much inventing new patterns as finding creative ways to apply them.

For instance, tonight I wanted to get the upload manager in WordPress working with my EditLive! WordPress plugin. There’s a whole bunch of logic and implementation detail in the JavaScript that WordPress outputs and it expects the editor to be TinyMCE. Fortunately, earlier on the plugin had drugged TinyMCE, taken it out back and shot it1, so it wasn’t available. Fortunately, it’s pretty straight forward to create an adapter for the TinyMCE runtime api that works with EditLive! and drop it in by assigning it to window.tinyMCE. Simple stuff and very similar to techniques I’ve used in other situations, but extremely powerful and easy to not think of. In fact, I’d tied into the autosave by manipulating JavaScript classes and all that became redundant now that I have the TinyMCE adapter.

Just for my sanity, before an OutOfMemoryError is thrown, the garbage collector should do everything it can to free up more memory right? What I’m seeing now is that the JVM does a partial GC, throws an OutOfMemoryError and then decides to actually do a full GC which frees up enough RAM to get things working again.

Very annoying and I’m not sure what I can do about it. There are definitely no remaining references, but I still can’t get that memory back.

If you’re doing some graphic work in Java that you want to look pretty (as opposed to most UI stuff where this doesn’t make any real difference), you probably want to make sure antialiasing is on for the graphics object. By default Java usually honors the OS settings for antialiasing, particularly for text, but you can get much smoother antialiasing by setting the appropriate rendering hint.

For example, originally the new progress bar I added last week didn’t have antialiasing on, but in the tweaks I’ve done this morning (now live on our internal systems) antialiasing is on and it looks much better. It’s also changed to be more dots than lines since the last post too. However, setting the antialias rendering hint for all the components in the editor simply slows everything down (rather dramatically) and doesn’t actually improve the appearance anyway. Definitely a setting to be aware of but be cautious in its use.

I’ve spent most of this week working on making the start up experience for our applet better. It should look better, take longer and most importantly feel fast to the user. The first step is obviously to profile and optimize – the faster the start up time actually is, the easier it is to make it feel fast for users. There were a few things we could do to help out there, but nothing that really made a dramatic difference.

It’s amazing the amount of work that can be required to get from something that is perfectly wonderful for internal use to something that’s ready to be distributed to the public. For quite some time, we’ve been using a custom made WordPress plugin to allow us to edit blog posts with EditLive! It’s used on three blogs within Ephox, two blogs on this server and I’m pretty sure Rob’s using it too so it gets quite a workout. It’s stable, feature packed and all-round wonderful.

So everyone knows Google has a vast amount of data about you and that Google tracking cookie that basically never expires. Now they own Double Click, completely with it’s own tracking cookie that tracks you on pretty much every site on the internet.

Essentially, Google now know what you search for, which links you follow and now between Google AdSense and Double Click, they probably know everything you do on that site pretty much regardless of which site you pick.

We took delivery of a brand new, shiny Dell preloaded with Vista today and I set about setting it up as a pairing station. Sadly, in the first 30 minutes I’d crashed Vista to the point of having to hard reset the entire machine.

Finding anything in the control panels is a nightmare, not just because it’s impossible to guess where things will be, but because it takes so many more clicks to get there now. Worse still, when you finally do get there, you wind up looking at the old XP dialog. What’s the point of all the fancy graphics and categorization if when it comes down to it I’m still confronted by an ugly, confusing dialog with a million tabs and settings coming out it’s ears?

I’ve reached a point where I really need a logging framework for a small library. The trouble with logging in small libraries is that you want to avoid using a full logging system like Log4J or java.util.logging so that applications that use your library are still free to pick the logging system they want to use. Jakarta Commons Logging has been the de-facto solution for these situations for quite a while now, but it’s a library that people love to hate because of “class loading issues”. Now I happen to know a fair bit about class loaders, I know why the problems occurred with Commons Logging, I know how to avoid them, work around them and I understand why they’re not a fundamental flaw of commons logging. However, I also know that commons logging is a very simple logging framework and there is scope for a project to build on the basic idea of commons logging but provide more power and flexibility rather than just the absolute lowest common denominator functionality.

I’m going to show my ignorance of the actual differences between URLs and URIs here, but I was a bit surprised by the fact that java.net.URL didn’t extend from java.net.URI. Along those lines URL.toURI() suggests that some URLs can’t be converted to URIs. Is this just in the context of the URLs that Java previously successfully parsed or is this a generic constraint?

My main reason for asking is I’m trying to determine, when implementing an HTTP caching library, I should be using java.net.URI or java.net.URL to identify the source of resources. My original thought was to use java.net.URL because that’s what pretty much everything uses and besides I’m used to support Java 1.3 (URI was only added in Java 1.4). Somehow that seemed overly specific and I should use URI instead. Partly this is because it felt more generic and partly because it just seemed to have more geek cred. I must admit though I really don’t have a firm grasp on which I should use when and why. So dear lazy web, can you help?

One of the new features in EditLive! 6.1 that we released today is a plug-in architecture that handles deployment of extensions to the editor. Plug-ins in Java apps are pretty common these days, but there aren’t all that many applets that have them so I thought it would be worth documenting some of the challenges we faced and how we overcame them.

The biggest differences between plug-ins for applications vs applets are:

One of the things we’ve been wanting to do for ages was automate our release process so that with the click of a button we’ve deployed a new version out to customers. Today, at least for EditLive! itself, that became a reality, with the “autodeployer” being tested out with it’s new release capabilities.

At first glance it looks like releasing a commercial product like ours would be really straight forward, but there’s a surprising amount to it. No one step is difficult, but there are a lot of different systems involved from our main web site, to our source control, build machine, release notification system, integrations, demos etc. As usual for automation projects, the biggest challenge is working out exactly what you are doing manually. Finding the right way to automate it tends to be pretty simple once you understand what you actually do.

Most issue tracking/customer relation/todo list things have a concept of resolving an issue for later or a remind me later option. The idea is that you don’t want to or can’t deal with the issue straight away, but you need to come back and review it or follow up later.

Unfortunately, it turns out that marking an issue for later tends to mean “make this disappear so I have no chance of remembering it” because the issue disappears from all the open issues lists forever. Many systems have an option to e-mail you regularly about issues that are marked as remind or later, but the e-mails are hit and miss – usually they remind you about the issue too early and you just get used to ignoring them.

There is an awful lot of money spent on user interface research, carefully tracking what users do with an application and trying to find ways to improve based on that. It’s a shame that so much of it is wasted because the captured data is misinterpreted.

The Office 2007 Ribbon is a classic example of this, it was clear that Microsoft had real world data to back up their decisions about the Ribbon, they’d spent millions on it. Yet somehow it just didn’t seem right to me. Turns out at least Damien agrees with me. It turns out that despite the fact that usage data shows that users work in different modes, designing an interface that reflects those modes isn’t ideal.

We’ve been looking at ways to improve the experience for end users when applets first start up. It’s unfortunate that the worst experience with applets is always the first one since that’s when the JVM has to start up and the applet has to download. Once all that happens subsequent usage of applets tends to be lightning fast – particularly with the latest JVMs.

Sadly, that awful Java coffee cup graphic just doesn’t make users happy while they wait for the applet to download. Equally sadly, there’s no good option to get rid of it. You can specify an image of your own to load, but then it replaces the progress bar and it can’t be dynamically resized to fit the applet. Heck you can’t even center it. Worse yet, by the time the graphic downloads and displays the applet is just about ready so the user winds up seeing an empty box for a while then a brief flash of the image and then the applet’s ready.

I’ve been interested in Mindquarry for a while now, so when they finally released a version you could download I headed straight over and grabbed a copy. Sadly, the copy I grabbed, advertised as for OS X, is actually a generic package for which there are no installation instructions. The instructions that are provided for installing Mindquarry all talk about executing ./bin/mindquarry – which would be good if there were a ./bin in the generic package.

So I need to improve out caching support and I really don’t want to got through all the pain myself if I can avoid it. Does anyone know of an existing library that just handles client-side caching with the appropriate If-None-Match and If-Not-Modified headers etc. We already have libraries for doing the actually HTTP stuff, I really just need something that knows how to store things on disk and tell me what headers to use to make the request conditional. Oh and we’re an applet so small and without dynamic loading or configuration files would be ideal.

If you happen to be running a VMWare Server instance on a machine with multiple network cards, make sure that when you upgrade you reconfigure which network cards your VMs should use, otherwise you may find that despite the fact that the network interface in the VM is up you have no network access. It usually reports “No route to host” or “Destination Host Unreachable”. Sadly, VMWare doesn’t seem to be too bright about how it picks network interfaces and will sometimes pick one that isn’t connected to anything.

So WordPress let a nasty hacker modify their download and I’ve got to go around upgrading everything again. Now upgrading WordPress isn’t difficult except that every time you do, it puts back all the useless crap on the dashboard that I remove. I have a perfectly good feed reader, I don’t need my dashboard showing me old posts from the WordPress development blog and irrelevant posts from Planet WordPress.

With one blog that would be bearable, but I’m currently maintaining five WordPress installs and I’m sick of it. So after way too long I’ve cottoned on to the fact that the index-extra.php file that causes all the problems is unlikely to change very often and I can probably just store a patch file and then quickly reapply it. That patch is here if anyone else wants it. It simply replaces the downloading and parsing of the RSS feeds with creating an empty array and it removes the mysterious “partner=wordpress” from the end of the call to Technorati to get the incoming links. Why is that there? If there were an upfront and honest explanation of it I probably wouldn’t mind but I really don’t like secret partnering deals going on in software that I run locally1.

Many people think that because Java has garbage collection, that memory leaks aren’t possible – this is totally and utterly wrong. One really good way to introduce leaks into Java programs without blatantly holding on to object references is to create a new thread and then not use it. Once you create a thread, it is added to the list of runnable threads. I’m not sure why, but even before the thread is actually started, Java treats it as a running thread and holds on to it. Obviously threads that are currently running shouldn’t be garbage collected, but it seems like an unfortunate side effect that newly created threads also can’t be garbage collected. Once you know about it, the solution is simple – only ever create a thread immediately before you call its start method.

For a couple of years now, Ephox has been very successfully using a wiki to provide communication within the company and helping to bridge the gap between our two offices on opposite sides of the pacific. Central to the success of the wiki has been the successful integration and configuration of EditLive! as the editor. It certainly helped that we have a high quality editor with lots of attention to detail but the most important aspect was the amount of attention we paid to correctly configuring the editor.

You’ve got to hand it to the Atlassian team – they are seriously clever. Wikipatterns.com launched recently to give people a place to go for advice on improving wiki adoption. The content is very useful to have in one place even if what’s there right now won’t be new to most people using wikis. It is undoubtedly useful and Lars Trieloff from the Mindquarry team has already linked to it (Mindquarry develop a soon to be opensource wiki that’s integrated with a bunch of other cool looking stuff). I’m sure other wiki vendors will be linking to it over time as well.

According to our stats, yesterday I deleted over a thousand lines of code – sounds like a productive day to me. Hopefully today I’ll find another thousand lines to delete. It’s amazing how unused methods can build up without you noticing. Eclipse will tell you about unused private methods, but not about public ones that aren’t used anymore. Most of them had unit tests even, just that the entire functionality had become unneeded. There’s probably some duplication in there as well that will let us delete more lines of code.

Scoble linked off to Blogwerx’ new plagiarism detector today which is awsome publicity for them and in most situations would be a sensational boost to a new or newish product. Unfortunately for Blogwerx, their system is completely non-functional, not even the help button does anything. Nor does the all important, money making upgrade button. There’s simply no excuse for releasing software that is this badly broken to the world.

I just hope they didn’t pay out the $30000 to speak at Demo or if I were an investor I’d be seriously annoyed at them wasting that much money without having a product ready to actually sell. It sounds like the CEO may have just sidled up to Scoble without paying to do a demo so they may be lucky.

Hey Scoble,

Your blog ate my comment. Most likely Akismet flagged it as spam incorrectly. I must admit, I’m not comfortable with the design of Akismet – allowing everyone and anyone to decide what is spam and what isn’t seems to be asking for trouble. That said, I have taken to using black lists for email recently. In that case though I’ve been careful about which blacklists I picked and checked that they all had simple systems for getting back off the black list. Akismet doesn’t seem to provide any information on how they decide a comment is spam or provide anyway to protest against it.

A new release of Java, a new wave of people saying they won’t buy Macs unless Apple get their Java update now, dammit now! John O’Conner joins the chorus. Of course, there’s no mention of why he needs Java 6 instead of Java 5, just that he has to be up to date with the very latest. Last time round at least people were eager to use generics and other new language features, but there’s really not that much that changed in Java 6 outside of a few specific cases.

I’ve said it before, and I’ll say it again – the best syntax you can use for your wiki is HTML, or probably better XHTML. It’s a defined standard with a wide range of excellent WYSIWYG editors, it’s well known, very portable and saves you from having to write a converter to display your wiki pages.

David Van Couvering is the latest to complain about incompatible wiki syntax and he’s certainly not alone in his frustrations. People don’t want to have to learn yet another crazy wiki syntax, they want an intuitive interface that just lets them write their content. HTML provides far more flexibility than any wiki syntax, particularly around tables, which makes presenting data a lot easier.

I was quite interested in a few of the new features listed for WordPress 2.1, since they look like they could be quite useful to build into the plugin I have to use EditLive! as the editor for posts. In particular

• Image and thumbnail API allows for richer media plugins

sounds very interesting, since I’ve not gotten around to doing anything much with images and EditLive! has a lot of support for media that is currently going to waste. Sadly, browsing through the WordPress documentation lists nothing about how to actually go about using these wonderful new APIs. In fact, it doesn’t mention them at all – it seems the documentation is still for the 2.0 release.

In What does No Java on the iPhone Mean? (and the follow up More on Java and the Mac) James Duncan Davidson looks at why Apple haven’t added Java support to the iPhone. In particular he claims there’s no killer application made with Java. In the strictest sense, that’s probably true, but there is a killer category of applications which are almost exclusively made with Java – games for mobile phones.

It’s this popularity of Java in the mobile phone world that makes the lack of Java on the iPhone seem so odd to me. I can understand Apple wanting to have complete control over the iPhone interface, and I’ll concede that most of the existing games for mobile phones probably wouldn’t translate very well to the keypadless iPhone, but it will be interesting to see if Apple can satisfy the great desire for cool little mobile games that today’s teenagers, a key market segment for the iPhone, without leveraging the existing knowledge mobile games developers have in Java. If the iPhone takes off it won’t be a problem, similar to how there are a huge range of iPod specific accessories, there will be – if and when Apple make it possible – a huge range of iPhone specific games and add-ons.

I’ve always considered Technorarti Tags to be annoying clutter on pages because people are far more likely to search Google for a keyword than they are to look up a particular Technorati tag and the million slight variants of it. I’ve just noticed however, that you don’t actually need to clutter up you page with unsightly tags to get into the tag list. Take for instance the Ephox technorati tag. I don’t tag my posts, and yet my Oh Happy Day! post is the top entry.

I’ve been playing around with OpenID on the few occasions I’ve had internet access over the Christmas break and have it set up, but not ideally. Currently, everything points to http://www.symphonious.net/id/ as the OpenID URL, but server and delegate. Ideally I’d like my identity URL to be just http://www.symphonious.net/ and the server ad http://www.symphonious.net/id/ but I’m not familiar enough with the whole process to make it work.

Mostly I’ve followed Sam Ruby’s instructions, but the process was done in a few different sessions following a few other explanations of OpenID, but I think it matches what Sam describes.

I found an unexpected source of frustration while getting LiveWorks! online. The site is primarily an instance of WordPress1 but the mailing lists are handled by Mailman and the archives by Lurker. All three of them are simple to customize their outward appearance, but all three of them use a different template language. So now instead of having one site design, I have to have three very similar looking, but completely different design implementations2. Every time we want to update the common parts of the site, we have to do it in multiple places in completely different ways.

I’ve long been a proponent of UI designers making decisions about what the best way to do something is instead of just providing configuration options to the user – after all, if you are a fully trained and experienced UI designer, shouldn’t you know better than your completely untrained users? Seeing some of the discussion going around about Joel’s criticism of the Windows shutdown menu, including Arno Gourdol’s comments on the Mac OS X shutdown options and they both seem to being taking it a little bit too far.

Robert Scoble:

I’d love to know what you think? Does any of the Office 2.0 vendors have a chance to edge in on Microsoft’s market?

Edge in – sure, it’s been happening for at least the past 5 years or more. I’ve lost count of the number of times we’ve sold EditLive! to companies who were replacing Word to make life more pleasant for users. Notably though, these aren’t situations in Word’s core target market – creating documents destined for print. Word has picked up a lot of market share in all kinds of weird and wonderful content creation scenarios that it wasn’t designed for and it’s picked up features to make it work quite well there. Despite that, these fringe areas are ripe for competitors to specialize in and provide solutions that fit better with the user’s intentions.

So Microsoft has decided to specify licensing terms for anyone who wants to develop an Office 2007 style UI. I can’t help but think that this is somewhat hypocritical considering Microsoft was the beneficiary of a rather important case against Apple regarding copying of user interface ideas. I also find it odd that Microsoft is using licensing techniques to enforce the way the ribbon and similar ideas work instead of just making the actual component implementations available to everyone thus guaranteeing that they always work the same way (including in future updates).

There’s a somewhat inevitable drawback of ideas that work out well – they continue on much longer than you anticipated. Such is the case with the mess of AppleScript, UNIX commands and Excel spreadsheet that makes up our engineering statistics tracking system. It tracks bug counts, test coverage, velocity etc over time and produces a bunch of graphs that we can look at and try and work out what they mean. It’s by no means mission critical but it is interesting to see what’s going on from day to day. The trouble is, now that we’ve come to like having those graphs, the fact that this mess of scripts can only actually run on my laptop is something of a concern.

A while back I complained that the FireFox 2.0 installer didn’t include an actual link to the Applications folder. This morning I saw that this wasn’t just a theoretical problem, nor was it just a problem for “stupid users”. One of our engineers, who is very bright and good with computers but with no real Mac OS experience, had to install FireFox 2.0 on one of our testing Macs. He dragged the FireFox icon onto the picture of the Applications folder. The lack of text made the problem even worse – he wasn’t familiar with the Applications folder icon on OS X so didn’t realize it was meant to be representative of the destination instead of being the destination in and of itself.

I installed the final version of Office 2007 this morning and when the installer finished I was actually quite impressed – it was the first Windows install I’ve done in ages that hasn’t taken the liberty to install a shortcut in my quick launch bar. Sadly, Outlook is one of the very few programs that I actually want in my quick launch bar so I went and added it myself. I thought, I’ll have to blog that – “Thumbs Up For Office 2007 Install”.

James Robertson mentions the importance of a good authoring environment in CMS solutions.

Considering that the primary purpose of a web content management system is to help staff to write and publish content, the editor has to be front-and-foremost when it comes to selecting a product. And yet, many organisations specify little more than “the CMS must provide a web-based WYSIWYG editor”.

I also found Seth Gottlieb’s comments on the subject interesting, particularly:

This weekend I switched from DSpam back to SpamAssassin because of DSpams high false positive rate and my dislike of having to review all the spam it caught constantly. While doing so, it occurred to me that the way I’ve set up my anti-spam solutions is really pretty cool. I’ve essentially set up a partially self-training system with a user-friendly interface for providing feedback via Mail’s Junk mail button.

SpamAssassin is set up to run via procmail and dump any spam into the folder that Mail.app uses for its Junk mail. Mail.app is also set to move any spam it detects into that folder so there’s a two layer spam filter in place to move everything into that one folder for review.

Anthony Towns:

Writely, the web-based word processor, was kind-of interesting, but in the end didn’t work for me. The potential killer feature for me would’ve been SubEthaEdit or Gobby -like interactive collaboration, which seems like something Google ought to be able to do with their whacky AJAX techniques. Unfortunately, it seems to just be some sort of automated merge-on-commit, which does nothing for me.

I believe Writely’s attempt at real-time collaboration is a little more advanced than merge-on-commit, but as far as I know it is based around diffing instead of actual change tracking. The trouble with this is that it makes it nearly impossible to preserve the user’s intention rather than just the effects of the operation. It’s certainly easier to get a system up and running using diffing than using actual tracking of changes, but the results just don’t sync up with what users expect as much and I suspect Anthony’s experience reflects that.

I’ve been trying to get the dates in my comments feed to show up correctly with all the right time zone stuff and so on, but even though I’m pretty certain the feed is reporting the right time and time zone, NetNewsWire seems convinced that the comment was made in the future. So can someone confirm for me that the date 2006-10-25T20:28:58+10:00 refers to 8:28pm (and 58 seconds) on the 25th of October 2006 in a time zone that is +10 hours from GMT. Thus if my time zone is also +10 hours from GMT the date should mean 8:28pm local time on the 25th of October 2006.

I’m in the middle of doing platform testing for EditLive! 6.0 which involves setting up and testing a huge array of OS, browser and JVM combinations to run through and manually verify that everything is working as it should be. Normally this is hard enough to do, but at the moment it seems every tech company has either just released a new version or is about to. We’ve got FireFox 2.0, IE 7, Windows Vista beta, OS X Leopard beta, Java 6 beta, OS X Java 6 beta, Office 2007 beta and who knows what Linux and Solaris is going to do to me.

The back button has been a great challenge for a lot of modern web applications and WYSIWYG editors are certainly no exception. Way back before I can remember, someone had the fantastic idea of preserving content entered into text fields and restoring it if the user hit the back button to return to the page. Unfortunately, with the advent of WYSIWYG editors, this was generally lost because the browser reruns the JavaScript in the page again, providing no indication that the user was returning to the page. I could be wrong, and I’d love to hear of other examples, but I don’t believe there is a WYSIWYG editors available today that preserves content when the user hits the back button. I find it somewhat interesting that despite the huge amount of user angst this causes1, there doesn’t seem to be a lot of interest in solving the problem.

I haven’t fully investigated all the details and combinations, but it appears that if you use the cache_archive_ex parameter to specify the Jars for an applet and at least one of them has a specified version, any that don’t have a version get ignored. So for instance, if you have cache_archive_ex=“editlivejava.jar;6.0.0.1,footnotes.jar” editlivejava.jar will be downloaded and used correctly, but footnotes.jar will be ignored.

Additionally, any jars specified in the cache_archive parameter get ignored (probably unless they have a matching version in the cache_version parameter).

You know, until just now I’d always thought that getting the JRE installed was a 10 minutes process or so. It turns out that this computer didn’t have the JRE installed so when I went to write a blog post1, I got the missing plug-in box and the instruction to click to install the required plug-in. So click I did and in under 2 minutes not only is the JRE fully downloaded and installed, but the page has reloaded with EditLive! fully up and running, ready for me to write this post.

To Jonathan Schwartz,

The Ephox engineering team saw the really cool work you’ve done on project blackbox and we’ve come up with a marketing idea that would help you show just how portable blackboxes really are.

In essence, the idea is to take a blackbox on a travelling road show around to Sun’s potential clients to show off what it’s capable of. The trouble is, there’s no point in just dumping a container outside their door and saying “cool, huh?” – it’s got to actually do something cool when you put it there. That’s where we come in.

We’re coming up to that point in development again where we need to do up a change log for a release. Ephox has never really gotten this process down pat so it inevitably leads to trawling through a back log of subversion commit logs and bugzilla reports. It always feels like there should be a better way to do it, but it’s always too easy to forget to add something to the change log if you try to keep it up to date all the time.

After much hard work and gnashing of teeth, the beta of our next major release is finally out, including the new productivity pack and track changes. The internet Gods tried to stop us by cutting off our oxygen, er internet supply but the files finally made it across the pacific to our main servers. You can check it all out in the productivity pack section of Ephox.com. It’s a real beta in that we haven’t done any real testing on it and so it’s likely to have stability problems that we’ll iron out before the final release. We’re very keen to get feedback on the release, both in terms of any bugs you find and in terms of how to improve the UI, extra APIs to add for integrating it into backend systems etc.

We’re looking to move most of our servers into VMWare so that we can easily back up the entire system and restore it in case of hardware failure. We’ve moved our Jabber server and now buzilla into VMWare and the next most likely candidate is our Subversion server. I’m wondering if it makes sense to store large amounts of data in the VM directly or if we should look at putting the actual data store on the host machine, but all the configuration stuff in the VM.

One of my keyword watch-lists pointed me to Matthias Ernst’s entry, A long term suspicion:

An observation: almost all WYSIWIG text editors suck at some point. We’ve been beaten up repeatedly for our CMS’s text editor but in comparison we’re actually doing pretty well.

Writing a good WYSIWYG editor is hard. Most people think it’s a trivial task until they actually try it and start getting user feedback about a million different things that they never thought about. That’s also why there are so many bad WYSIWYG editors out there – people start them thinking it will be easy and then find themselves in over their head, unable to keep up with the flow of bug reports, or simply sticking their head in the sand and complaining that the users just don’t get how to use the product.

We’ve been tracking a handful of metrics that we wanted to track about our code base for a while and it’s built up some pretty graphs that tend to indicate that our adoption of XP has improved the quality of our code. Obviously they’re just a guide, the real test of quality will happen when we ship the next release out to clients, but they’re useful none the less. The general trend shows a gradual improvement over time – fewer known bugs, lower complexity, more tests etc.

There is an interesting tendency in software development to try to keep the internal model as closely in sync with the output format as possible. When you control both, that’s probably not a bad idea – it simplifies serialization. What is important is that the user interface matches up precisely with the user’s mental model of how to use your software. Attempting to keep your model simultaneously close to the user’s model and the output format often forms contradictory goals.

A while back I saw a comment go by that raised my interest, it was something along the lines of: we know all our users want a better editor, but it’s just not something we can charge more for. My recent experience with Yojimbo and what I’ve seen of a number of people evaluating blogging systems, content management systems and anything else content-centric, tends to indicate that even if you can’t charge more, you’ll definitely sell more.

If you happen to run a lot of JUnit tests in one go with Eclipse, you may find that towards the end they start to slow down or throw OutOfMemoryExceptions. They do that because the Eclipse JUnit plugin creates a new instance of the TestCase for each test method, calls setup, test method, tearDown and then for some reason keeps the object around. This means that if your tearDown doesn’t null out all the fields in the test class you can wind up with a whole lot of wasted memory.

In most software projects, there are a few areas that are just really complex to get your head around. It’s not something that refactoring can solve, the underlying concepts are just difficult to work out. Fortunately as software developers, we can learn that information once and encapsulate it in code that will understand it for us and even apply it in new situations that we didn’t originally consider.

Recently, we’ve developed a really neat, simple example of this – the PushAndPopCalulator. A surprisingly simple class that determines the appropriate values for popDepth and pushDepth when inserting in Swing text components. The push and pop depth is a way to manipulate the element hierarchy of styled documents to precisely determine where the inserted content works out. Once you understand the details, it’s a reasonably simple concept but working out exactly what values you need for a particular insert requires a break in focus on the actual problem to divert into working out details of using the underlying APIs. It takes a little more time to generalize that to cover all the possible situations you might need.

While I’m waiting for Eclipse to do an update: What’s the one value you can pass to Math.abs(int) and get a negative value from?
Just ran into a bug caused because of it – fortunately it was picked up by the tests even before I checked in.

I find it somewhat amusing that Scoble is discovering the enterprise market at the SAP conference. One quote in particular stuck out for me:

SAP is no Web 2.0 business. The cool kids like Mike Arrington don’t follow its every move like, say, the way we follow Google or Microsoft. On the other hand, name the business and it probably runs on SAP.

Ephox is like that – tiny little company that no one seems to have heard about, but you name the big enterprise CMS and we’re probably in it. We have a client list that most companies would kill to get their hands on and unlike most Web 2.0 businesses – we not just burning VC.

Somewhere along the line I stumbled across Mark Pilgrim’s description of how the universal feed parser sanitizes HTML. A key part of this is that the universal feed parser strips styles because IE on Windows can wind up executing part of that style as JavaScript.

While obviously at the moment this how to be done, it seems completely unreasonable to me that any content that wants to be syndicated accurately needs to avoid CSS entirely. It seems to me that rather than stripping style information, we should be pressuring Microsoft (and any other affected browser vendors) to fix the browser so that it doesn’t ever treat information in CSS as executable code.

A lot of projects these days have taken to using a wiki as a way to get the community to write the documentation for them. This appears to work really well with a huge range of pages being created telling you how to do all kinds of stuff. Unfortunately, for anyone who actually needs to learn about the project, these pages are about as useful as tits on a charging bull.

Why is it that a hugely database dependent framework, that’s meant to be extremely quick to get up and running with is so infuriating when it comes to get it actually talking to the database? I know cross-language interfaces are always difficult, particularly when you try to make them work cross platform but if I can get Java, PHP and perl all talking to MySQL easily, why does it have to be so damn hard for ruby?

I realized an unintentional side effect of deciding to not use the ternary if (a ? b : c) – there’s a bunch of code that we don’t duplicate that we otherwise probably would have. In particular, when working with views there’s a very common pattern to convert between a Shape and a Rectangle1:

Rectangle bounds = shape instanceof Rectange ? (Rectangle)shape : shape.getBounds();

The reason for this is that getBounds() on a Rectangle will create a new Rectangle instance so if you happen to be getting the bounds a few thousand times every time you go to paint the text pane, you generate an awful lot of useless rectangles. Since every shape passed in is in fact a Rectangle, you can eliminate all of that junk without risking a ClassCastException if at some point you do get a different type of shape.

Quite some time ago I talked about redefining and rewriting handler functions in JavaScript, but what if you want to add or redefine methods in an existing class? JavaScript classes1 are based on prototyping so the model is somewhat different to actual object oriented languages. Here’s one way to define a class, MyThing:

function MyThing {
  this.doStuff = doStuff;
}
function doStuff() { }

Note the use of function assignment in there, just like for handler functions. This is one way to do classes in JavaScript, and it’s not really the best – once you get your head around prototypes you’ll probably find that using them directly is much clearer. There’s plenty of tutorials around the web on JavaScript prototyping so I’ll leave explaining it to the experts.

So Google, the company that does no evil, seems to be learning a few tricks from the evil empire of Microsoft. With the release of Google Checkout, Google have effectively implemented Microsoft’s failed passport initiative but with a purchasing spin. Most of the similarities I’ve seen compare it to PayPal1, but my first impression was that this is single sign-on for web stores and that’s more along the lines of passport than PayPal.

Jensen Harris has an in-depth look at the customization choices made for Office 2007 and why they made them. I’ve always been a proponent of getting the UI right in the first place and only providing limited or no customization abilities. Mostly this stems from Raskin’s The Humane Interface:

The central point of this issue is that if we are competent user interface designers and can make our interfaces nearly optimal, personalizations can only make the interface worse. Therefore, we must be sparing and deliberate in offering user customizations. If a user can, by a few judicious choices, really improve the interface, we probably have done a poor job.

In general, Java classes are lazily loaded into the system, so if you have something like1:

if (System.getProperty("java.version").equals("1.4")) {
    Java14OnlyClass.doStuff();
} else {
    AlwaysAvailableAlternative.doStuff();
}

The code will have to be compiled on Java 1.4, but will actually run on Java 1.3 without throwing a ClassNotFoundException. This trick however can be dangerous, there isn’t any guarantee that class loaders will always use lazy loading and even with current class loaders, there are occasionally some surprises.

I think this is the final episode in my series of responses1 to Alastair’s Responding to Adrian. What’s the aim of diffing HTML, how hard is it and how do you go about it?

The aim is really important to identify. The most common and most useful aim that I see for diffing HTML is to be able to show users what changed between two versions of a document. Since the management of most content is centralized2, this equates to showing the combined changes in each version between the original version to compare and the final version to compare. If you’ve ever wanted to see what’s changed on a wiki page, you’ve wanted this type of diff. If you’re sending Word documents back and forth between people you probably want this type of diff too.

Continuing with my response to Alastair’s Responding to Adrian, let’s take a look into the problem with invisible formatting tags in WYSIWYG editors.

The example I gave was to delete the final character of some hyperlinked text. The delete operation removes the internal formatting tag, and hence removes the hyperlink entirely, as well as the final character.

In this behaviour Outlook is no different to many other HTML editors, and is a completely appropriate example. The problem of the invisible formatting goes directly to the heart of the limitations of WYSIWYG editors. There is no visual representation of the tag, so there are bound to be some surprises when the user starts editing in the vicinity of the tag.

Jensen Harris has a post about how they tested out removing the ability to double click in the left corner of a window to close it, but that people couldn’t get used to it so they’re putting it back. Might I suggest that people didn’t get used to it mostly because Office was the only application that didn’t allow it?

If you want to really test out whether or not people could retrain themselves (and they can, even if they complain about having to) you’d need to remove the functionality from all the windows, not just a few of them. Otherwise, the other windows preserve the user’s habits and it just becomes really frustrating that the interface is inconsistent.

I love the fact that most OS X software is installed simply by dragging it to your Applications folder but there is one thing that really, really annoys me and I just can’t understand why developers who are clever enough to write useful software are so utterly incompetent that they distribute their software on a dmg that hides the sidebar in the finder so you can’t see your Applications folder. For instance, here’s the window that opens automatically when you mount the dmg for Microsoft Messenger:

We took a little time this morning to set up Jabber notifications of build failures since we now have a dedicated pairing machine with no email account (plus the sooner you find out about a failed build the better really).

It’s all pretty straight forward to set up, the latest CruiseControl includes a jabber publisher so you just put the details in an it all happens. I’d suggest you put it inside an onfailure element to avoid getting a message every time a build occurs (and if you’re checking in regularly that will be often).

Since its inception, our IWWCM integration has been a bit of a pain to install – the instructions differed based on which version of IWWCM you where running and which version of WebSphere or Workplace you used and you had to move and edit files in various locations. Nothing difficult but it was just too easy to forget a step and have things go wrong.

The simplest way to improve things – provide an installer that automates the process. The problem is, most installer tools are ridiculously complex to work with because they attempt to provide a way to do anything you could ever want. It would have taken us days just to get our heads around how to use the tools.

I’ve come to realize that there is a world of difference between requirements for content authoring and requirements for site design. This really becomes clear when I look at the different view points between myself (The Challenge Of Intuitive WYSIWYG HTML) and Alastair (This Is What You See, This Is What You Get and now Responding to Adrian) regarding WYSIWYG editing. (Snide comment: see, I can use cite and emphasis too).

Ah the synergies, Sam Ruby talk about getting MathML supported in planet, and the Ephox weblog talks about getting EditLive! for Java to automatically upload image versions of MathML equations so they render in any browser. The nice thing about this function of EditLive! is that you can still just double click the equation image to edit it again – the MathML is stored as an encoded attribute so that information isn’t lost.

Sometimes you write code that is really important, sometimes you write code that is not and other times you write code that is somewhere in between. Should you apply the same quality standards to all of that code?

It really comes down to a question of value – code that you write once, run and then throw away obviously doesn’t need to be pretty and certainly doesn’t need any documentation. What about code that you keep around, make the odd change to and run regularly but is of low importance? What if it just doesn’t matter if the code breaks? Where is the value in making the code robust and easy to maintain?

Well maybe not so much “killer”… Anyway, Scoble mentions the return of SmartTags due to bloggers choosing to add them to their site. I pretty much never actually go to bloggers sites unless I want to write a blog post about them in which case I open a new tab in NetNewsWire to remind me for later, so I don’t notice them much. When I do see them though they really annoy me – they look far too much like hyperlinks and distract far too much from the content.

Pages created on .Mac however are the source of a never-ending headache. Indeed, whenever one requests a page on an account that no longer exists (such as the former .Mac FJZone account), the Apple servers dutifully serve a tri-lingual error page… all the while returning a “200 Found” code. In other words, as far as robots are concerned, .Mac pages live forever.

— Francois Joseph de Kermadec

This has to be my single most hated server misconfiguration. The problem is much more serious than unwanted pages turning up in Google searches – any program that tries to download resources from the server without explicit user interaction gets bitten because the server delivers a 404 page instead of the expected file without warning. The client side program can then only assume the file is corrupt on the server and give up.

I stumbled across the article This Is What You See, This Is What You Get the other day and it points out a number of common pitfalls for HTML editors that have relatively simple solutions, as well as repeating a number of common misconceptions about WYSIWYG editors – primarily that Word or Outlook should be considered good examples of how to do it.

Perhaps an obvious point. At least, the web is not WYSIWIG. What you see on your browser is almost certainly not what I see on mine due to many factors. Differing font sets, typographic capabilities of the OS, use of subpixel rendering, browser rendering engine/version, user display preferences such as screen resolution/depth, display gamma, as so on.

A while back I commented that I should look into the Delta Web project – I’m doing some work in this area now but nothing more seems to have happened. The mock schema for tracking deltas only works on an element level so is useless for describing changes to XHTML documents which is a shame.

Anyone know of any further progress or other related projects in this area?

Oops, forgot the direct link to the Delta Web proposal by Andy Roberts.

We’ve begun testing out Hamachi at work as a substitute for our defunct VPN and it’s showing a lot of promise. It sets up a peer-to-peer VPN which is quite clever and simple to get working. When combined with installing Bonjour on the Windows boxes (Macs have it pre-installed) and tweaking the DNS settings to add .local to the search domains, the DNS look-up works brilliantly cross-platform as well.

I’ve now got access to all the important stuff from work to tinker with my little side-projects while still being able to store them in the work subversion repository so others can join in if they want. It’s an awful lot simpler to work with than the old VPN stuff too.

David Berlind has an interesting post about the potential of Derby as an offline storage mechanism for thin-client web-apps. I can see the potential and an awful lot of difficulties as well. Might I suggest the perfect integration point for this kind of thing is a really cool Java WYSIWYG editor?

Hey Andrew (CEO of Ephox and a big fan of offline editing), can I have some time to try and prototype this on Ephox’s internal wiki? You could edit your wiki pages offline easily…

There seems to be a great demand for HTML documents that don’t include white-space between paragraphs – similar to going into the paragraph formatting in Word and setting space before and space after to zero. This is very simple to achieve in HTML, but people just keep coming up with strange ways to attempt to achieve it.

The most common way people try to get around the problem is by putting their entire document in one P tag and using BRs. You can pick these people because they set their HTML editors to insert a BR on enter instead of inserting a new paragraph. The end result looks right in all browsers but destroys the semantic structure of the document. I imagine it would be much harder to navigate through the document using a screen reader too, since skipping paragraphs seems like a nice way to skim. The problem people most often run into with this approach is that their editor still treats the whole big block as a paragraph, so when they apply a heading style the entire document becomes a H1 tag with a bunch of BRs.

There’s a very persistent (and very stupid) referrer spammer pummelling my blog and I’m getting sick of it – not least of all because at one point it actually managed to completely overwhelm the tiny little virtual server my blog runs on so that nothing else could access it. Given that I’m using WordPress with Apache, what’s the easiest way to block it?

Someone really needs to sort this out once and for all. It’s pretty simple to get Java installed so that you can run it – getting it installed so that the packaging system is satisfied that it’s there involves various bits of voodoo and integrating it so that you can use applets in Firefox is still a pain. Can we just decide on a standard location for the plugin file so that it can be auto-installed, or maybe even just a standard config file that tells the JRE installer where the Firefox plugin directory is?

Quite some time ago, I argued that unprivileged users were too annoying to be usable in Windows. Today I took shipment of a shiny new computer and figured that for once I might actually try applying all these ideas that are meant to make Windows secure. Previously I’ve just kept Windows safely behind a firewall and done my web browsing and email on my Mac – essentially eliminating any way for unvetted code to get to my PC.

Often when you define an interface there is a series of generic tests which can be applied to every implementation of that interface. For instance, the java.util.List interface carefully defines the behavior that all implementations must have. A simple way of testing these behaviors are present is to use JUnit’s ability to inherit test methods.

Define a base class with a test method for what you want and an abstract method to create the actual instance:

It occurs to me that there’s no way to stop trojans from working – there will always be a way to disguise it so that it looks like a valid application – if by no other means than by making it a valid application. There is also no way to allow the user to get their work done and prevent trojans from doing any damage or being annoying – even with privilege separation, the trojan has access to all the user’s files. Note that this only applies to trojans, not to other types of malware – viruses for instance can be prevented by writing secure code.

Chuq Von Rospach has some interesting thoughts on tagging – must come back to this and have more of a think about it.

I’ve been mostly ignoring the uproar about how Steve Jobs “lied” about how much faster the Intel iMac is opposed to the G5 iMac. Expecting benchmarks to reflect any form of reality is just plain ignorant of the complexities of modern computing.

Despite thing, there’s a great irony which has just dawned on me. If Steve Jobs is clearly lying about how much faster the Intel iMac is over the G5 iMac, then clearly he wasn’t lying when he was pushing the G5 chip and pointing out how fast it is compared to those crappy Intel chips. You remember those old photoshop bake-offs that everyone mocked right?

Here’s a tip for you, anytime you hear someone say, “The XML specs aren’t really that complicated”, you know they haven’t tried to work with XML in any detail.  For a start, as soon as you start using the term “specs” instead of “spec”, you’ve got something that’s going to difficult to piece together.  When you get to the vast number of specs involved when working with XML, you’ve got a weeks reading just to get through them all.

I’d like to be able to play around with the photos in my iPhoto library without having to export them first. Do APIs exist for accessing this stuff or is there an opensource library available somewhere that can read the iPhoto Library data? Google searches tend to indicate there isn’t any but I thought I’d ask the lazy web.

Dave Winer complains about Apple’s photocasting RSS being broken and while I’ll agree that it’s a shame Apple didn’t put more effort into interoperability, it’s also a shame Dave had to end with:

Assuming their intentions are good and they’re not trying to kill RSS, why don’t they put some of us under NDA and let us help them get the bugs out before they ship.

As if Apple have any reason to kill RSS when they’re going out of their way to leverage and promote RSS. How about letting the plain facts be enough criticism instead of having to tack on insinuating comments just to beat up the story?

I’ve been subscribed to Digg’s front page RSS feed for a few months or so and more and more I’m just skipping over all of its articles. It seems that as Digg’s popularity grows, the quality of the content drops. This probably shouldn’t come as a surprise – as more people enter the community, the community’s interests vary and the signal to noise ratio for a given person probably drops.  Even when the community’s interest doesn’t spread too wide, just the increase in the number of articles is likely to mean that readers skip articles more often.

Scoble:

I would love it if my blog tool could tell me more about the things I link to. For instance, how much traffic did it send to that person? How many people linked to it after my link (that would tell me the viralness of an idea)? How many times have I linked to Graham? How does that compare to the number of times I’ve linked to Dori Smith or Dave Winer? What’s the reciprocity of a link? (Did Graham link back and continue the conversation?)

Berin Loritsch:

I’ve discovered why I’m lacking motivation for our Software Design Document at work. It’s tedious. It isn’t the creative part of the effort. It’s boring!

The trouble with boring design documents is not just that they’re boring to write, they’re boring to read as well – so noone does.

In our most recent round of design documents at Ephox, we took some advice from Joel on Software and started injecting humor into the docs. So instead of just anyone inserting a table into a document, now it’s Miss Piggy creating a table of Kermit’s good and bad points – complete with a picture of Miss Piggy and Kermit. At one point we actually had half the engineering team rushing off to read the specification documents just so they could laugh at the stories.

If by any chance you happen to be trying to get a Ubuntu system (or probably any Linux system) to talk to an Apple Airport (in my case the original 802.11b UFO style), don’t try to use the plain ASCII password for the WEP key – Linux and Apple seem to have different algorithms for converting the password to the actual HEX key.

Instead, open the Airport Admin Utility, double click on the base station in the list to open its configuration interface and then choose “Network Equivalent Password…” from the “Base Station” menu. Enter the hex key it gives you into the Ubuntu networking dialog as a Hexadcimal key type.

Jensen Harris: Help Is For Experts

One of the most interesting epiphanies I’ve had over the last few years seems on the surface like a paradox: “help” in Office is mostly used by experts and enthusiasts.

How can this be? I think my biased assumption was that experts know how to use the software already and eager novices would be poring over the documentation trying to learn how to be more effective using it.

Jonathan Boutelle talks about how he made a wiki work with his software developers:

• Start off maintaining existing documents
• Make it easy to login
• Insist on Wysiwyg

So, so true. The first two could be a little more generalized: Start with a reason to use the wiki and make it easy. Making it easy encompasses insisting on Wysiwyg but the editor is the most important touch-point of a wiki that it’s worth stating separately.

Interesting argument between Tim Bray and Robert Scoble about what benefit standardizing Office 12’s XML format will provide.  Tim Bray suggests that Microsoft should use the open document format as the basis for its XML format with custom extensions when required.  Scoble argues that documents are more complex than Tim believes and that it would be impossible to create a compatible version of Office around the open document format. Maybe Tim Bray has done a lot of work with Word documents and office documents in general that I’m not familiar with – I know he’s done a lot with XML but that’s not enough to comment on the needs of a office-style document format¹.

Though everyone should already know about this, there is a survey out on what features to add to Swing’s text packages for Java 7. While that’s still a fair time away, and a long, long time away before you can count on everyone having it, now is definitely the time to make yourself heard about what features are required if you want to get them in.

Most of what’s on the list is possible to do today but takes a heck of a lot of effort to achieve, as well as a lot of very specialist knowledge. Making it easier would be a pretty big plus.

Dear lazyweb,

Do you happen to know if the scripting engine stuff that will be part of Java 1.6 is available for previous versions anywhere? I know there’s things like bean scripting framework, rhino etc but would like to use the one standardized API to drive things if possible. I haven’t seen much information come through about the scripting APIs which is a bit surprising actually.

Thanks!

Too many companies make the mistake of promoting programmers to management despite the fact that they are awful managers. I’ve long thought Linux suffered from that problem and Torvalds threatening to laugh in the face of contributors who submit new features after the 2 week window at the start of each new version’s development.

Linus Torvalds has threatened that if developers add ’last-minute things’ to the next version of the Linux kernel he will ‘refuse to merge, and laugh in their faces derisively’ Laughing in people’s faces is never acceptable behavior and threatening to do so is childish and demoralizing. It definitely doesn’t help to build community which is what opensource lives on. It might be a very good idea to limit new features to the first 2 weeks of development (though fixing known bugs before adding new features also has a lot of merit), and being very strict about it might be the best thing. Regardless, childish tantrums are not appropriate in software development.

It seems like noone has managed to invent a bug tracking system that can do search well.  Bugzilla’s famous for having a lousy search interface but its not just interface – I’m yet to find a bug tracking system that can search for a find bugs I’m looking for consistently.

This becomes particularly annoying when you are trying to avoid logging duplicate bugs as you have to spend ages crafting search queries to try and identify whether or not the bug has previously been reported.

I know I’m way behind on this but there’s one thing I noticed at the end of Macword’s “Close-up on Aperture”:

• By the way, there’s no Save command in Aperture. As you make changes, those changes are recorded in a SQL database.

Expect to see a lot more of that as applications take better advantage of CoreData and/or provide built-in versioning systems for their files. When you think about it, why should you have to remember to save instead of just having the ability to undo as far back as you like?

There was a time when the word of Jacob Nielson was undoubtable right and everyone must bow down before it. Fortunately it appears that time is well and truly over. His latest article on blog usability completely misses the point of blogs, the target audience of most blogs and often passes a simple reality check.

1. No Author Biographies
   • Let’s face it, no one really cares about you. Unless your an A or B list blogger the vast majority of your traffic comes from search engines. Most of the time your readers are just looking for the solution to their problem. They don’t need to know whether or not your reliable or intelligent, they just check to see if your proposed solution works.
2. No Author Photo

   • You’re a geek – no one wants to see your photo. Also, see item 1.

I’m getting sick of running around web boards posting this over and over again so I thought I’d post it here to try and get people to stop breaking their OS X systems by following bad advice.

First up, the golden rule:

Never modify anything under /System. Any instructions from anyone but Apple that suggests you do this is wrong.

So if someone tells you to modify the symlinks in /System/Frameworks/JavaVM.framework/Versions don’t. It’s not supported by Apple and it will cause you problems when you install future updates to either Java, the OS or other software.

Scoble:

One thing I totally agree with Joe on is that Apple is WAY WAY WAY better than Microsoft at coming up with names. “Apple Front Row” certainly beats “Microsoft Windows Media Center Edition 2005.”

It’s not just the name that’s better though, it’s the fact that Apple is shipping it with every new iMac. I have no idea if the software is better or not but at least people know about it and will try it out. Geeks seem to know about Windows Media Center but few other people seem to. Everyone who buys a new iMac will know about Front Row and will probably use it at least once.

Scoble: Nick says Memorandum sucks

Nick Davis says he’s done with Memeorandum and is going back to Newsmap. Says I like Memeorandum cause I appear on top of that page.

Well, yes, but that’s only cause the rest of you haven’t started participating in the conversation. Memeorandum is done automatically by analyzing linking and posting behavior. If you wanna be on top write a more interesting post. Hey, I bet this one has a chance of getting on Memeorandum.

Office 12 to support PDF natively

Oh, the Office 12 team just announced native PDF support. That’s big.

[Scobleizer: Microsoft Geek Blogger]

I agree. It sucks to be Adobe right now though as a huge chunk of their PDF related profit just went out the door but they really should have seen it coming and I’m sure they’ll survive – they have enough going on around PDF that the Word document to PDF side of it won’t be a major set back for them.

Ah Scoble, you’re just digging yourself deeper and deeper. The cynical part of me wants to point out how unsurprising it is that a Microsoft employee doesn’t understand why an application isn’t a spec and why precise specifications are so important for avoiding vendor lock-in.

But what Dave did was give me an application. It works. And, as a user, I wonder “if the format is so crappy, how did Dave get it to work in his own application?”

I spoke a little while back about Feedburner vs Blogbeat. I wish I’d taken more time to give constructive advice to the Feedburner team about what they could improve because as is so often the case these days, they picked up on my entry and Matt Shobe responded in the comments. One of the things he commented about was being able to redirect your feed so users don’t have to update their subscriptions:

Robert Scoble takes James Robertson to task for criticizing OPML in response to Scoble’s feature request. Mostly I agree with Scoble, when a user asks for a feature you shouldn’t give them a rant about why the specific technology they’re talking about is crap, you should give them your considered, professional opinion and recommendation. Hopefully that means identifying their problems and providing a solution but it might mean just telling them that their proposed solution won’t work and you don’t have an alternative.

As soon as I saw the new Office 12 interface I questioned why you would add modes to a user interface when modes are so frowned on in usability – particularly by Jef Raskin. I was immediately banging my head against the table with the thought that to do anything in the new Office I’d have to figure out which mode the functionality was hidden under instead of just working with the menu categories that I was used to (no better than the ribbon modes but at least I’m used to them). Then I watched Scoble’s video about the new office interface and felt a little better – hey, they’ve got a ton of user data to back them up, people do seem to work in different modes with Word and now the user interface matches the user model. Something just kept bugging me about it though.

At some point Eclipse seems to have decided that I shouldn’t be doing J2EE development and randomly turned off some of it’s J2EE related features like being able to Run as server or create a new Webapp project. I suspect a software update went bad and it’s screwed over it’s configuration.

Sadly, with the massive number of plugins and crap that makes up an Eclipse install it’s nearly impossible to sort out if one particular plugin got corrupted, if it’s a configuration setting some that got corrupted or if the whole thing is foobar.  The log file for the workspace is particularly unhelpful just reporting that classes are missing – that’s great, where were you expecting to find them though? Which plugin provides that particular class and which plugin is requesting that particular class? Then on a subsequent run it just reports that the plugin is already loaded so it doesn’t need to load again – sadly, the plugin isn’t loaded and none of the functionality it provides is available.

It’s a little bit scary to look at the number of browsers I have in my dock and regularly use.  Mostly this comes about because of the need to test on all the different browsers but still. I have a similar range of browsers on my Windows box at work (it adds Mozilla and Opera to the mix but obviously takes out Camino, Safari and OmniWeb).

In case you don’t recognize the icons, the browsers pictured from top to bottom are: Camino, Safari, Safari (built from CVS), Firefox, Deer Park (Alpha of Firefox 1.5, needs to be upgraded to the beta still, but last week was too hectic to be testing browser betas), OmniWeb and IE.

There’s a lot of talk going around about Sparkle, Microsoft’s new UI design tool set, but I’ve been scared that this is the end of good user interface design since the second paragraph I read about it:

It’s the rise of the graphic designer!

(from Scoble’s blog)

At which point in time did we start letting graphic designers design user interfaces?  What happened to all the user interface designers?  You know, the ones that think about how to make software intuitive and user friendly, efficient and productive as opposed to flashy and cool looking.

I’m with Geek News – PDC has been a big build up for an average set of announcements.  I’ve got to stop reading Scoble’s blog, he over hypes everything and then the actual announcements disappoint.  Here’s Scoble’s summary of the announcements:

1. Office 12 demonstrated publicly for the first time. Tons of new features and new UI.
2. Windows Vista features demonstrated publicly, including search integration, new performance enhancements, new sidebar.
3. LINQ (Language INtegrated Query). Cool database stuff for .NET developers.
4. Windows Presentation Foundation/E. “E” for everywhere.
5. Start.com updates released.
6. Atlas (our AJAX Web development toolkit) demoed for first time.
7. Microsoft Max. A new photo sharing and display application.
8. Digital Locker. A new place to find, try, and buy software.
9. New sidebar and gadgets and new Microsoft gadget Site.
10. Coming later today? Sparkle. A new way to build Windows applications.
11. Coming later today? Lots of server stuff.
12. Coming later today? More Office stuff.
13. Coming later today? Workflow stuff.

All in all a pretty nice set of announcements but definitely nothing revolutionary.  There are a couple of things that could have been really excited but got slaughtered by the Microsoft, gotta maintain the monopoly policy.  In particular Windows Presentation Foundation/E – cool, cross platform support that includes all the major browser that’s awesome.  Wait, what’s that?  It’s a subset?  So when I develop my application I have to have two versions – a Windows only with all the cool stuff version and a other suckers version.  That’s great, for Microsoft at least.

About 5 years ago while I was doing some part time work for my university one of my lecturers walked by, looked at the program I was developing and asked: “You’re using antlr or something like it to generate your parser aren’t you?”.  I wasn’t, I’d written the parser by hand in an hour or two and it worked exactly as I wanted so I saw no need to go back and rewrite it.

In the comments for my post Ant Is Cool, Erik Hatcher points out that Groovy has ant support (see http://groovy.codehaus.org/Ant+Scripting).  While that’s cool and all, I really don’t want to add a new language into the mix just to call out to Ant so not a great option for my case, but useful to know.  What is more useful however is knowing that you can embed Groovy in ant scripts using the Groovy ant task (also via the ant script task since 1.5 or 1.6 I think).

I think I’ve reached the point in my programming career where I’m about as lazy as you can get. Fortunately I’ve also started heavily leveraging the power of ant recently as a cross-platform, heavily Java oriented, shell scripting tool.  Shell scripting isn’t really the right word for it though because it’s really the declarative nature of ant files that makes it work so much better than shell scripts, python, ruby etc.  I don’t think about how to do things I just say what I want done and it happens.

Dear lazyweb,

I’m writing a J2EE (well technically JSP/Servlets but no EJBs) webapp which has two goals:

1. Secure authentication is required.  Unauthorized users shouldn’t be able to access anything except a login page.
2. Easy access for authenticated users.  ie: Don’t prompt users for their password every time they open a new browser instance and don’t time out logins.  Prompting the first time they access the system using a particular browser/computer is okay though.  Even prompting once per day would be okay though not ideal.

I realize that these goals tend to conflict to a degree – ie: it’s far more secure to log users out every so often so that if someone gets access to a machine they logged in from they don’t get access automatically. We’ll have to live with that risk and so lets just assume that the systems users are accessing the system from are secure or users are smart enough to always click logout when they’re finished.

I pulled the big red lever on CVS this morning and converted over to Subversion at work.  It’s gone pretty smoothly though one of the engineers is complaining that Eclipse is now freezing up a lot.  Not sure if that’s related to the Subversion change of if something else is going on but it’s something I’ll have to investigate some more.

It’s sad but I get a buzz out of being able to move things around in the repository right from within Eclipse and knowing that the history is being preserved.

You’d think it would be obvious given the number of people looking for ways to clean up Word’s so called HTML output to make it look something like HTML, but apparently Google failed to google it.  Good work guys.

So the new Blogger Word plugin is going to result in a whole ton of really, really, really bad HTML being published to the internet (in addition to the current volumes of it).  Sigh, and we had been making progress….

Everyone’s been busy arguing about what the appropriate name for those things that NetNewsWire and FeedDeamon download, parse and display (eg: Scoble, Steve Gilmor, Scoble, Dave Winer, Scoble, Scoble and of course Scoble – not that Scoble posts a lot or anything….)  Anyway, no one really cares, the vast majority of people haven’t heard of any of those terms and will have to learn what they mean so whatever the majority of people wind up learning is what will stick and everyone will have to live with it.  I’m betting the easiest thing to say will win – most likely “feed”.

For the past few weeks I’ve been investigating and planning a move from CVS (CVSNT actually) to Subversion at work. Hopefully this week I’ll be able to do the actual conversion and see what mess it makes.

The main thing that’s missing from Subversion is an exact equivalent for CVS’s meta-modules. Subversion’s externals are good but need to support relative URLs so that branching is simple. Even so, it’s not that hard to go back and re-edit the externals property after branching and it can probably be automated with a small script.  It might actually be a good opportunity to automate adding new branches to our automatic build setup as well so whenever we create a new branch the build machine (affectionately known as Bob) automatically starts building it.

I’ve previously griped about the lack of a good IDE for Ruby on Rails and despite trying out TextMate for a month I just wasn’t happy enough with it – I’m still spending far too much time navigating files and not enough time coding.  There’s a lot of talk about just using VIM and being happy with that (example) but that VIM just doesn’t have a good enough interface for selecting which file to edit and navigating the file system, let alone for quickly scanning files to find the section I want.

I’ve got a few little projects in mind that I think Cocoon would be a good fit for and one system already in production that uses Cocoon (badly) so I’m thinking I need to work take some time and really learn how to use it properly and what the best practices are for using it.

So, oh great Cocoon guru’s who hangeth outeth at planet apache (and those who don’t), what’s the best way to go about learning Cocoon?  Book recommendations, article recommendations etc?  Where do I learn best practices and get a good overview of all the parts etc rather than just how to do specific things?

Wikis were invented to make it easier for people to contribute content.  They do this with two key features:

1. In browser editing.
2. Users don’t have to learn HTML.

That sounds awfully familiar.  It is in fact two of the key things that Ephox provided as we helped create the in-browser WYSIWYG editor industry which is now a key feature of most content management systems, knowledge management systems, document management systems and most other enterprise systems that people want to encourage contribution to (or reduce the complaints from people who are forced to contribute).  As a side note, I’m not sure whether the inventor of EditLive! had heard of wikis or not when he started his work and I’m not sure which was created first but it really doesn’t matter.

I probably should title this post, Why Big Media Is Dominating Podcasting, because I suspect that there are already more listeners of “big media” podcasts than there are of “little guy” podcasts.  I don’t have figures to back that up though so lets not worry too much about what the current state is and look to the future – will podcasting stay true to it’s roots and be a way for the “little guy” to have his say or will big media take over?

Scoble pointed to a random blog posting of someone saying they prefer Windows over Mac OS as he’s paid to do.  Usually I skip them but for some reason I read this one.  It really struck me as odd that in a post that’s meant to be pro-Windows, it’s really more about how to get by with Windows:

There are a few things to remember about windows. Turn on automatic updates and put everything on a broadband connection behind a router. They can be picked up for about $40 bucks. Don’t install every crappy shareware program or file sharing software that comes along .

A while back, Benjamin commented about a problem his parents had organizing photos:

Watching my mother trying to use Windows XP to locate her holiday snaps makes it clear to me that tagging is the right way to interact with personal documents. The traditional “one file, one location” filesystem is old and busted. The scenareo begins with my mother learning how to take pictures from her camera and put them into foldlers. Unfortunately, my father is still the one managing short movie files. The two users have different mental models for the data. They have different filing systems. Mum wants to find files by date, or by major event. Data thinks that movie files are different to static images and that they should end up in different places. The net result is that Mum needs to learn how to use the search feature in order to find her file, and is lucky to find what she is looking for.

Most people have probably already worked this out but if you’re new to rails or new to rails login components, you probably want to use Login Generator and avoid Salted Login Generator like the plague.

Login Generator is extremely quick and easy to get up and running with, is made up of a small amount of clean code, has no dependencies and works with Ruby on Rails 0.13.

Salted Login Generator on the other hand is crap.  Sorry to the author of Salted Login Generator for putting it so harshly but it really is a mess and has caused me a huge number of headaches.  The feature creep seems to be what killed it – the localization is particularly problematic and makes all the code horrendously messy but depending on a firstname and lastname field in the database is bad too (should have just had login and let each application decide how and if to store the user’s real name).  If it were trimmed back so it just focussed on handling authentication it would probably be a lot better for it.

So Sam Ruby is pandering for links in a vain effort to tell Apple that iTunes’s RSS support is horribly non-standards compliant.  Of course this effort will fail horrible because Apple doesn’t do anything unless there’s a Radar issue created and Radar issues created by internal staff carry far less weight and get far less priority than Radar issues created by external parties.

That’s right, you can create Radar issues yourself.  Apple has a bug reporter which is the official and the only official way to tell Apple anything.  Don’t like the fact that Apple only acts on things that have a Radar issue?  Log a Radar issue telling them so!  Don’t like the bug reporter? Log a Radar issue telling them so!  Don’t like the way iTunes works with RSS?  Log a radar issue telling them so!

Feedster seem to have improved their indexing times recently – they now find every new entry on this blog immediately when I publish it – the Feedster item (for my vanity search on “Adrian Sutton”) actually appears before the actual entry for this blog (I subscribe to my own feed to make sure it’s working correctly).  Good stuff.  I’ve probably managed to set up WordPress to ping something whenever I post (I seem to recall seeing something in that box) and Feedster has linked into that somehow (that or I’m pinging Feedster directly but I don’t think so).

Ah Scoble, when are you going to learn?  Everytime you mention Mac and anything Microsoft related in the same sentance, a whole bunch of Mac users are going to flame you something savage.  In this case of course their completely right – Windows Media Player for Mac sucks and I don’t bother watching any Channel 9 videos because it sucks so much.  I could probably use VLC which I use for anything else that Quicktime can’t play but it just seems like too much hassle just to listen to a bunch of people prattle on and on instead of just reading a quick text based blog entry.

So iTunes 4.9 is out and the podcasting support is pretty cool.  It does pretty much what you’d expect – podcasts appear in a new “Podcasts” “special playlist” just under the “Library” icon.  It does keep track of whether or not you’ve listen to a particular podcast or not so you can keep track of where you’re up to and has a bunch of configuration options for how to handle incoming feed entries and how long to keep stuff.

Ken Arnold talks about the complexities of generics and how he doesn’t feel they provide enough benefit for the complexity they add.  I have to agree, most people probably only understand the basics of generics and not the full spec which is a recipe for great confusion and the number and type of bugs that generics prevents really doesn’t give much benefit.

It’s odd that there’s such a push towards scripting languages with no compile time type checking at the same time the Java crowd are pushing for stricter compile time type checking.  I can honestly say that in all the perl, PHP and Ruby scripting I’ve done that not once have I ever had a bug caused by having the wrong type of data in a variable.  Not once.  You never should if you have clearly defined what each variable’s purpose is and if you haven’t done that you’ve got problems that extend way beyond what a compiler can help you with.

Recently Apple made WebKit development much more open which is great for people who develop products using WebKit as they get more control – it’s great for the KHTML developers as it’s easier for them to integrate Safari changes and for a bunch of people to test against.

The downside however is that people who have no business playing with development builds of WebKit just can’t help being cool and running it anyway.  What’s worse is that people encourage this stupidity by making prepackaged builds available.  If you can’t follow the very simple instructions to build your own version of WebKit from CVS, you really, really, really shouldn’t be working with a development build – you almost certainly don’t have the skills required to provide a worthwhile bug report anyway.  The build instructions for WebKit are probably the most brain-dead simple instructions I’ve ever seen for any open source project (you thought ./configure; make; make install was difficult – try just build-webkit; run-safari).  There’s even a simple script to pull the latest updates from CVS without having to know anything about CVS commands.

So I’ve been getting into Ruby on Rails and things started out well, had the basis of an application up pretty quickly and now I’m starting to get into the logic of the app rather than just pulling values out of the database and displaying them.  Productivity is droping, very rapidly.

The problems all seem to come down to the lack of a good IDE.  Specifically:

• I have no idea where the definition of methods etc are.  Are they inherited?  Are they dynamically generated by something?
• I have no idea where the documentation for stuff is.  Is it at api.rubyonrails.com, is it on ruby-doc.org, is it buried somewhere in my gems installation?
• Navigating files is just too slow.

It wouldn’t take much of an IDE to fix those problems just display a nice list of files in the project down the side with tabbed editors, then have the ability to select a method, variable, etc and jump to it’s definition or it’s documentation.

Ugo Cei commented on the new technorati interface and while I think it’s an improvement over their previous attempt, it still needs work.  Searches work consistently now, but it’s still way too hard to find the RSS feed for the search and there appears to be a few bugs which prevent you from getting access to one at all for some searches.

Oh well, looks like I’ll be using the Feedster integration with NetNewsWire for a while longer.  I did send them some feedback which hopefully made sense, I’m not really sure how to accurately describe the problems I was having with finding feeds.

While this site is graciously hosted by Iain I’m in the process of developing another site (non-blog) and the more I think it through the more likely it is that it’s going to drive a not insignificant amount of traffic and I definitely want it to be snappy all the time (as opposed to slowing down when Iain decides to download too much at once).  At the same time I keep playing with different technologies from J2EE to Ruby on Rails to custom perl apps and whatever else, so being able to set up whatever I want is a pretty big bonus.  Sounds like a virtual server might just be the best bet but I really don’t want to spend much money.

I’ve been developing a fairly dynamic website using Ruby on Rails for the past week and I’m really quite impressed.  I’d had no experience with Ruby before so I had to learn a new programming language and a new webapp framework but I’ve gotten going without too many problems and am already at least as productive as I would be with J2EE (possibly more so).

Like most things though, it’s definitely very possible to create really awful looking code in RoR though most of the time the easiest way is very maintainable.  I was quite surprised to see how big a mess the code created by the salted_login generator was.  It appears that instead of putting in effort to keep the views to purely display, the author tried to minimize the amount of stuff in the views.  Thus, instead of using HTML to create form elements which would have been simple, readable, maintainable and work in WYSIWYG editors, a helper function was used that output a complete table row with localized label and the form field.  What’s worse though is that the helper function is about as readable as a perl script.  It’s pretty simple to rewrite the view using plain HTML and then the helper function can be deleted entire without understanding exactly what it does.  One would hope you don’t need to write code like that to achieve internationalization – this project doesn’t make sense to localize anyway.

There are an awful lot of search engines out there and they all try to index as much of the web as they can, as quickly as they can.  For this site, search engines seem to cause more traffic than anything else:

<th align="center" bgcolor="#008040" colspan="2">
  <font size="-1">Hits</font>
</th>

<th align="center" bgcolor="#00E0FF">
  <font size="-1">User Agent</font>
</th>

<td align="right">
  <b><font size="-1">6866</font></b>
</td>

<td align="right">
  <font size="-2">12.63%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">msnbot/1.0 (+http://search.msn.com/msnbot.htm)</font>
</td>

<td align="right">
  <b><font size="-1">5600</font></b>
</td>

<td align="right">
  <font size="-2">10.30%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Googlebot/2.1 (+http://www.google.com/bot.html)</font>
</td>

<td align="right">
  <b><font size="-1">3546</font></b>
</td>

<td align="right">
  <font size="-2">6.52%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/</font>
</td>

<td align="right">
  <b><font size="-1">2633</font></b>
</td>

<td align="right">
  <font size="-2">4.84%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com</font>
</td>

<td align="right">
  <b><font size="-1">1706</font></b>
</td>

<td align="right">
  <font size="-2">3.14%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">sna-0.0.1 mikemuzio@msn.com</font>
</td>

<td align="right">
  <b><font size="-1">1120</font></b>
</td>

<td align="right">
  <font size="-2">2.06%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/5.0 (compatible; BecomeBot/2.3; MSIE 6.0 compatible;</font>
</td>

<td align="right">
  <b><font size="-1">1103</font></b>
</td>

<td align="right">
  <font size="-2">2.03%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">NetNewsWire/2.0 (Mac OS X; http://ranchero.com/netnewswire/)</font>
</td>

<td align="right">
  <b><font size="-1">806</font></b>
</td>

<td align="right">
  <font size="-2">1.48%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Krell-GeoScraper/0.1 libwww-perl/5.79</font>
</td>

<td align="right">
  <b><font size="-1">754</font></b>
</td>

<td align="right">
  <font size="-2">1.39%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com)</font>
</td>

<td align="right">
  <b><font size="-1">750</font></b>
</td>

<td align="right">
  <font size="-2">1.38%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)</font>
</td>

<td align="right">
  <b><font size="-1">746</font></b>
</td>

<td align="right">
  <font size="-2">1.37%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Planet HUMBUG +http://planet.humbug.org.au/ Planet/1.0~pre1 +</font>
</td>

<td align="right">
  <b><font size="-1">745</font></b>
</td>

<td align="right">
  <font size="-2">1.37%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Planet Linux Australia http://planet.linux.org.au Planet/0.2</font>
</td>

<td align="right">
  <b><font size="-1">744</font></b>
</td>

<td align="right">
  <font size="-2">1.37%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET</font>
</td>

<td align="right">
  <b><font size="-1">726</font></b>
</td>

<td align="right">
  <font size="-2">1.34%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Planet Apache +Unconfigured Planet Planet/1.0~pre1 +http://ww</font>
</td>

<td align="right">
  <b><font size="-1">702</font></b>
</td>

<td align="right">
  <font size="-2">1.29%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">NewsGatorOnline/2.0 (http://www.newsgator.com; 1 subscribers)</font>
</td>

<td align="right">
  <b><font size="-1">673</font></b>
</td>

<td align="right">
  <font size="-2">1.24%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net</font>
</td>

<td align="right">
  <b><font size="-1">655</font></b>
</td>

<td align="right">
  <font size="-2">1.21%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">NetNewsWire/2.0b45 (Mac OS X; http://ranchero.com/netnewswire</font>
</td>

<td align="right">
  <b><font size="-1">648</font></b>
</td>

<td align="right">
  <font size="-2">1.19%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gec</font>
</td>

<td align="right">
  <b><font size="-1">621</font></b>
</td>

<td align="right">
  <font size="-2">1.14%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">NewsGator/2.0 (http://www.newsgator.com; Microsoft Windows NT</font>
</td>

<td align="right">
  <b><font size="-1">601</font></b>
</td>

<td align="right">
  <font size="-2">1.11%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/5.0 (compatible; BecomeBot/1.86; MSIE 6.0 compatible;</font>
</td>

Now admittedly a lot of those hits will result in Not Modified responses but still, when you expand this to every site on the internet, that’s a lot of HTTP requests being fired around.

I’m sure I’ve mentioned numerous times that you should pay no attention to benchmarks, and apparently the benchmarks Anandtech used in their recent PPC vs Intel article were pretty bollocks too.  So if you read or heard about that article and now believe that threads on OS X are too slow, you should read this response from Ridculous Fish.  You should also read the first comment (which was the only comment when I read it).  Then you should return to the real world and forget about all this nonsense.

I hope not.  Not that I don’t think the Intel chips are superior at the moment, but because I really don’t want to have to deal with all the problems of major architecture changes – particularly such a major shift as PPC to Intel.  That’s a major headache for developers and a whole heap of compatibility problems for consumers.

Worse than that, it’s yet another platform for us to test on and there’s already way too many of those.

Not so long ago, I said that people tend to use Windows because of Microsoft’s monopoly more than because they actually like Microsoft.  Apparently, when Scoble asks them why they run Windows, these are the common responses:

Have you ever gone up to people and asked them why? I have. And the reasons people give are quite varied. Yes, “my boss gave me this” +is+ a common answer, but it’s only one I hear maybe one out of four times.

Currently at Ephox we use MSN Messenger for a lot of communication but the servers are somewhat unreliable and it would be nice in some cases to be able to tie conversations into various internal systems (for instance, when talking about a bug it would be nice to “invite” the bugzilla bug into the conversation so that the conversation is logged).  There’s a bunch of other things I’d like to play with but don’t know where to start.  For the following, please ignore privacy issues, this is for company related conversations only and I’m aware of the privacy issues and will deal with them in time – I just want to play with the idea and see how far it can go first.  Something like archives could potentially be written so that only the participants of the conversation can access the archive.

Scoble just doesn’t get it sometimes. Here’s a tip – there’s a difference between using Windows and liking Windows.  Most people hate computers in general – they don’t like Microsoft, they don’t give two hoots about Apple (though they love iPods). Most people these days are forced to use computers.  Most people aren’t qualified to evaluate which computing platform is better for them and can’t be bothered learning how to do so (when was the last time you met an average Joe who could accurately identify even a couple of key differences between OS’s).

I got my first SpamKarma report on comment spam for symphonious.net in over a week today.  It reports no attempted spam whatsoever.  Previously I’d been getting almost a hundred attempted spam comments a day and now suddenly it’s stopped.  I’m pretty sure I didn’t change anything and I don’t think my comments are broken (though I had to relax the spam protection a while back as it was blocking comments with more than one link which caused problems for a couple of people).  Course if my comments are broken, let me know below….

I just downloaded Eclipse 3.1 M7 and tried it out.  This is the first eclipse build I’ve tried on OS X 10.4 (on Java 1.4.2_07 and Java 1.5.0_02) and for the first time it actually appears usable on my powerbook.  Previous builds (on previous OS’s with previous versions of Java) were always frustratingly slow to use and I’d rapidly give up and go back to a Linux or Windows box (or vi).  With this release it appears to keep up with my typing, editor windows open and close quickly and startup is really fast.

Various people seem to be having problems with Mac OS X 10.4, particularly in the installation process.  Perhaps it’s just my innate desire to rub it in but I feel compelled to point out that my upgrade has been pretty much seamless.  The only problems I’ve had are with Mail.app – GPG caused it to crash which is to be expected since Apple explicitly said not to use those APIs as they are likely to change.  I’m also having a weird problem where Mail.app is no longer synchronizing mailboxes properly.  When you choose ‘Synchronize “adrian@intencha.com”’ from the Mailbox menu, it doesn’t do anything – nothing shows up in the activity viewer and new messages aren’t found.

There’s been a bunch of long term planning going on at work recently – lots of good stuff has come out of it but I’m going to skip most of it here.  Most of the responsibility fell to the managers in the company which leaves me to actually get some work done and push our products forward.  One thing I did pick up though is the responsibility to ensure that Ephox eats it’s own dogfood.  Henceforth I am on a mission to cram our products into every conceivable place and by any available means (we use a number of hosted solutions which we can’t directly integrate our products into but JavaScript hacks are a fun pass-time so I’m sure I’ll find a way).

So this CEO walks into an engineering department and asks about setting up the news section of the company’s website to use a blogging system instead of updating it by hand.  Great idea.  Now given that the company’s website is hosted on an IIS server with MS SQL 7, ASP, ASP.Net and PHP available – what software does the intelligent engineer recommend?

If there was a very significant benefit, a subdomain could be used for news and pointed to a Linux server with MySQL, Apache and PHP (and for which root access is available) but that’s a lousy option due to the extra overhead of having to maintain that server entirely ourselves and the difficulty caused by splitting our website over multiple servers etc.  It would have to be one heck of a big advantage to make it worth the trouble.

While the buzz surrounding the release of OS X 10.4 continues to build I’ve found it amusing how Microsoft have been desperately pushing Longhorn in the last couple of weeks only to receive criticism about it and then desperately try to point out that all the cool stuff just isn’t there yet, but it will come soon – honest!

I’d say more but I think Crazy Apple Rumours summed it up perfectly.

So apparently Safari now passes the Acid 2 test (or at least the CVS version does).  Excellent work.  What’s more impressive though is that the patches needed are all available so the KHTML developers should be able to integrate the fixes themselves reasonably quickly.  I have however heard reports that the KHTML team refused to accept a large amount of Apple’s patches for various reasons so the code bases may be far enough diverged now that merging is problematic – anyone know for sure?

So now there’s adsense for RSS feeds (or at least an early beta of such).  It raises some interesting issues.  I hate ads so I’ll quite happily unsubscribe from any feed that has ads in it, but what about the various planets that I subscribe to?  What if one person who’s syndicated through those planets adds advertising to their RSS feeds, would I unsubscribe from the whole planet?  Possibly.  What if a few people did?  Probably.

I have to give another congratulations to the NetNewsWire team – I just realized I’d been taking advantage of a very simple but very clever piece of user interface design.  The contextual menu in the NetNewsWire browser has two “Reload” menu items in it, one at the top and one at the bottom.

Regardless of where you click on the page and whether or not the contextual menu pops up or pops down from your mouse cursor, the reload item is always right next to your cursor so it’s easy to hit.  Now you might think that it would be better to just detect which way the contextual menu popped and move the refresh menu item to that end, or even flip the entire menu so that the distance to all the items is unaffected by how the menu pops.  The downside of doing that though is that it makes it much harder to find the items because they keep moving around.  With the duplicated item, the menu is always the same so it’s very quick to identify which way it popped and then move to the item while still keeping the most commonly used item close at hand.

The engineers at work are starting to find reasons to have an news feed aggregator running on their machines at work, mostly so they can keep track of changes being made to the wiki, but it provides a convenient mechanism to push content out to the entire team without being too intrusive.  For some reason RSS feeds seem to be able to handle more information flowing past before it all becomes to much to sort through.  As we get more of the engineers with RSS aggregators running, there’s a much lower barrier to entry for new information feeds and a much higher pay-off for developing them.  For instance, when I get a few free moments I’d like to set up an RSS feed for CVS commits (this is harder than it seems since somehow we wound up using CVSNT instead of just putting the CVS repository on one of our Solaris boxes).  If I’m the only one subscribing to that feed it’s probably not worth setting up, but if half the engineering team is subscribed then it can really start to increase the chances of stupid mistakes or missed scenarios being picked up.

It’s been quiet around here of late because I’ve spent pretty much all my spare time at rehearsals for a new play debuting in Brisbane titled The Last Of The Red Hot Irish Lovers which is probably best described as a light hearted drama.  I’ve been put in charge of the technical side of the show – mostly focussing on sound.  If you’re near Brisbane I strongly recommend you go see it, the opening night was tonight and there are shows Friday and Saturday from 7:30 and Sunday from 2pm as well as Thursday, Friday and Saturday of next week from 7:30.  All at the MetroArts building in Adelaide street.  Tickets are available at the door or bookings can be made by contacting me (contact details are in the sidebar).  I think it’s about $15 a ticket or $10 concession.   If you need further info give me a yell.

Monsieur Scoble asks:

Finally, I’m gonna ask a provocative business question of Opera (and other browser manufacturers): What’s your business again? If all the browsers have the same underlying features, and they should only add things that are standards, what differentiation are you offering your customers and investors? Are you saying Firefox’s developers can’t propose anything new that’d push the Web forward? Hey, how about some linking technologies like Greasemonkey? Is Firefox not allowed to add anything like that that the W3C didn’t propose and that the WaSP didn’t approve of?

If you’re having trouble with Airport reception dropping out a lot or the range doesn’t seem to extend as far as it should (in my case I could stand next to the base station and keep losing reception), it may be caused by something like Romeo sending out a lot of bluetooth traffic.  When I started using Romeo I also started getting frequent problems with connecting to the airport network and now that I’ve stopped using Romeo it’s been running smoothly again.

Iain Robertson and Robert Scoble have commented on Google’s new smart tags feature and while I agree with most of what they said I have to disagree on their views of producers rights.

Iain said:

Admittedly, the main people complaining will be those whose content has been altered, i.e. the producer of a given piece of work, who might find that some people are tracing new and interesting links away from their site. Fair enough that they complain, too — I know the idea of J. Random Company “hijacking” content in my site irritates me somewhat. To that end, I’d hope that Google would use the MSSmartTagsPreventParsing meta-tag, and/or a common tag standard can be agreed upon to allow content creators to opt out of the “service.”

Gianugo Rabellino has been playing with unit testing in Swing – an area that I’ve spent a lot of time complaining about in the past.  In his particular case he’s trying to send key events to Swing components programmatically so that his JUnit tests can run automatically.  As Gianugo discovered you can’t just create a KeyEvent and send it to Component.dispatchEvent() because it gets rerouted to the focus manager.

Gianugo’s solution was to display the component on screen in a JFrame and make sure it has focus.  However, we can go one better – we can simulate the component having focus.

The basic idea of unit tests is that you write them once and run them a lot – you don’t check in your changes unless all the unit tests pass.  This is a great idea.

There is one problem with it – it prevents unit tests from being used as a specification of work to be done later.  I don’t mean it prevents test driven development where the tests are written then the code is to make them pass is added and then the tests and code are checked in.  I mean where a particularly complex piece of functionality is going to be added and the best way to ensure that all the use cases are covered is to write unit tests for them all and let them act as the specification (note: this doesn’t work well if you have a non-technical client, but if you are working on off the shelf software it can be very useful).  You need to plan the set of features to include in a release ahead of time so you wind up with a whole heap of new unit tests which are acting as this specification that at some point in the future you will make pass.  At the moment though, those tests are okay to fail.

I’ve spent a large chunk of this week investigating what the best way to set up continuous integration builds is for our product.  Being a cross platform application we need a continuous integration set up that builds on four different OSs constantly (Windows, Mac OS X, Solaris and Linux).  Ideally of course it would do so on all variants of those OSs and at some point that might be possible using vmware of some kind.  For now though, just building and running the automated tests on some version of the OSs we support is a huge improvement.  Fortunately, our build process already uses ant and requires just a single target be achieved.

Ben Fowler points to AppFuse, a quick way to set up J2EE projects with more buzz words than you can shake a stick at.  Specifically it sets up a Tomcat/MySQL apps that:

Uses Ant, XDoclet, Spring, Hibernate (or iBATIS), JUnit, Cactus, StrutsTestCase, Canoo’s WebTest, Struts Menu, Display Tag Library, OSCache, JSTL and Struts (or Spring MVC)

That’s a huge number of different technologies.  It’s not necessarily bad to use all those technologies together (in fact it’s probably very good) but it’s important to be aware that pragmatic programmers don’t use wizards they don’t understand.

It’s funny, last night I read this reuters article about Microsoft beginning an initiative to improve their interoperability and thought wow that’s excellent!  I bet Slashdot completely misses the point of the message and rants on about how Bill Gates also mentioned that opensource doesn’t necessarily lead to interoperability.  Sure enough, Bill Gates Claims Linux Has Poor Interoperability.  Sigh.

Gates of course is right – there are plenty of open source products out there that aren’t interoperable in one way or another.   Further, while Linux systems generally adhere to standard network protocols well, they have an abysmal track record of interoperability on the same system.  The best example of this would be the lack of robust drag and drop support.

Brian put forward a Java pop quiz and it happens to be an area I know very well:

Without breaking out a compiler, or BeanShell ;-), what is the output from the following:

System.out.println("**" + foo != null ? foo : "NULL" + "**");

The statement is equivalent to:

System.out.println(foo);

Why?

The left most + operator is the first thing to be evaluated. So the if statement then evaluates whether or not "\*\*" + foo is null which it never is and so the value of foo is always output. Similarly, the "NULL" + "\*\*" may as well have just been "NULL**", in fact the compiler optimizes it to be exactly that as javap shows:

G. Andrew Duthie asks: “Will 2005 be the year of Least Privilege” (via Robert Scoble).

Not a chance.  Why not?  Scoble sums it up pretty well (unintentionally):

For those who don’t know what Least Privilege is, it means turning off a bunch of administrator rights so that no software can install without logging off and logging back in as administrator

People don’t want to have to log off and log back on to get stuff installed!  That’s awful.  People want to use their computer and have it get out of the way.  What should happen is that they are prompted for an administrator login when admin privileges are needed and it should all just work seamlessly once such a login has been provided.  Similarly, if they are logged in as Administrator, they should have to provide their password to install software anyway so that they know it’s happening.

Robert Scoble points to an article at Microsoft Monitor complaining that Microsoft is releasing too many betas.  Personally, I love the fact that companies are releasing previews of their software openly and regularly and I want to see more of it.  I know if it actually leads to better software or not, but it lets me evaluate new technology ahead of time and decide if it’s worth pursuing.  Most commonly it’s not worth pursuing straight away but later I run into a project that could benefit from that technology and return to see where it got to.  By that stage the product is normally out in a final release (often a 2.0 or 3.0).  The beta allowed me to see in broad strokes what the software was capable of and get a quick feel for it.

A while back I mentioned that spamassassin wasn’t filtering my spam too well anymore and that I suspected it might be because the bayesian filter had been neutered.  Apparently I was right.  Since I upped the score for BAYES_99 the amount of spam reaching my inbox has dramatically reduced.  I’m back down to one or two spam messages getting through per week instead of four or five a day.

Very happy.

Technorati seem to have worked out much better ways for claiming your blog in their listings than Feedster – particularly the new auto-claim mechanism (of course you need to trust them a fair bit to use that).  With Feedster I have to paste this stupid link in a blog entry and annoy anyone subscribed to my feed or any of the planets that aggregate my feed with a pointless entry.  So Feedster, when are we going to get a less annoying method?

I previously mentioned that I don’t get Technorati.  In response to that I got a bunch of good responses so I thought I should play with it more in light of those comments.  It seems that Technorati is best used for tracking news as it happens, so the release of the Mac Mini was the perfect test case.  To add to the fun I also ran the same search on PubSub to compare the results.

I’ve been reading Scoble a lot lately and he’s so excitable about blogging and related technologies that I figured it was worth checking out some of them.  So I started playing with Technorati and so far I’m just not getting it.  It’s a far worse search engine than Google, it doesn’t pick up on changes to content particularly well, it requires people to go to a fair bit of effort to ensure that they do wind up in the index and most of it’s search results just point to stupid people ranting on their blogs (you know, like I’m doing now).

A while back this article came through the Reuters RSS feed about someone bringing an antitrust case against Apple because he was “forced” to buy an iPod.  Now while my non-laywer opinion is that he has no chance of actually winning (hint: there’s a thriving competitive environment for online music with a wide range of viable alternatives that he could have gone to instead of the iTunes store), it does concern me that people think the courts should be used to enforce open standards.  Doing so would be hugely detrimental to the software world and harm consumers more than it helps them.

David Jericho revisits his complaints about the airport express and ends with the comment:

To that end, try as I might, from borrowing and using my sisters 1.2GHz iBook for a month, to using Kirsten’s G4 machines at home exclusively I remain unconvinced. I fail to see a use for the hardware (and the price tag – although the laptops have come down), and I just can’t see what benefits the software offers anyone in an enterprise.

pk commented on my last entry:

Wait a minute… where did you back up that it *could* have been written on another platform? Maybe yes, maybe no but let’s keep the playing field level.

Well firstly I based it on the fact that I’ve written a similar application on OS X myself which seems like pretty convincing evidence.  Secondly you could create a mathematical proof based around the fact that it would be possible to develop such an application on a Turing machine, however I’m on holidays and that’s a lot of work.  Finally it’s pretty easy to provide commercial examples of this being done on both Linux and Mac OS X as pb did in the comments on Scoble’s blog:

David Jericho complains that he had a hard time setting up a new Airport Express.  I’m not what caused his experience to go so badly but my parents managed to set up an Airport Express without any problems – I know because I’m using it right now.  You can troop out the old stalwart anti-Apple crap all you like but it doesn’t make it true.  Apple *does* play well with others, they go to great lengths to make their software compatible – it’s certainly not perfect but there’s a lot of good stuff in there to make compatibility easy – I know because I use Macs in cross-platform, cross-hardware situations on a daily basis.  Just because it didn’t do precisely the job you wanted it to on one particular occasion doesn’t mean that the company is crap.  Fortunately, Google had some good suggestions for David.

Benjamin finally got back to our discussion about exceptions.  The summary there is a pretty accurate picture of my viewpoint and the history of the debate.

It seems Benjamin doesn’t like the fact that python throws exceptions when stuff goes wrong:

Fine. An exception is thrown. I don’t catch it. I get a stack trace on the command-line, and everyone’s happy right?

No. The pygtk main loop catches it, reports it, and keeps running.

I’ve defended Sun a fair bit in the past and I still think that there’s a huge number of people making mountains out of molehills and just looking for any chance to knock them down a peg, but even I can’t believe the bone-headedness required to release source code under a "look but don’t compile" license (Graham Hamilton was the initial source of the news for me and the official site is over at jck.dev.java.net).

You know, I often ignore click-through licenses because I’m lazy, however I’m always suspicious of software that feels it would be beneficial for me to change my browser’s homepage to some cluttered, ad-filled portal with no useful content – it’s just a pet hate of mine, even if you can easily turn off the option.  Since the MSN toolbar asks to set your homepage before it shows you the click-through license, I thought I’d flick through it quickly.  Section 2 is enough for me to not agree to it:

I’ve brought home one of the SunBlade 100s from work with the intent of installing Solaris 10 on it and trying to get a grip on the idiosyncrasies of Solaris.  Eventually I hope to be able to put the e250 we have at work to better use and learn enough to actually admin it properly instead of just doing the very minimum possible to get things to work.

Being a programmer by trade, picking up sysadmin skills isn’t exactly the easiest thing in the world so it will be a bit of a journey.  Fortunately I have a basic working knowledge of UNIX and only a few fairly straight forward things to get up and running on the machine.

I’ve commented a few times before about how difficult it is to test text manipulation code and most of that is caused by the fact that it’s so hard to comprehensively define all the possible states, user actions and resultant behaviors that are expected.  There’s just way too many different cases that can occur (infinite in fact).  The two hardest areas for most rich text editors are lists and tables (not surprisingly since their really the only complex structure used in text markup).  In order to improve behavior in those areas and avoid any regressions we’ve started specing out the expected behavior.

XML databases and the standards to work with them (primarily XUpdate and XQuery) are just plain crap.  Why is it that I need to learn two completely different languages which utilize two completely different programming styles to work with a database?  Why is it that I can’t seem to convert between a String and XML nodes in either of those languages?  I mean, wouldn’t seem obvious that when working with an XML database someone might have some XML coming in as a parameter and want to deal with it as XML instead of as a String – say to put it into the database?  Isn’t it obvious that someone might want to get a part of the XML file they’re processing using XSLT as a string instead of a node-set?  I don’t mean get the text nodes within the XML fragment, I mean all the element declarations and namespaces etc, etc.  Sure they’re more than just random text but if you wanted to output it so it could be edited it would be nice to be able to without jumping through hoops.

A couple of people commented on my beautful code entry saying that the IDE should take care of code formatting and indentation and that ‘beautiful code’ is mostly about design.  I’d have to disagree, beauty is only skin deep.  The design of code doesn’t make it beautiful, it makes it maintainable – it makes it work.

You’ll have to excuse the inherent sexism in the following analogy – it happens to be the best one I can think of.

Any half-decent coder will agree that writing readable code is important and that good comments are a part of that.  More and more I’m getting the impression that that’s not enough.

Code should be a thing of beauty.  Not necessarily "oh that’s a beautiful algorithm" and definitely not "oh wow that’s clever" but beautifully laid out with every blank line considered for meaning and well thought out comments everywhere.  Every construct and every line should match the selected coding style and real thought should be put into why that particular coding style was selected.

I’ve been struggling with getting Cocoon and Exist to work together properly and allow me to insert an XML snippet into a document in the database.  Apart from the fact that the apparent assumption seems to be that no one would ever have an XML snippet being fed into an XML-based system and thus all the < and > characters should be converted to entities, plus the fact that Exists seems to only want to retrieve stuff from the database and not change stuff in the database, I’ve wound up with the following:

Maybe I’m expecting too much but I would have thought that a browser that claimed to be standards compliant might actually implement the standard….  In particular I’m annoyed that Gecko doesn’t fully support HTML 4 tables (it ignores the align and valign attributes on colgroup and col elements).  IE gets it right.  To be fair, Safari also ignores them.  Oh and of course you can’t achieve the same thing via CSS since CSS styles don’t get applied either via a styles attribute or via embedded styles (rightfully so since the standard doesn’t say they should be applied).   You have to manually markup each cell in the column. Yuck..

I thought it was worth mentioning that I got to play with Derby as part of the Time Tracker mini-project I mentioned earlier.  Very cool.  I’ve not had a need to work with an embedded database before so it was really nice to have a small, efficient library that handled all my data storage needs.  The only complaint I’d have is that it needs to use a directory to store stuff in (it’s a little more flexible than that but a directory is always created and used).  It would be really nice to be able to just point it at a file and say "that’s my database, do your thing".

Java Desktop Integration Components (JDIC) is a set of components aimed at making OS specific functionality available to Java programs and to allow Java programs to make better use of existing code within the OS.

The tray icon integration is an excellent example of this.  It’s quite common for a Windows application to make functionality available via a system tray icon and now via JDIC Java apps can do just that.  During some down time after our last release I had a chance to play with this while developing an internal time tracking application and it worked out really well.  The system tray icon acts as the main UI for the program so it’s neatly tucked out of the way so it doesn’t waste screen space but always quickly available so recording time spent on a task is highly efficient.  The downside is that it doesn’t seem to map to an OS X dock icon menu yet (the API doesn’t really fit that model particularly well anyway but it could be done).

I really don’t understand why there’s so much anti-Sun sentiment around these days.  I mean, the opensource movement in general just seems to have it in for Sun no matter what they announce or plan it’s either never good enough of there’s some sinister plan behind it.  I don’t get it.  Sun is one of very few big companies who are seriously weighing up the prospects of opensource and seem to generally be finding that opensource can work for them and moving in that direction.  The opensource movement however, far from encouraging this is pulling out all stops in a smear campaign against Sun.

ONLamp.com has an article comparing alternatives to make (heavily biased towards make).  In fact it only looks at ant and IDEs, I assume because it’s an extract from a book about make.  It also focuses very heavily on Java development which is a real shame because Java development is the one area that make is almost never a good option.

The main reason that make isn’t a good option for Java despite it’s technical abilities is that every Java developer is expected to know at least a little about ant – a huge number won’t know anything about make and very few Java developers who work on Windows will have it installed.  Every self-respecting Java developer will have ant installed and configured on their machines.  It’s just the done thing when you work with Java.  No self respecting C programmer would develop on a machine without make, no self-respecting Java programmer develops on a machine without ant.

It’s good to see Sun getting into the spirit of openness in the ways that really matter – weekly builds of the J2SE Mustang release will be available from now, including the source code (source Mark Reinhold).  I’m not sure exactly what’s in the source code drops yet but even if the native code side of things isn’t included it’s still extremely useful.  Most if not all of the people interested in fixing bugs in a Java release are Java programmers so they’re most likely to be submitting patches on the Java side of things anyway.

Andy Lester asks Uncle Bill to answer some technical support queries for his family.  Generally I agree with his sentiment but I have to pull him up on one thing:

At some point, we’ll get her HP laptop turned over to Linux, but until then, I have the weekly-or-so cussing from the dining room table.

and then later:

The bigger question for the industry: Why do we continue supporting a company that sees BSODs and registry hacks as part of normal computing life for the average user?

Because the alternative seems to be supporting a loosely coupled group of volunteers that sees competing GUI conventions and command line hacks as part of normal computing life for the average user.

I finally got around to setting up Synergy between my OS X laptop and my PC.  It’s simply awesome.  Generally the way I work has fitted well with having two keyboards on my desk but it’s definitely a bonus to be able to use my nice wireless keyboard and put the laptop up higher instead of hunching over it destroying my neck.

Definitely should have set this up years ago…

I’m pretty sure I’ve mentioned how cool Java Webstart is (or more specifically, JNLP) but I’d not thought about how other technologies achieve the same thing.  Martin Pool apparently has (hint: he’s using yum).  JNLP has the advantage that it includes security concepts (sandboxing in particular) and that it checks for updates each time the program is run rather than only when the user runs the updater.

Having said that though, JNLP needs to include finer grained specifications for security privilege requests (currently it’s essentially applet, servlet or full permissions) as well as the ability to turn off automatic checks for updates (or specify that the user can decide).  Currently though JNLP is excellent for deploying applications within corporate infrastructure and pretty good for deploying to outside customers.

This week was a big release week for us.  With new versions of EditLive! for Java, EditLive! for XML and EditLive! for Windows all going out.  My main focus has been on EditLive! for Java’s release and it was pretty huge.  There’s a lot of things in there that we’d been wanting to do for a long time but couldn’t because of limitations in the Swing text components.  We’ve now replaced so much of the standard Swing text code that they’re possible.  Stuff like spell checking as you type, inline image and table resizing and a much better selection model just makes life for users so much easier.

I finally got around to embedding EditLive! for Java into Movable Type so I don’t have to do all the HTML layout by hand.  It’s amazingly simple to integrate once you work out which template you have to edit as well, just 5 lines of JavaScript.

More and more spam has been gradually getting past spamassassin and winding up in my inbox lately. It’s getting really annoying. So today’s project is to upgrade to SpamAssassin 3 in the hope that it contains some new voodoo that will clean up my inbox again. It seems to have a new requirement on Digest::SHA so this time the build can go through CPAN but it’s on to the tests now and everything seems okay. Hopefully it will be a smooth upgrade.

Andraes Schaefer finally comes up with a solution to the great equals debate. It turns out that in fact it is possible to implement equals in such a way that it works well with subclasses that add additional constraints to equals.

Andraes’ solution is still not great though because of the restrictions he mentions:

• All sub class must overwrite and adjust the equals() method otherwise line 8 in the base class will create an endless loop
• The equals() method in the sub class cannot call the equals() method in the base class otherwise it ends up in an endless loop, too
• Line 11 in the Complex class cannot check against a sub class of Irrational in a different branch (meaning it is not a sub class of Complex, too)

I think we can solve the first two of those problems by throwing more code at it (I’m not sure we want to but I think we can). Here’s how, in the base class we have:

Benjamin still doesn’t like exceptions but I sense I’m making some headway.

Again, I think it comes to the number of paths through a piece of code. and much later:

Exceptions put a possible branch on every line of code, and that is why I consider them evil. It seems to be this belief that exceptions put a possible branch on every line of code that is making Benjamin dislike exceptions. Again though, this is just a case of exceptions making possible errors obvious. For instance, how many possible codepaths are there in the C code:

Benjamine Carlyle and I have been discussing exceptions. Put simply, he hates them, I love them. I think I know why now.

Adrian Sutton argues that exceptions are not in fact harmful but helpful. I don’t know about you, but I’m a stubborn bastard who needs to be right all the time. I’ve picked a fight, and I plan to win it ;) I’m also a stubborn bastard (just ask Byron about ampersands), so let the death match begin! Actually, as I mentioned in this case I think we’re arguing different things so we might be able to resolve this amicably. Benjamin’s biggest problem with exceptions is that code in between the throwing of the exception and the handling of the exception “gets hurt”. The answer here is called abstraction. Abstraction in this context has a very simple tenant: when I call a method, it does stuff and I don’t care how it does it. In other words, if I call the method doStuff(), I don’t care if an exception is thrown in a deeply nested function call or directly by doStuff, to me doStuff simply through an exception. This is then true for doStuff, it only cares about each method it calls and not any methods under that and so on. So for any given piece of code we only think about one level. Methods and classes provide abstraction. Now that we’ve abstracted things, there simply is no code left in the middle to get hurt. Each method should handle whatever exceptions it may cause in whatever way it needs to. It’s the whatever way it needs to that differentiates Benjamin’s viewpoint and mine. Benjamin has been talking about exceptions in C++ whereas I have been talking about exceptions in Java. In C++ when an exception is thrown you have to carefully sort out what memory you had already allocated and not freed then make sure you free it etc etc etc. In other words, an unexpected failure is catastrophic and pretty much unrecoverable. In Java however, it’s no big deal at all. Objects will happily be garbage collected when no longer required, it’s simple to tell if objects have been created and it’s also really simple to clean up any allocated resources. So in Java if we were working with a database, we’d do something like:

Benjamin claims that exceptions are evil, he couldn’t be more wrong. Exceptions are in fact one of the best ideas that has been added to languages, particularly checked exceptions which force people to deal with error situations. Benjamin’s first problem with exceptions is that they’re impossible to test. This assertion is flat out wrong. Exceptions simply make you aware of a case that you’re not testing. For instance, say we have a function that writes to a file, as part of our testing we should test that it behaves correctly (ie: produces the expected behavior) even in situations when the file can’t be written either because of a lack of permissions, a full disk, missing directory, network error or hardware failure. It doesn’t have to work in those situations, but it must behave predictably because one or more of those situations will occur when the program is in production at some point or another. Without exceptions, the code might look something like (using some convenient but totally fictional libraries):

If you ever try to download, burn and install the Ubunto warty ISO image on OS X you’ll find that Disk Utility crashes and hdiutil crashes. To get it to burn you need to install cdrecord using fink (fink install cdrecord) and then use the information on this page to work out how to burn it. On my PowerBook G4 with a DVD-R/CD-RW drive I used the command: sudo cdrecord -v speed=24 dev=IODVDServices warty-rc-install-powerpc.iso and it’s working fine. We’ll see how it goes from here on in.

Iain points to this gem of an article. Some choice quotes:

Dr Greening and colleagues asked 42 men to ejaculate every day for seven days Excuse me Sir. Yes, sorry to interrupt – would you mind doing me a favor? Yes, I just need you to jerk off every day for a week. … Say, how’d I wind up in hospital again?

I think it’s exciting Don’t we all Dr Greening. Don’t we all.

I’ve had my old powerbook lying around for a fair while now not doing anything so I thought it was time to put Linux back on my desktop. I’ve pulled the laptop back off the shelf, booted it and found Debian is already installed and running okay. So I officially have Linux running on the desktop! Now what?

Martin Pool gives an informative pros and cons list of Ubuntu Linux versus a number of other OSes. I’d like to jump off on a tangent. Martin gave a plus ot Ubuntu for:

• Less Macintosh historical baggage. Mac apps can be in the wierd state of running but having no windows open: GNOME designers just know this is silly. OS X suffers some friction between the Mac and Unix parts, in e.g. case sensitivity or resource forks. While there is definitely some historical baggage in Mac OS X (not much mind you – Apple were pretty ruthless with cutting things out), the ability for an application to be running without any windows open is actually a design feature and not historical baggage. When described as “the wierd state of running but having no windows open” it sounds downright stupid, but if you describe it as “the ability for the menubar to not disappear suddenly when closing a window” it makes a lot more sense. The Macintosh menubar is placed at the top of the screen because it’s significantly faster to access the menus when it’s placed there (plenty of research backs this up – essentially the sides of the screen have infinite depth so they’re easy to hit). Since the menus are at the top of the screen and not in the window, it makes no sense for the menubar to change when you close a window. Thus, when closing a window in an application the application continues running allowing you to open a new document, adjust preferences etc etc. That’s not the only reason though. Think of someone using pen and paper who has two tasks to complete. They start the first task by getting out a clean sheet of paper and writing on it until they are finished. Then they file that piece of paper away. Finally they start the second task by getting out another clean sheet of paper and starting writing again. In other words, the life cycle is:

1. Create a new document.
2. Work on document.
3. Save and close document (filing both saves the paper and removes it from the workbench)
4. Create a new document. If you were using Gnome, Windows or pretty much any OS other than Mac OS, step 3 would suddenly cause your pen to jump back into the drawer. You’d actually have to rethink the order you do things in to open a new document before you close the previous one. Now you might argue that people typically work on more than one thing at a time and they are likely to have both documents open anyway. They might indeed but in that case there is another subtle benefit. When they decide they’ve had enough of work and want to put it all away and play a game, they simply hit command-Q and all of the documents in that application close. It’s like having a bulldozer that can file things neatly. The disadvantage to the Mac OS way of doing things was that on OS 9, you didn’t realize the application was still running and either ran out of RAM or got really confused when you next double clicked on the application and nothing happened (in fact the menubar changed but noone seems to notice that). OS X (and the magic of modern computing) solves both these problems. Firstly, you won’t run out of RAM anymore because OS X has virtual memory and the ability to move things around in RAM (OS 9 loaded things in order and fragmented memory effectively couldn’t be reclaimed – plus the virtual memory was really bad). You could leave every application on your system open in OS X and it just wouldn’t matter because the unused applications would be paged out to virtual memory. Note that because they are unused they wouldn’t constantly have to be paged back in either so no hard drive thrashing. The second problem is solved in two ways. Firstly there is a reasonably obvious indicator that an application is open (the icon appears in the dock if it wasn’t already there and a black triangle appears under the icon). This helps a little, but not a lot. The biggest improvement is the Aqua Human Interface Guidelines change which now requires that when the user clicks on an application in the dock or double clicks it in the Finder (the same actions used for opening it) the application will bring it’s existing windows to the front if it has any, or create a new window if not. If you combine the effects of these two changes, the result is that the “weird state” of an application being open but with no windows effectively ceases to exist. The user can be completely unaware that the application is running and continue to go about their work without ever noticing. When they try to open the application again it behaves exactly as it would have when first opened and displays a new window. There is no noticeable impact on resources because the RAM can be swapped out to disk and made available to other applications and modern hard drives have plenty of space to cater for this. The user is now able to operate in their logical order without menubars unexpectedly changing on them. So as it turns out it’s the GNOME developers who got the UI design wrong predominately by putting the menubar in the window instead of at the top of the screen. Once they’d made a mess of that behavior, it wasn’t really an option to avoid the disappearing menubar problem because the menu is so inextricably linked to the window.

A little while back I said duplicate code was the biggest codesmell I knew of. I was wrong. There is a level of smelly code that goes well beyond duplicated code, in fact it goes well beyond anything a professional programmer would ever do. That is the smell of code that doesn’t have JavaDoc comments. From time to time every programmer will get lazy and not bother to add a comment for something but every so often you come across a code base that seems to have a systematic policy of removing any JavaDoc comments that might prove useful. Sadly, I was reminded of this when attempting to use BCEL. Vast swaths of that project are completely devoid of JavaDoc comments. Maybe the functionality of those methods are completely obvious to anyone who’s ever touched the codebase, but I’m trying to work out how to use the library so I need to at the very least be told “this method is not for you – go away”. In the end I gave up and used ASM which is still fairly light on JavaDocs at times and quite often the JavaDocs that are there are too abrupt to be particularly useful but at least there’s something. All the same, the process of using ASM is essential a case of trial and error unfortunately. So if you ever teach anyone Java, beat into them how useful, powerful and important JavaDoc comments are. It doesn’t matter how neatly laid out your code is or how obvious the functionality may be – most people who go to reuse the code are going to look only at the JavaDocs either through the generated HTML or through their IDE. Make sure the information there is useful.

It appears the java.net crowd picked up on my labeling some of them wingers – the response has been much more polite than I would have expected (or deserved really). The most well reasoned response so far is from Chris Adamson who points out the history of Java releases lagging behind on Mac as a reason for people to be worried. Before I go into Chris’s comments, let me point out that there are two main groups of people:

Oliver Hutchinson commented on my last code duplication entry:

Another way that duplication sneaks in is when people reimplementing something for which there is already a perfectly good library. For instance, your example of arranging dialog buttons in the appropriate order for the platform is taken care of very nicely by the JGoodies forms library. Actually Oliver, this is the perfect example of when you should duplicate code. There are a few reasons for this, the most obvious one being that JGoodies depends on Java 1.4 and our software currently only requires Java 1.3. Dropping support for Java 1.3 (and thus IE on Mac) isn’t really worth it to save writing 4 lines of code. Secondly (and far more importantly), the JGoodies forms library provides an entire library for laying out forms. We don’t want an entire library for forms layout – we already have that problem solved using the standard classes in the JRE. The JGoodies jar file would add an extra 80k to our applet which while not awful, would be 79k or more we didn’t need. Thirdly, adding an extra library increases the chances of a conflict occurring in our JavaBean version if one of our customers is using a different version of the JGoodies library. To avoid this we’d have to rename the package which results in duplicated code again except now the entire library has been duplicated. Finally, the JGoodies code is significantly more complex than it needs to be. It’s simply not a good implementation given the set of requirements we have. The JGoodies implementation uses a Factory class, a Builder class, an AbstractBuilder class and a custom layout manager as well as a host of other classes. Our implementation uses a JPanel and an if statement. Granted, the JGoodies implementation is capable of creating a wide range of different button dialogs that work in a much wider set of situations, however none of that provides any benefit to our project. Adding dependencies is a choice that has to be weighed up very carefully and should never be done just to avoid writing a few very simple lines of code. Furthermore, when replacing duplicated code with a common library, it is imperative that the requirements (as opposed to the implementation or results) of both pieces of code are the same. It is not enough that they be similar, they must be the same – they have to logically be the same function before they should be made actually the same function. On the other hand, if JGoodies had already been in use by our application, Oliver would have been completely correct that we should have used the existing function because it did meet all the requirements and despite the fact that it met a lot of requirements that weren’t needed because we would have to test the custom layout manager etc so the complexity wouldn’t matter so much.

Well David is right. Queen Fans do love A Night At The Hip Hopera. Definitely worth a listen and props to David for bringing it to my attention.

Benjamin makes some good comments about code duplication and software design. I think by and large I agree with him though I do think that less code duplication implies better design, it’s just that there are also other factors which may be more important as well. The critical thing about eliminating duplicated code is to identify duplicated code based on it’s requirements, not on what it looks like. For instance, today I added a new method to our UIUtils class (GUIs are the most common place where duplicate code is created unnecessarily), the method took as parameters an OK button and a Cancel button, created a panel and put them in it in the appropriate order for the platform. It’s about 5 lines of code all up and it’s duplicated all over the place when it should not have been. In many cases, when this code is duplicated, it doesn’t actually look the same – it might use different layout managers, be mixed in with other code or a wide range of variations. However, the requirement is extremely clear cut: On Macintosh the cancel button is on the left and the ok on the right, on all other systems the ok is on the left and the cancel is on the right. The buttons are separated by a gap of 4 pixels. Regardless of where ok and cancel buttons are used or how they are implemented, that requirement must be upheld. Once I re-factored it out into a utility method it also became apparent that OK and Cancel buttons also have to have their preferred size set in a specific way – something we had another utility function for already since the behavior is similarly well defined. So a call to the sizing function is added to the layout function and we have a standard library function that gives us an ok and cancel button pair ready for adding to a dialog. There are other things that need to be set about ok and cancel buttons (for instance their action listeners are almost always identical because of the way we write our code) but that’s getting too big for a static utility method and it’s not always common to every ok and cancel button. Having said that, I really should re-factor out the cancel listener because I can’t think of a cancel button that has different requirements plus it has to be used for the escape key and a bunch of other places. Getting rid of the extra class files the anonymous listeners create would probably save us a hundred k or so. Coming back on topic, the design used isn’t correct in terms of object oriented programming, in fact it’s almost aspect oriented programming as it forms a cross cutting concern across all GUI dialogs, however it is extremely effective, has an extremely clearly defined purpose and is short and simple to understand even at a glance. Possibly the biggest indicator that a utility method should be used is that if the requirements for one dialog’s layout of the ok and cancel buttons changes – the requirements for every dialog will also change. The need for consistency in implementation is a big red flag that there should be a single implementation. So of course now if we ever support a new system that orders the ok and cancel buttons like Mac OS does or perhaps some other layout, we only have to make the change in one place. Essentially, I think Benjamin and I are saying the same thing in slightly different ways, except that I don’t class code that should behave different as duplicated code – only code that has the same requirements is duplicated. On a side note, Benjamin used the phrase:

What’s the thing that most stands out as “gee, this is going to bite me later” code? For me it’s without a doubt code duplication. I really hate seeing duplicate code even if it has slightly different behavior (it should put the common behavior into a parent class or utility methods etc and then just have different code where the behavior is actually different). Every time I see code duplicated I just know that a bug will turn up in it at some point and it will wind up only being fixed in one of the copies. I also tend to design using generic plugins a lot. Sometimes I probably take it a bit too far but most of the time the generic approach winds up being much more flexible and robust, plus it’s usually easier to understand generic plugin style designs because they separate out functionality a lot more clearly – this plugin does this, that plugin does that. Of course, generic non-plugin type designs tend to have the opposite effect. There’s also this really messy middle ground between a plugin style design and a non-plugin design where things just wind up coming out ugly. Usually you get into that area when the functionality hasn’t been split up carefully enough or when the desired functionality is different enough to require separate functionality. One of the areas I haven’t found a design I’m happy with yet is for menu and toolbar code – particularly when the menu and toolbar is configurable. It’s exceptionally clumsy to manually create each menubar and toolbar as if they were completely different things (they’re not – they’re just buttons of different styles that wind up firing an event) but it can also get very messy if you try for an MVC approach because the behaviors can become disturbingly complex (think submenus, comboboxes, toggle buttons, radio buttons, enabled and disabled etc). I’ve tried out a number of ways to handle this and I’m not quite happy with any of them. I’d like to try out a design where each menu is created from a particular definition (a set of properties that describe how it behaves) but that there are separate categories for the different types – so there’d be a toggle button MVC, a plain button MVC, a sub menu MVC, radio button MVC etc all integrated somehow (an overarching MVC kind of pattern?). I don’t know, I’ll have to find a project that will let me put it into code at some point and see what comes out of the concept.

Sun released Java 1.5 (or 5.0 or something) yesterday and today suddenly there’s a whole bunch of people complaining that Apple hasn’t yet ported it to Mac OS X. What the? People do realize that porting the JVM takes time right? Apple don’t get the final source code from Sun until Sun finish it (oddly enough) and Sun only just finished it. Sure Apple has had access to prerelease builds from Sun (and similarly Apple Select and Premiere developers have access to a seed release of Java 1.5). It will take time to for Apple to port the JVM and add in all the extra little details they provide (where little amounts to an entire L&F plus more). Then it has to go through QA (you do want it tested right?) and then it will be released. Daniel Brookshier” leads the charge with the whining. Of particular note is the comment:

Anthony Towns give a good rundown about how there’s no evidence to support the claim that “pirate” DVDs support terrorist organizations. It should be noted that this is a highly tangential topic to what Anthony was writing about – he just got me started on this so linking to his entry serves as a means to let you catch up with my thought process while giving the added benefit of linking to a highly informative piece. I do feel somewhat compelled to point out that regardless of whether or not pirated DVDs support terrorists or not, the large scale for-profit infringement of copyright is highly unethical. Don’t take that to mean that I think it’s unethical to make a copy of your favorite CD so you can play it in the car, or put a copy on your iPod or even making a copy for a friend. In fact I’d go so far as saying that downloading a copy of your favorite song so you don’t have to pay for it is not necessarily unethical (questionable but not clearly unethical). However, when you start to make money off of copyright infringement I believe you’ve taken it too far. Artists do have a right to compensation, it’s all well and good to say that they still have their copy so they’ve lost nothing, but it is important that people can make a living out of creative works. Selling illegal copies of an artists work really rubs me the wrong way. Now, on the other hand if these “pirates” had taken an existing work and given it a swashbuckling sea shanty feel then rereleased it for profit then I’d put the actions back into the questionable area (and in fact I’d lean towards saying that it’s a good thing). Extremes are generally not good for society. So it’s not good that the MPAA, RIAA (and other similar organizations) want to completely eliminate fair use, nor that they want to lock up creative works under copyright law forever. Similarly, it’s not good to simply throw out all the copyright laws and leave artists with no way to recoup costs of creation or make a living out of their works. The trouble is finding where to draw the line. One of the things I find so frustrating about the current copyright laws is that it’s practically impossible to build on to existing works to create something better. This is essentially the case of our pirates providing a swashbuckling version of the latest hit that I mentioned earlier. I firmly believe that it is in societies interest to have a very active creative culture and I believe that building onto existing work is an important part of that culture of creativity. There are two main benefits provided by allowing people to build on top of other people’s work easily. Firstly, it provides a simple and effective way for new creators to get started. Thinking up a good, original idea for a song, screen play, musical, whatever is really hard and doesn’t happen often. Thinking up an interesting new viewpoint on an existing work however is fairly simple. This provides a simple starting point for people to get into creating and develop the skills required to execute a creative project without running into the roadblock of the original idea. More importantly however, building on top of additional works can improve on the original work either by being better written/performed, by taking a more interesting view point or simply by updating it for modern times. Romeo and Juliet is a classic love story, the essence of which can be enjoyed by people old and young alike. However, culture has moved on from Shakespearean times and while some may find that disappointing, it would sadden me to think that the pinnacle of writing was in the past. More importantly, it would be sad to lose the classic story line in Romeo and Juliet just because the writing style is outdated and doesn’t appeal to today’s audience as much. Fortunately, because the copyright has expired, people can build on Shakespeare’s work to produce updated versions like the sensational West Side Story and the movie of the same title which brought these works to a whole new audience to be appreciated in whole new ways. Having said that, I would be really annoyed if someone took my work, made only very minor changes and published it as their own. I would be thrilled if they added substantially to it and gave credit (even if they made a lot of money off of it and didn’t pay any royalties to me). I’d probably be annoyed if they made substantial changes but didn’t credit me with at least giving them some inspiration. So there is a fine balance here and a set of ethical behaviors need to evolve and be recognized. Such a set of behaviors is already in place when it comes to quoting research however those behaviors don’t directly apply to other mediums and other types of creative work so new sets of ethical standards need to be created and where (and only where) required backed up by law. The end aim, at least in my opinion, must be to improve the availability and quality of creative works within our culture and not to build or support an industry around creative works. Those two aims however are not entirely at odds with each other, in fact it is essential to have some kind of industry built around creative works to provide the monetary backing required to actually fund creative projects as well as to allow creators to pay their living expenses while devoting their full time effort to creative works. Where the right balance is and how you find it, I don’t know. I do know however that it isn’t what we’ve currently got and it doesn’t seem to be in the direction we’re heading.

Brad makes an excellent comment regarding the root logins via ssh issue:

I think the biggest disagreement we’re having here is where should this be solved. Adrian , as a developer, thinks it should be coded around. Myself, as a sysadmin, think the user should take some responsibility for their actions and check their setup on the critical pieces of software – in this case, the Internet accessible ones. It all makes sense now. I think the movie Babe probably best covers this when the sheep dog goes to talk to the sheep and talks very slowly and clearly because every sheep dog knows that sheep are stupid. When the sheep reply they do so very slowly and clearly because every sheep knows that wolves are ignorant. In this case every developer knows that all users are stupid and so they try to make everything work as safely and easily as possible. Then of course every sysadmin knows that programmers are incompetent and so they double check everything and always assume that the developer has stuffed up. It’s then quite natural (and indeed beneficial) for Brad and I to have differing reactions to this issue. As a sysadmin Brad knows that he should read the README.Debian files for the security related packages he installs and that he should double check configuration files (plus he often knows how to do this off the top of his head). As a developer however, I know that when the software gets into the hands of end users (as operating systems for PCs generally do), the users aren’t going to do any of this (mostly because they won’t know how) so the default configuration should be secure and stable to avoid the user having problems.

Brad comments on my condemnation of root login being enabled in the default SSH config for Debian systems (noting again that SSH is disabled by default).

Debian’s SSH package explicitly asks if you want to run the ssh daemon, and by choosing to do so, you take a certain level of responsibility into your hands. Granted – Sandra acknowledged this and I acknowledged this.

I don’t agree that its the software’s fault more than the users – as a maintainer you make some assumptions, some of which will not match the users requirements, and its up to the end user to ensure that it meets their needs. The assumptions should always err on the side of security. It is trivial for a user to turn something on if they discover it is missing – it is effectively impossible for them to (knowingly) turn something off if they don’t realize it’s turned on. Microsoft is very often criticized for leaving unneeded services on by default and not being configured securely by default – Linux should receive the same criticism when it falls into the same trap.

With the real planet humbug down, I’m only occasionally checking the temporary planet humbug since it’s not coming through my RSS feeds at the moment. While I wasn’t looking, there seems to have been a little bit of a stink kicked up about Linux’s security. The story as far as I can tell seems to be that Sandra Mansell had her Debian router compromised because the root password was a dictionary word, ssh was available to the world and root logins were allowed. She took responsibility for the compromise pointing out that it was “lazy” (I would have said careless or possibly even crazy) to use a dictionary word as the root password and then complained that Debian’s default settings were to allow root logins over SSH. Brad Mashall and Greg Black then provided some useful advice on setting up SSH and security in general. However, from both posts (particularly Brad’s) I got the impression that this was entirely the users fault. I’d very strongly disagree with that. Was the user at fault? Absolutely, and Sandra acknowledged that. Was the software more at fault? Absolutely. It is very common knowledge that allowing root logins via ssh is an unsafe practice and should be avoided. It’s completely inexcusable for the Debian developers to have root logins enabled by default and even worse that it (apparently) doesn’t even display a warning about this. It is simply not good enough for an OS to ship with insecure settings and expect ordinary users to perform a full security audit and know all about arcane config files to make their system secure. The OS installer should always make the system as secure as possible given the features the users have indicated they need. Specifically, the installer should have disabled SSH by default (I think it did) then when told that SSH was required, it should have left root logins disabled. Better yet, it could have asked which accounts were required, from which networks and hosts and whether or not it could generate a key pair to be used for authentication and disable password authentication. All of that however does make the installation more complex so there’s a balance to be reached, but since there’s almost never a reason to allow remote root logins I’d be quite prepared to criticize the developer responsible for that configuration being the default. I’d then set about making sure that the users understands where they went wrong and how to avoid similar problems in the future (most likely by pointing them at Brad and Greg’s comments, which I definitely suggest you read). Why is it that in commerce the customer’s always right but in software the user’s always wrong?

Byron continues on the ampersand issue:

I’m not going to accept your argument that it’s not harmful to produce invalid HTML. What would your code produce for: http://example.com/entities.cgi?entity=& The requirements are that it should produce exactly that since that will work in all known browsers and would break in all known browsers if the ampersand wasn’t escaped. Since I didn’t personally write the code I can’t be certain that it does output that, but that’s what it should do. It should output whatever it is that makes things the most compatible so our users are the most happy.

I try not to link to everything that comes through the Oddly Enough feed, but this was just too funny to resist. Have Sex Until The Cows Come Home Source: Reuters.

We’ve acquired a new engineering manager at work so at long last we’re starting to put in place some of the things we’ve always said “we should do that” about for a long time but never actually gotten around to doing. One of those things is establishing how accurate our estimates are by actually tracking the time taken to complete the task. Other metrics may be useful later, but for now we just want to track time taken since time is our most limited resource. The trouble is, I don’t know of any really good time tracking tools. Here’s the rough requirements:

A long time ago I made some comments about String interning and Anton Tagunov made some interesting comments. It turns out he was very much right and that I was smoking something…. There are definitely still times when string interning will improve performance, even in multithreaded situations (XML parsing turns out to be one) but my comments on threading and synchronization should probably be ignored unless you’ve got the mythical hardware I had in mind when talking about it. Essentially, to achieve what I was talking about you need to be able to add a map entry (not create the entry, just add it) as an atomic operation. You would have to be able to replace the existing map with a new one as an atomic operation as well (don’t add the new entry until the expanded map is in place), however with multi cpu systems, such assignments are likely to wind up in a single processors cache and not be available to other processors. You’d need the ability to tell the processor to put this straight into RAM (reads could still come from cache but the main memory version would have to be checked in a synchronized block before creating a new entry). In Java it is definitely not possible to tell the CPU to put just one variable directly into main memory and I haven’t found any reference to this algorithm even theoretically working on any common computer system. Shame, it seems like such a good way to do it…. Good sounding design trumps working design right?

Byron commented on ampersand redux:

Yes, an ampersand is valid as part of an attribute value (as represented in an HTML document) where that ampersand is part of an entity reference. An ampersand that is not part of an entity reference is not valid in an attribute value, in an HTML document. Serialization has nothing to do with it, since an HTML document is not the serialization of a DOM tree, although it can be viewed as such. I did not mean to say anything about serializing attribute values, I meant to say that an attribute value in an HTML document cannot legally have an ampersand that is not part of an entity reference. If your document does have such an ampersand, it will not validate. It might work in current browsers, but down the road it might not. Don’t do it. If a browser gets it wrong, file a bug against the browser or avoid ampersands entirely, don’t force every other author of HTML parsers to work around your markup’s faults. I still disagree with the first part – ampersands are perfectly valid in HTML comments but when serialized they must be escaped as entities. It is critical to consider entities as equivalent to the character they represent, otherwise é wouldn’t be the same as é which is clearly ludicrous. Regardless, the point is entirely academic so I’ll leave it at that. The last part however is crazy. If a browser has a bug and you need to support that browser, you should do whatever it takes to make your application work with that browser – standards be damned. It is in no way acceptable for a software developer to skip requirements just because it would mean conflicting with a standard. If adhering to the standard was also a requirement then the higher priority requirement should wind up being implemented and the other one revised to not be in conflict. If you can get the browser vendor to fix the issue and you consider it acceptable to make all your clients upgrade to the fixed version then by all means follow the standard – otherwise through it out. Standards are designed to enhance interoperability, if they reduce interoperability in areas that are important to your project they are completely worthless and should be ignored. The comment about forcing every other HTML parser to work around the markup problems is a red herring as well – HTML parsers already have to deal with that kind of thing and that’s not going to change. XML parsers on the other hand do not have to handle invalid mark up and most don’t which is precisely why I pointed out that you should always escape ampersands correctly in XHTML despite the fact that most if not all browsers will get it right either way. Software development is about achieving the project’s requirements. It’s not about politics, it’s not about standards and it’s not about making yourself feel good. If you can meet your requirements and do any of that, then great, but the requirements are the only thing that have to be achieved and they override anything else. That said, any of those things could be made a requirement of the project, but it’s quite rare that they would actually be requirements let alone high priority ones.

It seems I wasn’t clear enough with my ampersand related comments. I’m not talking about standards here, the standards are very clear – & should always be escaped as &, no ifs no buts. However, we live in the real world and many things don’t follow standards correctly. So while David is correct that the validator will complain if you don’t escape ampersands in HTML documents, some browsers will get it wrong if you do escape them in some cases (it’s exotic and the actual test cases are at work not here unfortunately). In XHTML however, you really seriously have to escape them because a) browsers get it right when kicked into XHTML mode, and b) XML parsers barf if you don’t. Byron also chimes in with a comment:

If you wanted to create a hyperlink to a file called “Me & You”, which of the following should you use?

<a href="Me & You"> or 

<a href="Me &amp; You"> In other words, should you escape the ampersand or not? It depends. If you create a plain HTML page, you must not escape the ampersand or it won't work (browser dependent obviously), however if you leave it unescaped it will work in every browser. If however, you create an XHTML document you should escape the ampersand, otherwise XML parsers will break when parsing the document and browsers will get the link right as long as they are kicked into XHTML mode by the appropriate declaration at the top of the file. If you want to test this try linking to an URL with 

& in it (ie: the file name literally includes the HTML entity for ampersand). Better yet, don’t put stupid characters in your URLs.

This blog is a very informal place for me – I write what I want, when I want, how I want. Thats why I have a blog. Recently, Greg Black took issue with my (admittedly fairly regular) misuse of the apostrophe. I am actually quite familiar with the rules of when to use an apostrophe and when not to, however since this is informal writing, I tend not to proof read me comments and also tend to think much faster than my fingers type. If this were formal writing not only would I proof read my comments to ensure that apostrophe’s were used in the correctly locations but also rewrite those long sentences with many sub-sentences in brackets (I have a bad habit of doing that – it matches my though patterns), and of course the over use of dash’s to tack on additional points. I would probably even go so far as using paragraph’s to delimit separate points instead of using them whenever I feel like some white space is required. Heck, I might even run a speeling checker over it. Its also worth responding to Gregs comment:

(In case you haven’t noticed, it seems to be a very work oriented evening this evening.) I’m often a little envious of people who get to work in cool places developing brand new technology and speaking at conferences (and more importantly having more than a handful of people actually care about that area of development). I’ve been in the same job for about 3 years now and while we are and always have been on the very cutting edge of content related technologies (see, not even a cool name for it…), it’s a little bit old hat to me. I’ve beaten my head against all kinds of standards, HTTP, HTML, CSS, XML, Namespaces, XPath, XSD, XSLT, Word – if there’s content written in it, I’ve probably had to deal with it at some point and if it’s at all web related I’ve probably had a lot to do with it. Now I don’t mean to say that I know everything and I certainly don’t want to imply that I’m any more knowledgeable than anyone else – quite the opposite, I still have a lot to learn and there are a lot of people that I’m constantly learning from. What I am trying to say however, is that as a product becomes more mature, the coolness factor of it’s development tends to wear off. 5 years ago, the ability to replace a standard text area on a HTML page with a WYSIWYG HTML editor was nothing short of astounding. These days most browsers have (very) primitive WYSIWYG editing modes built in. In the past week or two however, I’ve gotten my teeth sunk into some awesome new features that once again have me really excited about the technology space I’m in. With the features I’ve put in during the last week or two and a couple of the features that will go in this week and next, our boring little editor is rocketing forward in usability. When Ephox started making it’s editors the general practice was to look at Word, FrontPage and DreamWeaver to see what they did and how they did it then try to find a way to make that possible within the confines of a browser. Now I’m looking at how those programs handle things and finding them lacking and quite buggy. I used to think their behavior was how it was supposed to work and that working out what the user meant was just difficult – I’m really excited to have discovered that’s not the case: most programs are just really buggy and make it look hard. Now I’m not about to suggest that our product is completely bug free, it’s not – it has a lot of room for improvement (and I’m excited that we’re really focussing on making those improvements happen), but when you look back over the past few years and see the journey that Ephox has taken to get here and follow the progress of the entire content management industry, it’s really quite exciting that we’ve gotten here. So keep an eye out for our 4.0 release (or whatever the heck marketing decide to call it) when it comes out (waiting on management and marketing to decide what the best time for a release would be and what features we want in it). It won’t be announced on Slashdot and seeing as we don’t really sell to end users and most people don’t read the kind of news sites we do get mentioned on, you’ll probably never notice the release, but I’m telling you – it’ll be awesome. I’m excited.

I don’t use our product outside of debug mode often enough apparently. Having played around with the wiki system I mentioned previously (you do read your planet’s from bottom to top right?), I suddenly noticed that the progress bar our editor applet shows while it’s starting up wasn’t displaying. Turns out the applet was loaded and ready instantly so the screen wasn’t getting a chance to repaint. Awesome! This by the way is only on a 1.4Ghz AMD machine with 512Mb RAM so it’s about the average for corporate desktops these days, maybe a little above and it is running the Java 1.5 beta which provided some massive improvements in start up time. Still, our ActiveX based editor doesn’t load this fast. Even better, I’m using an old version of the applet since I didn’t have a recent copy on my laptop when I came home tonight and couldn’t be bothered downloading a new version. It should be a faster still with the performance improvements we’ve put in.

I’ve been playing around with writing a mini-wiki that uses the full compliment of HTML as it’s syntax (instead of forcing me to learn yet another markup language) and use EditLive! for Java as the editor – eating ones own dog food and all that. Frankly, that’s the way a wiki should work, no messing around with mark up at all, just simple, easy to use WYSIWYG markup. Anyway, I wrote the back end in PHP since we don’t have any PHP examples in our SDK and I couldn’t be bothered working out why perl refused to install the MySQL drivers. Loading and saving from the database is simple enough, and I settled in to make the CamelCase works hyperlinks. The obvious answer: regex. The obvious problem: working out which regex expression to use (I don’t use regex often since I usually live in the land of custom automatons instead). I’ve wound up with:

Byron complains about what he calls a limitation of XPath. It’s not actually a limitation of XPath at all but rather a very common mistake people make when working with XML namespaces. Lets take a tour into the dark depths of XML namespaces to discover what’s really going on. Originally, XML didn’t have namespaces at all, every element was identified purely by it’s name. So the element <html> was known as html and the world was simple. Then people discovered that they wanted to combine XML documents and that quite often they’d wind up with two elements called html that had completely different meanings and uses – ie: they were actually two different elements. To solve this problem, the clever folk over at the W3C changed the way XML elements and attributes were named. Now instead of a simple string, elements would be named using a QName (short for Qualified Name). A QName is a compound data object consisting of an URI and the regular name we’re used to. Now the important thing to note in this distinction is that it is impossible to refer to an element only by it’s local-name, you have to use a QName to refer to it and that QName must have a namespace attribute (all QName’s do). It is however possible to assign nil to the namespace attribute of a QName, which is referred to as an element in the nil namespace. Now, one of the important things about adding namespaces to XML was backwards compatibility so there had to be some way to assign a namespace to all those elements which were previously referred to only by their local-name. This is where the default namespace comes in (it also happens to be quite convenient). By default in an XML document, any element that doesn’t have a prefix to declare which namespace it is in, is assigned the “default namespace”. The default namespace however isn’t an actual namespace, it’s just a default value for the namespace attribute. By default, the default namespace is the nil namespace. Now, in XML you can specify what the default namespace is by adding an xmlns attribute, eg: xmlns="http://www.w3.org/1999/xhtml". The new value for the default namespace is then in effect for that element and any element under it (unless it’s changed again). It is important to note here that an XML document doesn’t have a default namespace, but rather each element has a value which it inherits (attributes never ever use the default namespace). There can therefore be as many different values for the default namespace as there are elements. So if we were to add Byron’s idea of matching any element in the default namespace, it would never match anything because every element (and attribute) would have an explicit namespace. Worse still, the default namespace would change depending on which element we were in and what the specific representation of the XML was (maybe the default namespace was left as nil and every element used a prefix or maybe no elements were prefixed and the default namespace was changed all through the document). Depending on the representation of XML instead of the actual data it represents is very bad practice and will cause problems. This leads right into the other common mistake in Byron’s post: //*[name() == 'foo'] will do nothing particularly useful. What it will do is match any element with a local-name of foo, in any namespace as long as the element was represented without a prefix. It will not match <my:foo xmlns:my="http://www.intencha.com/my" /> because name() for my:foo will return my:foo. Elements are the same if they have the same QName regardless of whether or not a prefix used or if different prefixes were used. The correct way to select any element with a local-name of foo in any namespace regardless of what prefix was used is: //*[local-name() = 'foo'] If I were to take a shot at selecting any node which had a namespace-uri the same as the default namespace in effect in the original representation of the XML document’s root node it would be: //*[string(/*/namespace::*[name() = ""]) = namespace-uri()] Which is to say:

One of the major challenges in my job is testing our product. Now most people think that testing is reasonably easy but requires discipline, this is not true if the product you write happens to be a styled text editor and it's nearly impossible to do really well if you're working with something as flexibly defined as HTML.

The problems start at the unit testing level. Try taking the standard JTextPane class and writing unit tests for it. How do you test that it can render a HTML list correctly? You could write a test to make sure that the list numbering at least comes out in order and in the right format but that still won't guarantee that the list numbers actually paint correctly. For instance, we recently had a bug where the list numbers painted correctly if the list fit on screen, but if it required scrolling, whatever item was at the top of the screen was numbered 1 even if it was actually the third item in the list. Our list numbering unit tests had no way of picking up on that because it was the actual rendering code (which we didn't write) that was wrong.

Andrae Muys provides some excellent advice on implementing Object.equals() however I do have to correct one thing. Andrae presents the code:

class A {
  int n1;
  public boolean equals(Object o) {
    if (!(o instanceof A)) {
      return false;
    } else {
      A rhs = (A)o;
      return this.n1 == rhs.n1;
    }
  }
}

and suggests that it is incorrect. It is not. This is absolutely, 100% the correct way to implement equals for that class. The alternative he presents on the other hand is incorrect:

I have slightly obscure tastes in music – particularly, I like musicals, not the highlights CDs the full recording of the original cast. It’s certainly not the most obscure taste in music but it does lead to an awful lot of trouble tracking down what I want and worse still I know what I want ahead of time unlike most people with really obscure tastes who just stumble across things they like. Coming back to the point though, you can’t just walk into HMV or pretty much any music store that I’ve found and pick up a copy of the original 1986 cast recording of The Phantom of The Opera or Miss Saigon or Les Miserables. However with this new fangled technology intarweb thingy I can head on over to Amazon and order it from there. There’s a bunch of other online stores around that may or may not have what I want for prices roughly equal to Amazon but what I love about Amazon is watching it try to predict my buying habits. I know most people freak out about privacy violations when computer systems start gathering data about them but with Amazon it’s like a fun game. It managed to pick that I was looking to purchase Miss Saigon and The Phantom Of The Opera the last time I went there and offered a package deal on them. It’s recommendations are also really quite good – including detecting that I tend to buy the versions that Lea Salonga is in (she tends to be part of the original cast of a lot of big musicals). Sadly, it doesn’t seem to have worked out that I only buy CDs from them as it keeps offering books and sheet music. While I do tend to buy a fair bit of sheet music of musicals I won’t purchase it without first flicking through it to make sure I have a chance of being able to play it. The big downside of buying from Amazon though is I have to wait two and a half weeks for things to arrive (that or pay an extra arm for postage).

Tor Norbye politely requests that the caps lock key be removed and the control key put there instead. There’s one very good reason why that shouldn’t be done: Everyone (except old school UNIX geeks) is used to the control key being where it is. Moving the control key would seriously annoy people. If you’re one of the people who are used to control being next to ‘a’ then imagine the whole world being as annoyed as you every time they use a computer and find that control is in the “wrong” place. More importantly though, putting control beside ‘a’ isn’t a good place anyway. The little finger is the most difficult finger to control on the human hand and is used least commonly. In touch typing, currently the left little finger is positioned over ‘a’ and moves up for ‘q’ and ‘z’. If you’re British or Australian, ‘q’ and ‘z’ are incredibly uncommon letters (American’s customi_z_ed their language by putting a bunch of Zs in). Now think of the most common keyboard shortcuts used on computers these days (think Windows users, not emacs users):

There seems to be a growing trend for projects to use XML configuration files – fine. There seems to be a growing trend for those projects to provide a schema for those files – good. There seems to be a growing trend for those projects never to validate their configuration files against the schema – bad. As I’ve previously mentioned, my job involves creating an XML forms editor and it turns out that this forms editor is really quite good at editing configuration files (see our very own configuration tool). We thought it might be nice to create a simple editor for Maven POM files. Sadly, it seems that the schema for a POM file doesn’t come anywhere near close to describing what should actually be in a POM. Maven has support for inheriting a POM and using the information in it, thus allowing that information to be omitted from the POM itself. Now admittedly, it’s not possible to specifically describe this in XML schema (the POM being inherited from can omit any information it likes as well assuming that it will be filled in by the extending POM), but the current schema insists that everything be specified in every POM file which makes validation completely useless. The POM files from the plugins don’t validate against the schema for a number of reasons as well. JDNC is even worse – Xerces finds errors in the schema itself (and I’m fairly sure Xerces is correct). This, combined with fairly poor documentation, makes it extremely difficult to implement tool support. It’s a shame, I’d probably use Maven if it wasn’t such a pain in the neck to create the POM file correctly (that and dependency management which is wonderfully simple for things in the public repository and annoyingly difficult for things that aren’t). Maybe I’ll come back and create a more useful schema for Maven at some point, but updating the schema doesn’t really help me identify the areas of our product that need improving.

Anton Tagunov added an excellent comment to yesterday’s entry:

The one thing that has always stopped me from doing this was: interning must use some global lock. So, while interning would cause little to no harm for desktop application it is likely to introduce extra synchronization bottleneck for server applications. Thus potentially degrading perfromance on some multi-cpu beast. Would you agree with this, Adrian? Firstly, let me make something very clear: string interning will cause bugs in your code and they will be hard to track down – they will however be easy to fix once found. At some point, someone, somewhere will forget to intern a string and let it pass into your code that assumes strings are interned. Also, string interning is an extremely situational optimization. In most cases, it will worsen performance because the overhead of interning the strings will not be made up for by the reduced complexity of comparisons. Even of the cases where it does help, most of the time the difference will not be noticeable. As always, don’t bother optimizing until you know that you need to and that it will help – this is an optimization that causes the code to become less maintainable. Having said that, lets go back to the original question. Firstly, does string interning require synchronization? Probably, but not in terms of Java. The String.intern() method is a native method and works via JNI. It would be difficult to imagine a way of achieving the behavior without at least some synchronization though. The synchronized block however would be very small, and very rarely encountered. There are two situations to consider, either the string is already in the interned list or the string is not. If it is, then no synchronization needs to occur because the list is only being read. So multiple strings can be interned at once so long as all of them are already in the interned list. Synchronization will be needed however whenever a string is interned for the first time (ie: it doesn’t match any String constant that has been loaded or any previously interned string). So on a multiple CPU system, it would be very bad to intern a lot of strings that are only ever used once or twice as they would require a lot of synchronization for no benefit. Of course on a single CPU system, doing this would be a bad thing anyway because it would incur the extra cost of comparing strings to check if they match an interned string without gaining any real benefit. My theory would then be (and only real world application profiling will confirm this in any particular situation) that the string interning technique is slightly less likely to pay off on multiple CPU systems, however because the situations in which string interning is useful require that the vast majority of String.intern() calls match something already in the cache (most likely one of the string constants they’re to be compared against) the question of how many CPUs will be in use isn’t going to have any significant impact. I can’t stress enough though that if you don’t have specific profiling data that shows String comparisons as the biggest bottle neck in your application, you shouldn’t apply this optimization. Great question. UPDATE: Here’s an interesting discussion of interning relating specifically to this question. The automatic google search on the side (if you actually click through to this blog entry) is very handy at times.

Elan Meng investigates the behaviours of string constants, interning and == compared to .equals. Very informative. The question remaining is, why would anyone ever use == instead of .equals on a String considering how likely it is to cause confusion and the potential for disaster if one of the strings for some reason isn’t interned. The answer is performance. In the average case, the performance difference between == and .equals is pretty much non-existent. The first thing .equals does (like any good equals method) is check if the objects are == and returns true immediately if they are. The second thing .equals does is check that the strings have the same length (in Java the length of the string is a constant field and so requires no computation). If an answer still hasn’t been found, the characters of each String are iterated over and as soon as they differ, false is returned. Now, consider the possible cases:

This is a great example of getting stuck in a mindset. A piece of very poorly written Java code is presented followed by a much shorter piece of Groovy code and Groovy is declared the winner.

The original Groovy:

list = ["Rod", "James", "Chris"]
shorts = list.findAll { it.size() <= 4 }
shorts.each { println it }

Java:

for ( String item : new String[] {"Rod", "James", "Chris" } )  if ( item.length() <= 4 ) System.out.println(item);

oooo, one line! It must be good… Of course if I were actually going to write that, I’d write it as:

Continuing my journey of learning regarding scripting languages (starting here):

I like it because it saves typing. There was a good example of a 20-line Java program that was reduced to 3 lines in Groovy. I’ve lost the link though. Posted by: Jonathan Aquino I like readability. Java can definitely be overly verbose at times, particularly with IO:

BufferedInputStream in = new BufferedInputStream(new InputStreamReader(System.in()); however, number of lines required to do something isn't something that I really consider when choosing a language. Obviously if it's at either extreme it matters, but few languages are either so short or so long that it really impacts readability. I'm also far more concerned with maintenance time than with development time because maintenance time almost always outweighs development time. Having said that, many people do like really compact languages and some scripting languages provide that (HyperTalk certainly doesn't). Greg Black adds some 

interesting comments as well. I certainly see the benefits of scripting for quick and dirty solutions or small programs as I tried to point out. Greg also quotes me as saying:

As much as the title seems to suggest one of my rants, this is actually a valid question along with a bit of my own pondering. Scripting languages seem to be the flavor of the month these days and I’m not really sure why. I’ve got nothing against scripting languages but I don’t see why they should be considered the be all and end all solution that people seem to think they are. Interestingly, when I first seriously got into programming, it was using HyperCard and there was a constant barrage of insults coming from the “real programmers” about these hobbyists using scripting languages. More than ten years later and all of a sudden you’re just not groovy (pardon the pun) if you’re not using a scripting language. I love being able to write a quick perl script to munge a text file in an odd format or to run through the Xalan codebase and change the package so that it doesn’t conflict with the version in the JRE. Our support auto-responder at work is a cool little perl script that I wrote to take the incoming (evil MS HTML formatted email) that comes from the web form, parse it, log the details in our tracking system and fire back an email to the user with the tracking number. Works a charm. It would be a real pain to write that stuff in anything but perl because of perl’s awesome support for text parsing and abundant if unwieldy and occasionally unreliable libraries in CPAN. I’ve also written a major business system in perl with database interaction, workflow and all that jazz. It worked well but it was certainly no easier to do it in perl than Java or most likely C given appropriate libraries. I wouldn’t consider C an option unless performance was absolutely critical for server systems however because it leaves open the risk of buffer overruns and similar security holes that can be completely eliminated automatically by most other languages. Even if speed were critical I’d recommend buying faster hardware or using a distributed system before writing a server in C. I’ve also written little scripts and smallish sites in PHP. It’s a nice language that I enjoy using but again I don’t see anything hugely wonderful about it. Everyone seems to be very python oriented these days and I must admit to having almost no knowledge of the language but from the code I have tried to modify in python I really don’t see any reason to be overly excited about it. Again, there doesn’t seem to be anything particularly wrong with it, but I don’t see why it would be so much better than C, Java, Visual Basic, C# etc. I also use JavaScript a lot at work and use it as a full programming language, not just to do roll over graphics. It can do some cool things but the resulting code is far from easily maintainable and again, I don’t see the advantage other than it’s the only option for code-in-a-browser when working cross-browser. The most common reason I hear people giving for why they like scripting languages is because they “just flow better”. I just don’t buy that. I grew up on scripting languages and I just don’t find that they flow any better than any other languages. They do tend to be easier to learn because you get to ignore most of the rules of good programming while you learn (think perl without the use strict directive). If you want to write good code though, you should put that use strict line back in and pay attention to all those little details that make initial coding harder but maintainability easier. Once you’re thinking about all those little things I find scripting has the same feel as “programming”. So am I abnormal or am I just missing something? Maybe it’s both….

A while back Kyle Rankin questioned why people would use InstallShield under Linux. He suggests people use the standard package management schemes that the various distributions provide and he’s dead right. You should make things consistent for the user because it’s consistency that makes a user interface easy to learn and makes it more productive (once you’ve learnt it once you can use it in a whole heap of places at once without taking the time to think of the right way to do it). There is however a problem with this. There are just way too many linux distributions. Even taking into account the fact that there’s a lot of overlap in package management tools, it’s a big ask to expect a company to provide different packages for all of the different systems. Given that, they now have two choices:

On a more positive note, if you need to work with the Swing Text APIs in any detail,would like to do something with text that you currently can’t or for some reason are implementing a text API, take a look at this overview and the articles here (the text articles are towards the bottom). Some of the docs seem to be fairly old but the basic features of the API really haven’t changed all that much and it’s far more important to understand the design than the specific methods that are available anyway. Once you understand the design, you’ll know where to look for the methods.

I’m getting really tired of the amount of bigotry in regards to programming languages. First let me admit that I’m certainly guilty of technology bigotry – I’m definitely biased towards Mac OS and Java to name a couple of things, but I’m at least aware of it and willing to admit it. Furthermore I can provide reasonable explanations as to why I like them if not actually explain in every situation why I favor them (read: at times I’m just being bigoted about it). When it comes to programming languages and in fact most things in technology, there are few things that are not the best choice in at least some situation – even if it is only one incredibly specific situation. This is even more true when you only take into account the things that are widely used. Lately Java has been coping it (more than usual) with people actually booing at demonstrations that used Java and a lot of fairly immature comments being made. Lets take the view point that Java is a pathetically hopeless, good for nothing language. Everything about it sucks, the people who use it are obviously idiots and it’s too slow to do anything useful with. Obviously the company I work for made a massive mistake implementing their product in Java and at long last we’ve realized it and want to reimplement it in a good language (we’ll probably have to fire all those stupid java developers too but one step at a time). Lets look at the (highly informal) requirements for these products:

I ranted earlier about dependencies and the way Java programmers are always pining for the latest and greatest. The comment by Stephen Thorne to the article deserves being published with the same level of visibility as the original post so I quote it below. It also deserves some rebuttal which is also below.

I’ve been saying this for years, and yet I still run into rabid pro-java programmers who managed to rattle off a list of reasons why java is the bestest programming language in the world include “good library support” to which my response is “hold on, slow down, good library support? let me tell you a story…” and I recite any one of a dozen anecdotes about dependancy hell in java. The only java project I’ve seen that manages to avoid this without extreme pain is kowari, which takes the slant “We’ll distribute the ENTIRE DAMNED THING as a .jar, libraries included, just make sure you have a recent enough JVM.” They don’t have the windowing troubles because its a database. ;) The first thing I want to correct is the implication that java doesn’t have good library support – it absolutely does. Look around at the huge range of libraries available for Java and particularly look at the fantastic standard library that comes with it. Plenty of top quality libraries for pretty much every moderately common task. There is however a separate issue of dependency management. Problematic dependency management (aka dll hell) is caused by two things:

Greg Black comments on the IBM donation of code to the ASF (at least I think that’s what he meant). Ken Coar has already provided the explations. For the record Greg, your site just gives me “access denied” so I can only read your blog via Planet Humbug and can’t leave this as a comment.

James Strachan puts forward a proposal that Sun opensource the standard Java libraries and I see no problems with it. In general I don’t think we’ll see all the benefits James predicts from it though we would see some. In particular I want to point out one benefit that James claims we’ll get but that we definitely won’t get:

more eyeballs are now looking closely at the code This is the biggest advantage opensource proponents put forward for opensourcing any code but it just doesn’t apply to Java. Why not? The source code to the standard Java libraries are included with every copy of the JRE. You can already go and inspect the source code, find bugs and submit patches to Sun. The fact is, most people can’t be bothered. They have real work to do and don’t want to be wasting time analyzing the source code for the library their using, they just want it to work. When it doesn’t work then people tend to turn to the source and people are already doing that. I also don’t believe that people really want to write Java code that can run on the .Net platform. Java has always suffered from people feeling that it was second rate due to it’s cross platform nature. This is true to some extent, when writing any cross platform code you inherently make it harder to use the platforms native resources due to the extra layer of abstraction. Why would you use the .Net platform when all of the extra functionality it would provide (like tie-ins to word, infopath etc) wouldn’t be available from Java (because they’re not cross-platform). You’d have to use JNI or similar to call “native” .Net code and so you have to learn C#, the .Net libraries and their toolsets etc. Also, opensource developers who are refusing to use Java now will refuse to use Java then. Read some of the blogs coming out of OSCON at java.net and you’ll realize that the majority of the opensource world is just unreasonably bigoted against Java. That’s okay though, I’ve grown to really dislike the GPL anyway.

This article entitled “Great Hackers” really pissed me off. It is just so far off-base that it’s annoying. It’s not the kind of article that is generally wrong on facts, but just wrong on intentions and the concepts of what’s good and what’s bad. There are a few really, really stupid and outright wrong statements in it though. The worst of these is:

The programmers you’ll be able to hire to work on a Java project won’t be as smart as the ones you could get to work on a project written in Python. That statement referenced a footnote as well which I thought might point to some scientific survey of Java programmers vs Python programmers so that the statement at least had some backing. Sadly, the footnote was:

I’ve never understood Java programmers attitudes to dependencies. The Java runtime and libraries was designed specifically to allow cross-platform deployment and increase compatibility. Write once, run anywhere. What confuses me then is this obsession Java developers seem to have with being incompatible or rather, using the absolute bleeding-edge, never released, not available anywhere version of stuff – particularly the JRE. Now certainly Java is unusual in that it is rapidly developing. Not only that but the language, the runtime and the libraries are all tied in together so if you want to live on the bleeding edge with one you have to live on the bleeding edge with them all. Contrast this with the C world where the language is effectively static, the standard library is minimal and evolving relatively slowly and the “runtime” (I’m thinking of the ABI) is fairly static (if platform specific). It’s not often that C programmers have to make a decision to update to the latest version and it rarely impacts deployability if they do. Most of the time if a C program requires a specific version of a library it just statically compiles it. In the Java world however, people are always pining for the latest enhancements. Suddenly generics are a huge deal when everyone’s been getting by without them just fine for years. Suddenly it’s too much effort to work around bugs in the libraries and it’s better to just force users to upgrade. Frankly, that’s simply not good enough. If I develop an application for Windows, I have to decide which versions of Windows I’ll support. I could use APIs that are new to Windows XP and thus tie my application specifically to it, or I could not use those APIs and let any Windows version run it. Better yet, I could design my system in such a way that I can optionally use those APIs if they’re available and fail gracefully otherwise. C programmers are really good at this, most likely because they’ve spent so much time trying to write C code that’s cross platform (it’s possible, but you have to be good at doing this kind of thing). Unlike C, Java actually makes it easy to optionally use APIs because everything is dynamically bound. Consider the class:

I’ve been following the JDNC project recently and along the way mentioned that it might be nice to combine the authorization dialogs for WebStart applications that have multiple jars signed by different certs into a single dialog to make it more user friendly. Mark Davidson who seems to work at Sun (I assume on the Swing team) promptly responded that he’d spoken to a WebStart engineer about it, logged an RFE for me and said the WebStart guy was going to talk to the security guys about it during the week. That’s of course no guarantee that it will actually be implemented anytime soon (or any time at all) but it’s nice to at least know that the message got through. Particularly impressive is the fact that I wasn’t even trying to get a message through, just spouting ideas off the top of my head. It’s generally been very good to see the Sun engineers working to involve the community on JDNC, I had initially thought that JDNC would be a “token” opensource project where Sun employees still control everything but make the source code available. I’m very glad to say that I was wrong about that, there is definitely a strong effort being made to get people involved and already a number of knowledgeable folks from outside are getting involved and providing some very good advice.

The story so far:

1. Preston Gralla commented
2. I commented
3. Brian McCallister commented
4. I commented again
5. Brian McCallister commented again At least I think that’s how it went. Firstly, Brian was right to call me on my use of cygwin to bring UNIX capabilities to Windows. It’s not in the default install, it’s not at all obvious and 99% of Windows users will never even hear about it. As Brian says, “if you don’t use it, you don’t learn it”. So if we conceed that the command line is a killer attraction then Linux has it’s big advantage over windows. That’s where I think Brian and I may disagree. First let me start by saying that the command line is great, it’s an incredibly powerful tool with a lot of really great advantages. It does however require a lot of learning and it’s not once off learning either. The command line requires you to constantly learn – every new task requires research to find out what command does what you want. Then you have to constantly remember all these different commands so that you can use them when you need them. Nothing is obvious, nothing is intuitive. Everything is powerful.

This is a paradigm thing — the drive to ubiquitize computers required them to have an interface comparable to that of a toaster. Now that they are ubiquitous, lets bring back the idea of a powerful interface. Please. I agree that we need to make user interfaces more powerful and let people do more with their computers but don’t throw the baby out with the bath water. People are no more capable of learning interfaces today as they were 15 years ago. The ubiquity of the GUI does not make it easier for people to learn the command line, in fact it makes it harder due to the unlearning required. Having a command line available with powerful tools is great for advanced users that want to get into that but its still not an option for the vast majority of computer users because they will never get enough benefit out of it to justify the learning cost. Furthermore, the learning cost will be excessively high for casual users because they will continually forget the commands that are available to them. So how do we reconcile the two goals – having a simple interface and providing full power to advanced users. Most people will suggest creating two interfaces, one for novices and one for advanced users (the novice interface is usually called a “wizard”). This is outright bad user interface design. Jef Raskin provides the best argument I’ve seen on why this is bad user interface in The Humane Interface but sadly I don’t have a copy at hand to give an exact reference. Essentially though the argument is that instead of users having to learn one interface, they must now learn two to be able to use the software. When they first start using the program they are a novice and learn to use the wizard interface. Then they become more familiar with the program and consider themselves advanced so they switch to the advanced interface. Unfortunately once they change the interface they are no longer advanced users – they are completely new to the interface and are in fact beginners. All the learning they did with the beginner interface is worthless and they have to start from scratch with the advanced interface. Worse still, the advanced interface will almost certainly have been designed with “these are advanced users in mind, they’ll work it out” in mind and is thus much more difficult to learn that it should be. The other big problem with having two interface modes is the amount of extra developer time that is required to achieve it. That time could have been better spent making the advanced interface easier to learn. How does this relate to the GUI vs Command line debate? Firstly it shows a weakness in interfaces like Linux where you can do a lot with the GUI but quite often have to switch to the command line, as well as a weakness with Windows where you can do a lot with the command line but often have to switch to the GUI. It’s also a weakness with OS X both ways (some things are GUI only, some things are command line only). More importantly though it explains why we can’t expect people to learn a command line interface now any more than we could when computers first got started. So how do we make things more powerful while keeping the baby firmly in the bathtub? The first thing I’d point to is AppleScript which is an awesomely cool way to bring some of the power of the command line to the GUI. The ability to pipe one program into another is realized through AppleScript and in fact extended much beyond what the command line pipes can do. AppleScript is shell scripting for the GUI. AppleScript however is difficult to learn and the language is awful but these are implementation details – the idea itself is still sound. The biggest problem with the AppleScript concept though is that you effectively always have to write a shell script which involves firing up the script editor. Too slow. What if we mixed the concept of the GUI and the command line together though? Most of the time you’re in the GUI just like normal because it’s easy to use and for the most common computing tasks it’s the most efficient way to do things (how many sighted people surf the web exclusively from lynx?). When you need the power of the command line though you hit a key combination and a command line pops up to allow you to write AppleScript snippets (though in a more intuitive language that AppleScript). Oddly enough, HyperCard contains pretty much this exact interface. If you hit Apple-M in a HyperCard stack, the message box pops up and you can enter any command you like to control that stack, another stack or even execute an AppleScript. One key thing here though is that it’s not a terminal window that pops up, it’s a floating window that by default operates on the current application. So if I’m in Microsoft Word typing away and I think to myself: “I need to insert all the jpg images of my charts into this appendix” today I would have to click “Insert->Image…->From File…->chart.gif” however many times but with the built in command prompt I’d just hit the magical key combination to bring it up and then “insert /Users/aj/Documents/charts/*.gif” and let Word do the rest. Note that insert would be an AppleScript command defined by Word and tab completion is a necessity. Similarly, if I wanted to attach a zip archive of a particular folder to an email, I’d bring up the command prompt with a keystroke and enter something like “attach `zip /Users/aj/Documents/emailDocs`” or better “zip /Users/aj/Documents/emailDocs and attach it” which is much more HyperCard like. That scheme combines the power of command lines with the power and simplicity of GUIs. Coming back to Brian’s comments though:

Brian McCallister comments on my earlier comments on Preston Gralla’s comments on Linux on the desktop. By and large I agree with Brian, the UNIX command line is a sensationally powerful thing which provides awesome flexibility and power for those who wish to learn it. The downside is it’s awful trying to learn it. I spend a lot of time at a bash command line and I still couldn’t tell you off the top of my head what Brian’s examples do. They’re simple and straight forward to him because he uses those tools every day, I don’t so they’re very foreign and require learning (I use different command line programs). GUIs have a major learnability advantage because the options are (or should be) visible to the user. More importantly though, in the context of Linux on the desktop the power of the command line disappears to a very large degree. OS X is a very good example of this because it is UNIX on the desktop and you find that most people don’t use the command line very much if at all. Mostly that’s because they can’t be bothered learning it and because they typically don’t have a need for the power it provides. The key point though and it was the main point of my arguments is summed up so well with Brian’s comment:

Coolness. I’m definitely going to have to take some time and learn about JOGL at some point. I was amazed at how smooth the graphics were, not to mention the seamless, click-a-link installation – even on my Mac.

Preston Gralla’s comments on how Linux didn’t impress him too much really got me thinking. Preston didn’t bash Linux or try to argue that Linux was inferior to Windows – he just pointed out that he can already do everything he wanted to on Windows and didn’t have any problems with it, so why change? That’s Linux’s big problem. It’s biggest feature has always been stability and security. In other words, it’s biggest feature is that it doesn’t have Windows bugs. There’s a curse on depending on being better than your competitors by having fewer bugs though – eventually your competitor fixes their bugs. Lets assume for a moment that Linux is perfect software, it has no bugs, no security flaws, never has and never will. This is clearly not the case but lets work with the best case scenario for Linux. Windows starts out as barely usable because of all the bugs in it. There’s barely a single piece of functionality that isn’t affected by bugs and users constantly have to keep in mind how to work around bugs while they use the system. Again this has never been the case but it is the best case scenario for anyone competing with Windows. Now, what happens when the next version of Linux comes out? It has no bugs. Great! What happens when the next release of Windows comes out? It has fewer bugs. Most applications become more stable as they mature even if they add more features at the same time because in each release you typically add a few features (which are probably buggy) and fix a bunch of bugs in existing features. Windows certainly has become more stable over time, though there was that ME release which might have been a step backwards… So here’s the curse of Linux: even in the best of worlds, Linux’s biggest feature is eroding out from underneath it and will continue to do so until it is negligible. You simply can’t survive forever on a product that is better because it has fewer bugs – the competition will always catch up. You have to add features to differentiate yourself and they have to be innovative – really, truly, oh-my-gosh-that’s-awsome, I-never-would-have-thought-of-that innovative. Linux doesn’t have that kind of innovation from anything I’ve seen. Linux was created as a clone of UNIX and a Linux command line still looks and acts pretty much like every other UNIX out there. It has the same basic set of commands, similar programming APIs etc. Fortunately, desktop users don’t care about any of that. Unfortunately, Linux’s desktop environment doesn’t show any real innovation either. If I were to describe my Windows desktop when it first boots up I’d say something like:

Mark Pilgrim posts an interesting article entitled XML on the Web Has Failed and he’s right to some degree. Character sets remain a huge mess on the internet, but I think he’s pinning the failure on the wrong technology. It’s not XML that’s failed, but RFC3023 which specifies a set of rules for detecting the XML charset when combined with HTTP. The reason RFC3023 fails is because noone likes the way it works and it’s just not implemented anywhere. The one part of the specification that causes problems is what to do when XML is transferred over HTTP with no Charset header and a Content-Type of text/xml. The one reason that rule is so screwed up (it says to ignore the charset in the XML file) is because a bunch of proxies translate the character encoding without knowing anything about the content being transferred (except that it’s text/something). So what’s the solution? Put some common sense back into the mix, if there’s no charset in the HTTP headers and a charset declared in the XML file, use the charset from the XML file, then fix the proxies that are destroying content – they’re probably destroying a lot of HTML files as well since they wouldn’t pay attention to the content type specified in a meta tag. Claiming that XML has failed is throwing the baby out with the bath water, the problem is just that there’s some stupid proxies doing things that, while currently allowed, are pretty obviously going to destroy content at least some of the time. So I propose a very simple solution to this problem. Add one new rule to the HTTP spec:

During the week I wrote some documentation to help people write XSLTs that work really well with EditLive! for XML. Generally any given XSLT will work but there are some techniques you can use to make them work better with the augmentation process we use to add editable fields into your XSLT’s output. Our official document writing took exception to one sentence about where the best place to put “action buttons” is in an XSLT. Action buttons are clickable things that are a cross between a hyperlink and a button which perform operations like adding another element or attribute or moving things up and down in the document. The sentence was apparently too confusing but contained a subtle but important point which couldn’t just be removed. I still think the clearest way to phrase the sentence was as a haiku: most cases intuition used button works Sadly, they won’t let me put that in the docs. So now the challenge goes out – what is the subtle but important point that the haiku so eloquently reveals? Maybe if enough people get it I’ll be allowed to put it in….

Well after a being so excited about getting my ADSL connection, my ADSL modem decided to crap out. Every so often it would just lock up hard – all the lights off except the power light – and stay like that even when power cycled. So yesterday I ordered a Billion 7100S which arrived today. It seems to be working after a slight bit of worry that it wasn’t getting an IP (turned out I’d turned the DHCP client off instead of the DHCP server). So now I’m happily *and* reliably connected to the interweb thingy again. Yay for that.

Wireless Driver will definitely work on OS X 10.3 (Panther) and also claims to work for 10.2 (Jaguar) and 10.1 so it should cover all your needs. Simply run the installer then either kextload the extension or just reboot and specify the network to connect to in the new “Wireless Config” control panel and configure the new “Ethernet Interface” that will appear in the network control panel.

We finally have ADSL at our new place. The world has resumed revolving at normal speed.

When did this happen? Google doing image ads? This doesn’t bode well for the future of man kind. Fortunately:

We currently have no plans to show image ads on Google.com. But that “currently” still concerns me.

Well I figure if you’re going to do something you may as well do it properly and I’ve been getting into the whole PGP thing lately. The trouble is noone has signed my key making it pretty much worthless. I’m also a long way away from everyone I know who uses PGP and fate seems to work against me when I’m closer to them. Bertrand was in Australia recently and even in Brisbane, unfortunately I was over in San Francisco (btw Bertrand if you manage to read this, my old email is offline at the moment so I can’t find your contact details to let you know that I’m back in the country but I figure you’ve moved on from Brisbane anyway). Meanwhile, when I was in San Francisco I missed the train and didn’t make it to the first Apache gathering which had people present who used PGP. I made it to the second but I don’t think anyone there used PGP so there was no key signing done. So I figure I should check out a certificate authority and at least get them to sign my key. CACert is the new, free CA service and since I think it’s about time someone provided a free CA I wanted to support it by getting involved and if the service stacks up with money. It seems however, that there’s no trusted authorities in Brisbane yet. Fortunately they do have a more complex process involving faxing IDs around and finding a couple of lawyers, accountants or bank managers to verify your identity. Since I happen to know a few lawyers and accountants (oh, the company I keep….) that might just work out. So if you’re a lawyer and see me coming with a bunch of paper work, you know what it’s about. With all the viruses impersonating people these days, not to mention actual identify theft (which I’ve personally been a victim of before, though fortunately there were no serious consequences), it’s beginning to get more and more important that your identify can be accurately identified online. The first step is for lots of geeks to get involved, then for more email clients to have builtin support for PGP and make it easy, then we should see much more widespread adoption. Hopefully at some point it will be so ubiquitous that ISPs can refuse to relay email unless it’s signed with a registered PGP key and spam is no longer a problem (though email server resources would be under much higher load so it’s unlikely to actually be done). Client side filters would definitely be an option though.

Technology is wonderful sometimes but it can have serious drawbacks. Having recently moved house and finally managed to remember my new address and find out what my new phone number is I set out to send my new contact details around to friends and family. It occurred to me that attaching a vcard with all my details might make life easier for some people to get in sync with my new details. The trouble is if you attach a file to an email people inevitably think it’s important and just have to open it. So my email ends with the words:

John O’Conner raises a good point – Opensource projects tend to have poor documentation. Java tends to have good documentation (IMO, the best documentation I’ve seen for a programming library). Will Opensourcing Java reduce the quality of it’s documentation? The obvious response is that Sun and IBM etc will still be investing so their doc writers can continue writing the documentation just like now. Except that most of the documentation for Java is written by the same people who write the actual code within Sun – at least that’s the impression I get. If people outside of Sun are writing code it would be up to them to document it – how well would that be kept up if there was no performance review checking up on them?

Once again the Mac world has led the way in pioneering new technology – this time in an area that they are traditionally criticized for the lack of software options: security exploits. Close on the heals of the Safari URL exploits, Mozilla caught up with a shell: exploit and now at long last Microsoft catches up. (Okay it was a few days ago, but that was when I started writing this entry.) My favorite comment from the slashdot thread would have to be:

So I was invited into GMail today, particularly good timing considering I’m about to leave for JavaOne tomorrow and while I’m away I’ll also be moving house meaning that my home mail server will be going offline. I figure this way I can just redirect all my email to my gmail account for the couple of weeks I’m away and see how it goes. If I like it I’ll keep my mail redirecting over to Google, if not I’ll switch back. The biggest complaint I have so far is that GMail doesn’t have any support for PGP which I was just getting into the habit of using. Sigh. Guess that’ll be a feature request. Either way, I’ll get to feel cool because I have a GMail account. I’m just a little surprised it doesn’t come with a Mickey Mouse badge….

So apparently I’m off to Java One. The details are all very sketchy at the moment but apparently the flights involved will be something like: Depart Brisbane on Friday 26th June, to Auckland, then Los Angeles (I think) then “SFO” (can you tell I have no idea about where I’m going yet?). Should arrive at the final destination on Friday 26th June (apparently just before I left). Stay somewhere, do something of which the only certainty is going to the Moscone centre while the conference is on, probably get arrested trying to sneak into WWDC which is also on at the Moscone centre and watching the 4th of July fireworks from somewhere. Fly out on the 6th of July via Sydney (skip the New Zealand connection this time) and arrive back in Brisbane on the 8th of July. So yeah, pretty disorganized…. Any advice/pointers/suggestions/opportunities to meet up would be welcomed. More details as they unfold and I’ll do my best to blog as much as I can while I’m there, this is probably as good a place as any to store my notes.

Every time I see someone talking about the wonders of GPG I think to myself, “I should probably at least have a GPG key” and then I promptly forget about the whole deal. When Bertrand mentioned he was getting into it I went through the same thought process but this time, because of the exceptionally convenient link to an exceptionally conveniently packaged OS X version of GPG, I actually got around to doing something about it.

Joshua Marinacci has started writing a new XHTML renderer in Java. He’s done a pretty impressive job at getting started but I get the impression that he has no idea what he’s in for. At the moment the rendering is pretty good but the speed is absolutely awful. While you shouldn’t ever optimize too early, it’s certainly possible to optimize too late and HTML rendering is one of those areas where you suffer death by a thousand cuts. I wouldn’t be at all surprised if he has to rework large parts of his architecture to reduce the number of string comparisons to speed things up. That’s the kind of thing that you probably should think about before hand (note that I haven’t even looked at the code so he may have already considered that). The other thing that I notice is that it doesn’t seem to support incremental layout yet so it has to wait for all the images to download before it can do anything. In an HTML editor that pretty easy to do because you’ve already got the code to update the layout quickly in response to changes but I’m not sure if there’s an easy and efficient way to do it without having to implement all that. The big complaint I have though is that there doesn’t seem to be any licensing information which is something you really should sort out at the very start of the project so people know what they’re getting themselves in for. Anyway, it will be interesting to see how the project goes. Update: The project page says the license is LGPL. Oh well, guess I won’t be having anything to do with it then.

Java.Net is having conversations about removing stuff from the standard Java libraries because they’re getting too big. The comment below really stuck out at me:

Combine that with a program to remove redundant functionality (by declaring it deprecated so it gets killed 2 releases later) and you have a powerful tool to get people to stop using old stuff like Vector and StringTokenizer.” Vector and StringTokenizer aren’t “old stuff” – they’re extremely useful classes with a specific purpose. Vector is a guaranteed thread safe implementation of List – something you can’t possibly get other than declaring that you want a Vector. To clarify that, consider the method signature public void doStuff(List items) what are the thread safe requirements on items? There’s no way to know and the default assumption would be that it doesn’t have to be thread-safe. Whereas with: public void doStuff(Vector items) the code may be less flexible now but the compiler now guarantees that you’ll get something that at least claims to be thread safe. It is still possible to extend Vector and perform non-thread safe operations but that would be a bug in the extending class because it failed to live up to the contract of the parent class. StringTokenizer is a particularly useful class in that it provides the simplest way to break a String down into tokens. You could iterator of the characters yourself or you could use regex to do this but manually iterating is reinventing the wheel and regex are highly inefficient for doing what StringTokenizer does. Furthermore, nothing in StringTokenizer is marked as deprecated in my copy of the Java 1.4 API, nor anything in Vector. Seriously, just because you don’t find something useful doesn’t mean it’s not useful.

Apple has a bug tracking tool called radar and they let any ADC member log bugs into it which they may or may not get around to fixing. They don’t however let anyone outside of Apple view the current bugs in the system. This causes great amounts of flamage on Apple’s java-dev list (and I imagine many of Apple’s other lists as well) about whether or not the list should be open. Every few weeks someone just has to come along and ask how they search the open issues and it starts again. Last time it came up some actual progress was made on the issue. Someone suggested creating a publicly accessible, unofficial bug listing that people could log bugs in, in addition to logging them with Apple, and those bugs would be publicly accessible so that people could find any work arounds that might be available. Of course there are three draw backs to this: 1. People may log the bug just with Apple and not in the 3rd party bug reporter. 2. People may log the bug in the 3rd party reporter and not with Apple. 3. People may see that the bug is already logged in the 3rd party bug reporter and not log it with Apple. Problem 1 is a non-issue. It just means that that particular bug report isn’t publicly available. Too bad, so sad. Problem 2 is a major issue because it means that Apple will never know about the bug and thus never fix it. (Nothing gets done at Apple without a radar issue being created according to the Engineers who’ve commented on the matter) Problem 3 is also a major issue because Apple prioritizes bugs based on how many duplicate reports they get. If people see the bug has already been reported they won’t report it again and Apple will think it’s not important and fix it whenever they get around to it. Obviously the combination of problems 2 and 3 is particularly disastrous. So the question then becomes, how do you create a system that lets people easily log bugs and search for existing bugs while still getting most people to log the bug with Apple? The ideal solution would be to have any bug logged in the public system automatically logged in Apple’s system as well. Existing bugs could have a button that allowed users to log a duplicate bug complete with a reference to the original issue with one click. The trouble with that approach obviously is that it requires access to Apple’s system which I don’t have. I’m not sure they’d appreciate having an automated system screen scraping their bug reporter to log bugs. It also doesn’t encourage people to specifically log their own case and thus can deprive the Apple engineers of that one subtle bit of information that suddenly makes the bug simple to reproduce. It’s probably possible to make the form look just like Apple’s so that it’s a simple matter of copy and paste to log the bug in both systems. A big red notice about how Apple wants to get duplicate bug reports might help alleviate problem 3 but it definitely wouldn’t solve it. All up, I’m just not sure if a publicly available, 3rd party bug database would help or hinder the situation. It would definitely be useful to have a place to go to search for existing work arounds though.

JavaScript is one of those odd languages that noone really appreciates the full power of. Mostly that’s because it’s also an awful language that’s hard to get the full power out of, particularly when working with multiple browsers. Still, my work often calls for large amounts of JavaScript hacking.

Some interesting things I’ve learnt lately:

In Safari, if you use window.open(‘file:///Users/aj/file.html’, …) it will either not open the file at all or refuse to execute any JavaScript in the file in the first time the page is loaded (reloading the page causes the javascript to execute). However, if you use window.open(”, …) and then window.location = “file:///Users/aj/file.html”; it works perfectly. Go figure.

Eclipse M9 was recently released and I’ve been using it all day today. It seems to work well though it reset a lot of my preferences which I didn’t think previous versions had done. Perhaps they were stored outside of the workspace somewhere… The most noticeable new feature is the code folding which generally I’m not a fan of until I realized it could automatically hide all the import statements at the top of the file. I’ve always hated having to scroll past a screenful of import statements to get to the meaningful code so it’s good not to have to. The other really noticeable change I saw was the ability to have calls to abstract methods shown in a different font/color as well as references to static variable, class variables. While this seemed like a really useful feature when I first saw it, turning it on made my source code overly colorful and really quite difficult to read. I’ve left it off for now though it’s probably possible to be more selective about which things are highlighted and in what colors. I’m not sure it’s really going to be a particularly big productivity gain though. I also noticed somewhat by accident that a little icon appears in the sidebar when a method overrides a super class’s method. That’s really useful to me. I spend vast amounts of time extending classes and overriding methods, mostly to add improvements or fix bugs in features that come with the JRE. It’s very nice to not only see that you’ve got the signature of the method right and that it really is overriding something but also have a very quick way of getting to the super implementation to make sure you’re taking care of all the things the method is supposed to. I’m just now reading the release notes and the mark occurrences feature looks very useful – particularly with it’s ability to highlight exit points of a function. We have a number of complex functions that have multiple exit points but that wind up being much harder to understand when broken up into smaller methods. For performance reasons though (and yes we’ve profiled and know that it’s required) the functions exit as soon as possible and thus have multiple exit points (refactoring to use a single exit point while maintaining performance just complicates the code). Having the exit points show up clearly should help a lot in these cases, particularly since it appears to also highlights points where an exception may be thrown and cause the method to exit. Back to the imports related features. The new ability to keep track of required imports when copying and pasting code looks really cool. Not that I’d ever copy and paste code….. Anyway, lots of cool new stuff to play with in Eclipse M9. The project is really coming a long way – I haven’t tested its speed on OS X yet but I’m hoping it will continue the trend of optimizations there and if my new RAM ever arrives maybe it will even be usable.

I really hate computer hardware. My PC started to randomly freeze every so often a month of two ago. It’s been totally out of action for a few weeks now and I’ve finally gotten around to trying to get it up and running again. After replacing the motherboard, RAM and CPU I can boot linux again and it seems stable if lacking a lot of features. It doesn’t seem to have support the ethernet card, sound card and has some issues with the hard drive controller. Windows on the other hand blue screens on startup – even in safe mode. I’m now trying to backup all the document from the windows hard drive to the linux hard drive and there’s this awful clicking noise. Somehow I think I’ll be replacing the hard drive too. Sigh.

I got involved in a long argument last night centering around different language paradigms and how the perl language fits into them. My “worth adversary” argued that perl could be considered a functional language at least in some ways because it supported function language constructs. In particular the example was given: map { freq $_ } @list The particular point being made here was that perl supported passing code into functions as parameters and that was a principle construct of functional languages. In Java you’d need to use an object to pass the function in as data thus it’s object-oriented and not functional. Now that I’m home and have access to my text books, I can safely put this argument to rest too. From Programming Language Essentials by Henri E. Bal and Dick Grune and published by Addison-Wesley:

I may have missed the mark on my attempt at an over the top, yet at least vaguely humorous reaction to Marc’s comments about explicit or implicit imports in Java. The golden rule of coding styles should apply: do what the rest of the team is doing. Having said that, I have to point out a couple of things in response to Marc’s update:

Code get’s read (at least) 100 times more then it gets written. (Forgot where I catched this quote) So the bit of time *you* win in typing is asking for a tough pay-back on wasted time on reading/interpretation. Please abondon the idea that you are doing someone a favor. Code gets read 100 times more than it gets written, but imports should never be handled by humans at all. Seriously, when I read Java code I never look at the import statements. Why would I? My IDE can read them, parse them and use that information far more efficiently and accurately than I ever could. It doesn’t care if explicit or implicit imports are used, it doesn’t care if it’s seen that particular package before or not. When I want to know what package a particular class is in I just hover the mouse cursor over it’s name and my IDE tells me the full package name. If I want to see the source of that class I control-alt-click the name of the class or usually the name of the method I’m interested in. The use of implicit or explicit import statements only affects readability of those people who don’t have a decent IDE and those people are sacrificing so much in productivity that the type of imports used are the least of their worries. For the record, both emacs and vi can be configured to use ctags and get an understanding of the contents of your project’s source code, so you don’t necessarily have to abandon those if you find their keyboard shortcuts make you more productive. Emacs users however will probably find that most IDEs support emacs key bindings anyway.

Marc Portier discusses the current recommended practices for import statements in Java. This is a never ending argument that is so completely irrelevant to development that only fools get involved in it. So let me jump into the argument. The rule is very simple, always use .* until you hit a conflict then add the appropriate specific import. Mostly with my code that means you wind up with:

import java.util.*;
import java.util.List;
import java.awt.*; Why is this rule in place? Because my time is more important than some fool who doesn't either use a half decent IDE or know the APIs well enough to guess where a class is. I don't think about import statements when I'm coding – I've got better things to think about. Every time I hit compile and get an error back (or get a red squiggly underline in the code) that amounts to “You're missing an import statement” it's a waste of my time. I hit control-shift-O the import is added and I get back to being productive. If I use specific imports that interruption happens every time I use a new class, if I use .* imports, that interruption happens every time I use a new package. The interruption is even worse for the people who don't use an IDE because they have to manually navigate to the top of the file, add the import and then find where they were again. I'm doing them a favor! If you don't like my rule, that's fine. Just hit control-shift-O or your IDEs equivalent and it will reformat the imports to however you prefer them. While you're there you may want to hit the reformat button and the position of braces will be what you want too. What? Your IDE doesn't do that? Tough.

It’s incredibly sad when people let their egos get in the way of developing something useful. I haven’t run into this in Apache yet but I’m seeing a really bad case of it in the on-going quest to create an opensource replacement for HyperCard, Apple’s now abandoned rapid application development system. There are now four or five different projects each struggling to get something up and running, none succeeding because the developer resources are stretched so thin. Even my latest efforts to get them to work together by saying I’m happy with any language, any architecture – just get a 1.0 out the door has been shot down because people are so insistent on writing it all themselves and doing it precisely their way that they’ll cut off their own nose to spite their faces. I’m very much on the verge of giving up on the whole idea to be honest. I’ve got enough skills in Java that something like HyperCard isn’t exceptionally useful to me anyway – I’m not sure I should waste any more time trying to create something like that anyway. It’s sad though, HyperCard is the best way I’ve ever come across to get people interested in programming and start them learning and thinking about what their computer can really be made to do. No other language provides that kind of opportunity for creativity and power to Joe User.

Recently there has been a very interesting and at times heated discussion about optimization on the HUMBUG mailing list. It’s worth reading the archives (look for the “Which is better?” thread) to get the whole picture.

There were two main topics being discussed:

1. Whether you should look for optimizations as you code and implement them if they don’t affect readability and
2. Whether or not memory allocation in Java is slow and thus efforts should be taken to avoid it.

Issue 1 – Premature Optimization

Inside The Loop or Outside The Loop?

Most people argued that premature optimization is the root of all evil, and that in Java particularly it’s a waste of time because you almost always predict what the compiler and the JITC are going to do incorrectly. There were a couple of people (one in particular) who thought that optimizations that didn’t affect readability were a good idea. The example:

It seems that the proprietary hooks in .Net are becoming more and more clear. This Netcraft interview with Miguel de Icaza ends with the comment:

Longhorn has kind of a scary technology called Avalon, which when compounded with another technology called XAML, it’s fairly dangerous. And the reason is that they’ve made it so it’s basically an HTML replacement. The advantage is it’s probably as easy as writing HTML, so that means that anybody can produce this content with a text editor. This is the great proprietary catch that ties .Net solely to the Windows platform. No language is useful without a solid set of libraries to work with and the best libraries for .Net (XAML and Avalon) will be Windows only and managed solely by Microsoft, not a standards organization. It’s not like people haven’t warned about this before though…. Some people still think they can get a good deal out of the devil.

Continuing my spam theme of late. Today I received a message from “* Rochelle *” that wound up in my inbox. Since I have so many carefully crafted filters to dump mail in the appropriate mailbox anything that winds up in my inbox is immediately suspected as spam that spamassassin missed. This message hit massive alarm bells in my head and I just couldn’t work out why spamassassin had missed it. The subject line was “hey!!!” and the body of the message was bright pink with embedded images that Mail.app had blocked. Worst of all, it was from a hotmail address similar to “I’m your lollypop”. Naturally that first quick glance was enough for me to start dragging it to the spam folder. While doing so I fortunately noticed the phrase “Tom’s birthday party” and it just seemed too much of a coincidence considering I’d attended my cousin Tom’s birthday party a couple of weeks ago. Turns out the mail was legit and from someone I’d been waiting to hear from for a fair while. The moral of the story – trust in spamassassin. Or maybe, look before you hit delete as spam. Or maybe that you shouldn’t use hotmail addresses that include talk of lollipops along with HTML email and bright pink backgrounds. Yeah, definitely don’t do the pink background thing…..

Richard Giles comments on SpamSaver, one of the worst ideas I’ve heard of in a long time. Essentially SpamSaver aims to feed a massive number of useless email addresses into the spammer’s database making it all but useless and swamping them with bounced mail. Sadly, the flaw lies in the last part – the spammers don’t see the bounced mail – the poor sap who’s email address they forged does. PLEASE don’t contribute to more of this “bounce-spam”, it’s already reached the point where I receive more bounce messages than I do actual spam. Mostly that’s from virus bounce messages, but an increasing amount is from spam bounce messages too and things like SpamSaver will only make that worse. So please don’t use SpamSaver and if you’re a sysadmin configure your servers to never send automatic emails for any reason. I don’t care if the email address doesn’t exist, I don’t care if the message had a virus and I really don’t care if the person is on vacation. Unless you can be 100% sure that the From: address is real (which you can’t) don’t send mail to it.

Ian Holsman comments on a vacation responder that deletes all incoming email to avoid their mailbox overflowing while they’re away and asks if email is dead. The answer is a clear no. Email is now a critical system, it just can’t be killed, perhaps morphed into a different set of protocols but the concept of sending mail electronically just can’t die, unless you count being replaced with sending video messages electronically as killing email. However, email is no longer a time saver, it’s a great time waster. It’s a chore that you have to put up with to stay in business or to stay in touch with your widely dispersed friends. With the recent demise of our exchange mail server I’ve had to change the way I filter out spam which requires training up a new spam filter and in the process receiving massive amounts of spam. I honestly never realised just how much spam I received until I suddenly didn’t have a filter in place to get rid of it. Without that filter, email is completely unusable and with a poorly trained filter it’s a great waste of time. We’ve now outsourced our exchange server and I can go back to happily using SpamBayes with Outlook. I’ve run it over the 14000 stored spam messages that I managed to recover from our dead exchange box (massive amounts of real email got nuked, but the spam survived….) and trained up the new filter, but it’s still not as good. I’m not sure why. I’ve also found that when you receive large amounts of spam the best thing you can do for your productivity is turn off the new email notification. Noone notices if it takes you 30 mins to reply to an email instead of 5 mins and by turning off that notification you’re no longer interrupted by spam messages (or real messages) every 1-5 minutes. You check you’re email when you want not when it wants. Also, by leaving your email open you still have a reminder on screen to pay attention to email every so often and it’s quick and easy to bring the window back up and check if you have new email. The other thing you need to do is have a really huge amount of space available in your mailbox, whether that’s achieved by actually having a large mailbox or just having a fetchmail (or similar) process running to constantly download your mail to your own computer for storage, it’s got to be there. That avoids the problem of spam filling up your in box. Finally, train your spam filter to treat any bounce message as spam and discard it. At least 50% of the unwanted email I get is actually bounce messages either for incorrect addresses or virus detection notifications. If you’re a system administrator, turn off all bounce messages, they no longer serve any useful purpose and are just clogging up people’s inboxes.

Well, after two very long days and a standard day of fighting our exchange server, I’ve given up. I did manage to get our data imported again and get the server up and running, unfortunately the database is very corrupted and we’ve lost at least half of the email. We went back to the tapes to see if we could pull off the last backup before the server died (the exchange drives were untouched in the crash but the databases were corrupted because exchange didn’t shut down correctly) and found that all the backup tools are reporting the tapes as completely empty. Now I know that the tapes aren’t empty, we retrieved stuff off them no more than a week ago, but try convincing the software of that…. So I’ve given up. We’ll probably ship off the tapes and the drives to some exchange recovery company to see what they can retrieve and we’ve decided we won’t be setting up our own exchange server again. Now we’re outsourcing it. There are (to my surprise) a large number of companies that will set up and run an exchange server for a surprisingly low monthly fee. Access is available via Exchange and IMAP so we can retrieve our data easily and most of them even include licenses for Outlook 2003 for free with your subscription. Pretty sweet deal if you ask me. I never want to play sysadmin again and with this setup there’s a good chance I won’t have to.

Lately there’s been a number of blogs from people looking for Java WYSIWYG editors that work in a browser. The latest one from Bertrand Delacretaz is enough to prompt me to speak up. Firstly, I work for a company that develops a browser based WYSIWYG editor so I’m biased, but I’m also an expert in the field. We’re the top of the line in browser based editors and there’s really only a couple of other companies that can compare with our product. There are thousands of other products on the market that provide varying ranges of functionality, most of them are tied into a particular browser, particular platform or both. Mostly, they don’t provide a full editing experience to users or are difficult to integrate. Here’s a few things you should look at when you are shopping around for a WYSIWYG editor:

I need a drink. Oh Yeah! I’ve spent the 40 hours trying to restore our exchange server to operation after a power outage on the weekend took out the motherboard of the server. It’s 9:45pm, I’m still at work and I haven’t even started the hard part yet. I was here until midnight last night only to discover that the server needed to be completely replaced. Fortunately, it looks like we won’t loose any data in the whole episode – assuming I can eventually get exchange to do what it’s told. We are so changing our server setup.

I think I’ve created a monstrosity. I currently have (and am happily switching between) 2 screens, 2 keyboards, 2 mice and a trackpad – all connected to the same laptop. Why you ask? Because each input device is in an ideal position for a particular task. No I didn’t plan it this way but it is working out pretty well. Here’s what my desktop currently looks like (excuse the dodgy iSight photo quality): It started off pretty simply – a laptop with an external (bluetooth) mouse. When I’m switching quickly between the keyboard and the mouse I tend to use the trackpad, when I want to use Exposé or am doing something more intensive with the mouse, I use the external mouse. Then I brought home a monitor from work so I could set up a new development box ready for Monday – the box is now all set up so I put the monitor to use as a second display for my laptop. Except that the keyboard and mouse were still sitting in front of it from when I was using it with the PC and I kept turning to the second monitor and trying to use the keyboard and mouse that was in front of it. So now I’ve got the keyboard and mouse plugged into my laptop so that I can turn my whole body and work on the second screen without getting confused between input devices. I’ve never understood why everyone’s so keen to use a single keyboard and mouse with multiple computers on their desk – it’s just so much more logical to me to use the keyboard and mouse that’s in front of the screen I’m looking at.

Which OS are You?

Brian McCallister got me started on this. Apparently he’s slackware. I’m Palm and everything it says is so true it’s scary.

Update: I have to point out this:

if you ended up with GNU/HURD it means you didn’t answer all the questions.

And the RAM is taketh away again… My shiny new RAM arrived today. Unfortunately it appears to be the emporer’s RAM as it looks real, feels real, I assume it tastes real, but when you put it in my laptop Apple System Profiler reports the slot as completely empty. Needless to say, it’s being sent back. Unfortunately, since it came from Melbourne the replacement process will probably take most of next week. Sigh. I swear my computer was faster with the dead RAM stick in too.

(I’m starting a $NAME has lost it series if you were wondering. Blame Leo.) Anyway, Stallman’s lost it.

But the major source of this problem today is Java, because people who write free software often feel Java is sexy Oh Java, you sexy thing you. Admittedly the sexiness of Java is starting to make life a little difficult. I can’t get any work done because I just sit there looking at it. When I walk down the street hand in hand with Java, people wolf whistle. I go to a bar and every guy in the place tries to steal Java away from me. Maybe I should get in touch with “Queer Eye for the Straight Guy” and get them to give me the full makeover so I can be as sexy as Java is.

A young friend of mine who’s just starting out in the world of programming and has ambitions of becoming a software engineer asked me for work experience today. I couldn’t think of anyway that could give him the support he needs (note: if you know of a company in Brisbane or surrounding areas that could provide some work experience for a year 11 student that’s starting out in programming please let me know – I think a September timeframe fits in with his school well). Instead I suggested he sink his teeth into any and every programming problem he could find. Naturally he asked me to suggest some and I figured while I’m at it I may as well publish them. So over the next few weeks or months I’ll be writing up some sample programming exercises along with some of my thoughts on how to go about solving them and posting them to the web. The first one’s available now. I’m sure there’s plenty of examples and tutorials around on the web but I wanted to take the contrived examples and make them a little more real world. As such, all the exercises will start with a semi-formal set of system requirements (similar to what most contractors are given) and a discussion of design techniques and other software engineering considerations is included. The examples are language agnostic and are intended to focus on giving problem solving and programming experience rather than learning a specific language. I’d love to hear of any other good resources people can point out as well as any comments on the examples – things to add, things to improve, things to cut out etc.

Following my complaints yesterday I think I’ve finally managed to get XSLT working correctly in perl. The secret: ignore the obvious choice (XML::XSLT) and use XML::LibXML and XML::LibXSLT in combination. Then use fink heavily as well. Try to avoid using CPAN or installing perl modules by hand until you don’t have a choice. Step 1: Install expat, libxml, libxslt and sablotron using fink. Install anything else that looks vaguely XML related while you’re there too. Step 2: Using fink, install every XML or XSLT related perl package you can find. Step 3: Using CPAN run: install XML::LibXML. Step 4: Using CPAN run: install XML::LibXSLT Then use code similar to: `my $parser = XML::LibXML->new(); my $xslt = XML::LibXSLT->new(); my $stylesheet = $xslt->parse_stylesheet_file($xsl);

I’ve been wanting to develop a “dashboard” style system to information from a bunch of different systems at work. Since we use an awful lot of XML around the place at work the easiest way to do that is most likely to use XSLT to transform the various XML feeds into HTML to be displayed and piece it all together into one page. Systems that don’t speak XML could have a gateway developed for them easily enough. Anyway, I figured I’d write a few simple perl scripts to piece the various XSTLs together and make it all work, hopefully develop something fairly pluggable so that it’s easy to add new feeds as required. So I went looking for XSLT processors for perl and wasn’t particularly impressed with anything I found. The XSLT I had been using and that worked with both XAlan and MSXML but barfed with XML::XSLT. The other XSLT libraries I found either failed to compile on OS X or failed to run once installed. Now admittedly that’s mostly going to be my lack of knowledge of C and perl but I was surprised there wasn’t a simple, standard XSLT processor. So then I tried PHP and while it has XSLT commands that looked promising, they weren’t compiled in to my copy of PHP. It appears to be off by default in PHP because it depends on a bunch of native libraries that may not be present on most systems. So in the end, I think I’ll write the system in Java simply because it’s actually good at handling XSLT. Why isn’t XSLT straight forward in other languages? It seems that they’re trying to leverage off existing C libraries to save having to write a complete XML parser and XSLT processor natively in the language. Unfortunately, that seems to introduce a number of headaches when actually trying to use the libraries. I guess Java makes communicating with C libraries more difficult than with languages like perl or PHP and that in combination with it’s dedication to cross-platform support seems to have given it an big advantage in situations like this by not encouraging the reuse of existing code in another language. It certainly makes it a heck of a lot easier for users of the language. So what am I missing? Is there a simple way to get a fully compliant XSLT processor in an effective scripting language? Is there another scripting language that’s good for piecing together simple systems that does have straight-forward XSLT support? Python? Ruby? Help!

I hate how slow Entourage is when accessing an imap server. Most of it’s problems stem from the fact that it only downloads things from the server when you actually ask for them. While that’s great for saving bandwidth, it means a lot of latency between clicking on a message and actually getting to see it – particularly since Entourage and imapd don’t seem to get on much and occassionally just hang the connection.

In response to this, I have a few things to say. Firstly I thought the original article was very much on target – opensource software *does* suffer from poor user interfaces and it is typically because it’s so difficult to get a cohesive UI out of a widely dispersed group of people, from different cultures and in different time zones. The particular issues I had with that particular response:

The newly designed save and open dialogs are a great example of the simplicity the GNOME project seeks to achieve. True, the dialogs may irk those who like tab completion and other esoteric features (I’m guessing their issues will be worked out in future releases) the boxes are notably simpler and clearer than their Windows counterparts. And they look just like the OS X file dialogs. Not to mention the fact that the extra mouse click introduced by the “browse for other folders” toggle makes the user interface slower and less intuitive. It was a mistake when Apple put it in and it’s a mistake when Gnome copied it – hopefully the Gnome mob were smart enough to make the dialog remember whether the user wants it expanded or not (as Apple eventually managed to do).

Jono Bacon comments on how XUL could be great for providing more interactive web applications. My take on it: he’s gone off the deep end. I just don’t get what the current obsession is for having to run everything in a web browser these days.

This framework not only brings the web browser out of the dark ages in terms of the potential for interaction, but the Mozilla developers made the right choice and picked XML as the language for specifying user interfaces. See here and here for why XML not a good programming language.

Kathy Sierra writes about being a loner and how it makes pair programming very frustrating. I can see her point – some people enjoy being alone and don’t like feeling crowded by others. That is different to not being a team player – loners can be very good in a team, but they do run into trouble when the team happens to be using pair programming or is just a very “in each others back-pocket” type of team.

Okay, I have to ask – why the sudden massive interest in JXTA? It seems to have taken over all the weblogs on java.net and most other places I look around. Maybe if I actually had a clue what it was really about I might be excited about it too, but I just haven’t had time to look into it. Sigh.

I can’t resist picking apart this comment to my earlier post.

It’s easy to look at the code and modify it, but it’s not as easy to actually test that code. You need to mess with bootclasspaths and things like that. How is this different to an opensource version of Java? You would still have to set up all the correct bootclasspaths and make sure you had things in the right place for it to work. That’s the design of Java, it has nothing to do with whether or not it’s opensource. Additionally, it’s not that difficult, just replace rt.jar (on Sun based JREs, classes.jar on Mac) with your new modified version and viola, you have a modified JRE.

I noticed this on Java.net. In it, Eitan Suez suggests that having open source J2SE libraries would rapidly increase their quality because of all the developers contributing patches. The major problem with this argument is that the source for the Java standard libraries is already available and in fact is included in pretty much every J2SDK. It’s right there, in a format that can be used to create patches against and submit them to the publicly available bug parade for java. How many people do you see actually doing this though? Some, but very few. Sure people are going to be more inclined to submit patches if Java becomes buzz word compliant and gets a mickey mouse badge from the FSF, but there will still be far more people demanding improvements than there will be fixing them. Open source isn’t this magical bullet that suddenly makes life easy and gives perfect quality – there are plenty of extremely buggy opensource programs with too few developers and there always will be. Maybe I’m just more careful in choosing which commercial software I use, but on average I’ve found opensource software to be lower quality than commercial software, however opensource is more towards the extremes – it’s either really great or really bad. I’m sure that’s going to start a massive flame war but so be it. Opensource isn’t always better quality because generalisations are always false. Finally I’d like to draw attention to what I consider the best rebuttal I’ve heard in a long time:

Jim Winstead has posted a couple of entries on character encodings (1, 2). Some good info in there. My three big tips for dealing with character encodings is this: 1. Know your character encoding and make sure you’re using that encoding everywhere. No look again because you probably missed a place where you didn’t think about encoding. Particularly don’t forget that printing something to System.err or System.out in Java uses the platform default encoding and so characters that can’t be represented in that encoding become question marks. 2. When practical, use US-ASCII and escaped characters for anything outside of it’s range. Most formats which support different encodings also provide a way to represent a character which isn’t in the current encoding through something like HTML’s entities or java’s escape codes (\u8222 etc). Most encodings are compatible with US-ASCII (EBDIC being a notable exception) so even if people forget to use the right encoding they can generally get away with it. 3. Remember that character encodings, despite their name do not apply to characters – they apply to byte sequences which represent characters. If you have a char variable in Java it has no character encoding as far as you are concerned, it’s just that character. The JVM can choose any representation it likes for that character in physical memory and you shouldn’t care (it actually happens to choose UTF-16 I think but you still shouldn’t care). You *do* however have to worry about character encodings when you convert from characters (or Strings which are really just a fancy array of chars) to byte streams. This happens when you use String.getBytes(), or print the characters to any kind of output stream. You also have to worry about the reverse process, new String(byte[]) and reading from an input stream. The first two items should be pretty clear to you if you’ve done any work with character encodings, the third may seem unimportant, but it will help stop you from expecting code like the following to work: String str = new String("my string"); byte[] utf8Bytes = str.getBytes("UTF-8"); String strISO88591 = new String(utf8Bytes, "ISO-8859-1"); Naturally this won’t work because of rules 1 and 3. Rule 1 is broken in that you used a different encoding when working with the same data and 3 is broken because you expected strISO88591 to be a String using ISO-8859-1 character encoding, but it doesn’t because String objects don’t have a character encoding (as far as you should be concerned). The big exception to rule 3 is when you’re using a language which doesn’t guarantee it will support whatever characters you throw at it, in which case you basically either have to work only with byte arrays and never let the language string functions near them. In general though I’d suggest you find a better language or a better library. If I were to add fourth suggestion, it would be: remember that just because your character is valid, doesn’t mean the font you’re using can display it. Most fonts can’t display anything more than the characters in ISO-8859-1 and a few select others so if you’re working with mathematical symbols or characters from other languages you’ll need to find a special font that supports them. BTW, yes I have spent far too much time working with character encodings and tracking down where people stuffed up with character encodings.

Glen Stampoultzis complains (rightfully) about the Windows XP look and feel. It fits in a lot better than his example makes out for our usage – not sure if that’s just because we never use JFileChooser (use java.awt.FileChooser instead, JFileChooser is awful in every L&F) or if it’s some particular setting somewhere. Anyway, if you’d like to pick up a whole bunch of fixes for the Windows L&F (both XP and non-XP), take a look at WinLaf. It provides some really nice fixes for the L&F on Windows and is very easy to integrate. Be warned though, it occurred a very heavy performance penalty in our application, though I still can’t say why exactly. We wound up removing it again because we just didn’t have time to fix it before release. If a few more people get behind it and particularly if someone hits it hard with a profiler it could be a very useful project. OS X users should check out QuaQua which is a similar kind of thing for OS X. I haven’t used QuaQua myself though.

I’ve reclaimed top spot on the google ranking! Ha! Take that Adrian Sutton! (If you have no idea what I’m talking about see the front page and this entry.)

I have come to the conclusion that URL escaping is evil and must be banished from the face of the earth. I’ve got no idea how it manages to work at all – every implementation seems to be different and the support for different character sets is a major hit and miss affair. Take for instance the string: © Adrian Sutton It looks like a pretty simple string and all. It should be encoded as: %C2%A9%20Adrian%20Sutton assuming UTF-8 character encoding (and I literally mean assuming since there’s no possible way to know for sure). If however you were to use the javascript escape() function you could get any one of: %u00A9+Adrian+Sutton %C2%A9%20Adrian+Sutton %u00A9%29Adrian%20Sutton It’s impossible to tell if the + sign in the first two is an encoded space or an actual plus sign (there’s no requirement for + to be escaped in URIs so many implementations leave it as is). Then you have to deal with the rather odd %u00A9 syntax which seems to be half URI escaping, half HTML entity and finally you get to worry about which character set was in use. For the record, here’s what your browser makes of it:

Nick Chalko talks about setting onsubmit dynamically. The solution he received from Alan Gutierrez which is good, but overly complicated. Since I work for a company that just so happens to do some amazingly funky stuff with JavaScript, here’s some fun you can have with it. Firstly, lets take the original solution:

<form name=&#39;CDiceComponent_0&#39; id=&#39;findApplication&#39;
    action=&#39;/cdiceWebApp/custom/component/bridge.jsp&#39; method=&#39;post&#39;>
<script>
  document.findApplication.onsubmit = function() {return checkForm();}
</script>

and simplify it to:

<form name=&#39;CDiceComponent_0&#39; id=&#39;findApplication&#39;
        action=&#39;/cdiceWebApp/custom/component/bridge.jsp&#39; method=&#39;post&#39;>
<script>
 &#160;document.findApplication.onsubmit = checkForm;
</script>

JavaScript functions are just string variables so you can pass them around by just dropping the () at the end. Here’s how you could store the existing onsubmit function:

I discovered something really quite annoying today. The JDBC drivers for MySQL have been re-licensed from LGPL to GPL (happened quite a while back actually). Now while it’s their code and they can do with it as they please, that’s really, really annoying. I’ll be moving all my development away from MySQL in the future. Here’s the problem. I came up with a really cool new feature for a product I’m working on which involves interacting with a variety of databases and inevitably someone will want to use it with MySQL. This feature would be a small add-on to a much larger commercial product. In order to use the official MySQL drivers though I’d either have to pay MySQL or GPL my application. Neither of which is a realistic option considering how insignificant this one feature is in the overall product. So now I’m developing the prototype with Microsoft SQL Server and it’s going great. MySQL, there appears to be a bullet shaped hole in your foot.

I’ve been playing with Linux on my powerbook, mostly just for something to tinker with. Generally it runs pretty well, but I don’t have a lot of space on my internal drive and wanted to use my firewire hard drive for /usr. This is a major mission apparently. After many hours of compiling kernels with massive struggles in finding a set of options that will actually compile, I managed to build a 2.6 kernel. Sadly it freezes the machine during boot. It seems to not like running discover to automatically probe devices. I’ve removed discover and will be installing my modules by hand from now on – most things are compiled into the kernel anyway. So I’m finally up and running on 2.6. HFS plus support went AWOL so I’ve now applied those patches and am recompiling again – hopefully without drama this time. On the really cool side, the sleep light on my powerbook flashes when the hard drive is accessed.

There’s been a few comments to my rant on choosing the right language for the task that I wanted to comment on.

Firstly the simple one from hammet:

Dude, you can say you hate C#, not .Net. .Net is not a language.

No, I definitely hate .Net. I still can’t tell you what went wrong with it or where to point the blame but my short spell of .Net programming rates as my most hated moments in computing. I have no particular problem with C#, it’s a language that’s very close to Java has some nice ideas, leaves out some nice ideas and makes some good and some bad design decisions – so be it. My beef was definitely with .Net because it didn’t matter if I used ASP.Net, VB.Net or C# nothing went right. A good deal of the blame definitely lies in my not knowing the language or the environment but I definitely shouldn’t have been able to waste a full two weeks getting obscure error messages that even intensive Googling and asking experts couldn’t sort out. I suspect something went bad in my Visual Studio.Net install which then broke something in my .Net framework install or in Windows itself. .Net has never worked for me since, nor has my IIS installation been all that rock solid. Oh well, maybe next time I have a need to use .Net it will go better.

It appears that Apple have killed the dogcow. A highly endangered animal that was taken under Apple’s wing and blossomed in the days before OS X, the last known wild Dogcow now appears to have died. The soul remaining vestage of Doggycowiness (sic) is TechNote 31 which now can only be found in captivity at Google (or is that cacheivity?). Sadly because Google doesn’t cache images, the actual Dogcow that was in captivity in TechNote 31 at Apple now appears to be dead. The body of that last Dogcow cannot be found below: Let us all solemnly recite the great Dogcow muse:

I’d seen a link to this a few days ago and it was presented in such a way that it seemed like an intelligent discussion of the relative merits of C++ and Java – certainly the reply I read was. I didn’t have time to read it until now and I’m extremely disappointed. If I could remember who linked to that rant I’d probably give up reading their blog. I’m not going to bother rebutting most of the article, there are plenty of incorrect statements in it just as there are plenty of incorrect statements in every article like it comparing any two languages. How hard is it to make people understand that they should choose the right language for the right job? XSLT is a good language for manipulating XML, Java is a good language for cross platform work, high level apps, applets and a bunch of other stuff, C is good for writing low level stuff, speed critical apps (or speed critical sections of apps), as is assembly, C++ is good for a bunch of stuff, perl is great for a bunch of stuff particularly quick little scripts to manipulate text files, shell scripts are great if you need to perform a bunch of file operations or combine a bunch of small pre-existing tools. Prolog is good if you need a solid logic processing/reasoning engine, HyperCard is fantastic if you want an easy to comprehend language with extremely rapid development, MetaCard is great if you need a cross-platform HyperCard or just hit up against HyperCard’s limits. Javascript is a great language if you need to add dynamism to web pages, Visual Basic is great if you want a rapid prototyping language or a simple GUI tool, .Net is probably even useful if you can make the #*(#@ thing work… I’ve programmed in all of those languages and probably many more, they all have strong points and weak points but if you want to succeed at using any of them, you have to stop trying to use them like they were your “favourite” language. You will always think Java is crap if you try to write C or C++ code in it – it has a different paradigm and a different way of doing things. By the same token you will always hate .Net if you try to write Java code in it (trust me I hate .Net because of that). The idea that if you can program in one language you can program in any language by just learning the syntax is completely wrong. You have to learn the syntax, then you have to learn the paradigm, then you have to learn the best practices, then you have to gain experience and finally you have to accept the philosophy of the language and stop trying to beat it into submission. In short, learning a new language is just that – a learning process. You can’t expect to know everything and you must go into it assuming nothing and learn from scratch – everything you have come to expect and assume about your programming language may have changed and you need to find out if it has or not. If you find yourself criticising the programming language constantly you’re doing it wrong. Thousands of people are most likely happily using that exact same programming language without cursing and swearing – do you really think you know better than all of them?

java.net blogger Malcom Davis comments on why you should use the Sun Java coding style. The first comment is the perfect example of “ego” that he talks about:

Why not adopt the Sun coding conventions? Simple – because they are derived from an antiquated and BAD way of coding.
There are very good reasons why C programmers moved away from the coding styles used by Sun, and the excuse that ‘modern IDEs have code colouring and brace matcing, etc.’ is NOT a good reason to have to rely on it. Gah! Could there be a more reactionary, inflammatory and unsubstantiated comment? How does the poster know what the Java coding convention was derived from. What makes it “bad”? Why is an antiquated coding style not a good thing? Why exactly did C programmers move away from the coding styles used by Sun? What evidence is there that C programmers actually did start of such a coding style and move away from it? What problem does code colouring and brace matching supposedly not overcome? In other words, why should I believe some random comment on a web page which provides no solid argument, no reasoning, no attempt at justification and absolutely no facts? As Malcom noted:

Sylvain Wallez comments on his new PowerBook hardware and among the good and bad he’s found are a couple of comments that come up a lot but shouldn’t. There’s three problems Sylvain is running into that I want to take a look at:

1. No page up/down
2. No scroll wheel
3. One mouse button The solution for all three problems is to keep your left hand on the keyboard. When you’re typing, keep both hands on the home row (asdfghjkl;), when you use the mouse, slide *both* hands down, your right hand slides down to the track pad and your left hand slides down to sit on it’s “mousing” home row – (fn, ctrl, option, command). You now instantly have access to your second mouse button by pressing down the control key – it’s actually much faster and easier than a real second mouse button because you don’t have to turn your thumb under to reach the second mouse button. Now, the next thing to notice is that on a laptop, moving between the keyboard and the trackpad is a very cheap operation (unlike when using an external mouse), the two things are very close together. This is most noticeable when moving from the trackpad to keys at the bottom of the keyboard – like the arrow keys. So when you need to scroll, flick your right hand up to the arrow keys and suddenly you have access to scrolling and page up/down (fn on the left hand and the up/down arrow). This falls down when your editing documents rather than reading web pages etc because the up down arrow just moves the cursor and not the view (though some programs use command-up to move the view and not the caret) – page up/down still work but are no good for careful positioning. This is *far* better than using the side of the trackpad as a scroll wheel though (at least as far as I’ve found). All this probably takes a bit of getting used to, but once your fingers get used to it you’ll probably find that you don’t miss the scroll wheel or second mouse button. Even when I do have an external mouse plugged into my laptop I use the trackpad because it’s so much closer, faster and easier. If however, I have my laptop out of reach or closed and am using an external keyboard and mouse I’m driven absolutely nuts by the lack of scroll wheel and second mouse buttons. But wait! There’s more! Now that you’ve got your left hand on the bottom left of your keyboard you’re ready to fully appreciate the power of keyboard shortcuts. Ever wonder why undo, cut, copy and paste are Z,X,C,V – they’re right there in the bottom left of the keyboard for easy access. Save is a flick of the fingers away, you can bring up the find dialog while your right hand flicks back up to the keyboard and starting typing immediately. If you’re a command tab junky, that’s close by for you as well, though once you realise just how close the track pad is (and get used to flicking over to it) you probably won’t be so addicted to command tab anyway (but don’t loose the skill, it’s still essential on desktops). Lock your dock down so it doesn’t shuffle around all the time and your mouse becomes the ultimate switching utility (short of rapidly switching between two applications which command-tab is king of). Then start mapping special functions to things command click, shift click, command-shift-click, command-control-click, command-option-click, option-control-click, function-control-click, command-control-function-option-click, oi! How many buttons do you need? :) Oh yes, learn to use the escape key – it will cancel pretty much any dialog and a whole bunch of other things too. Plus with your left hand always on the keyboard, it’s really fast and easy to reach. Now if you haven’t given up on me being a stark raving lunatic yet, here’s the tip that every mac user should know about but much to my surprise don’t seem to. Command-H. Try it! The current application hides instantly letting you access whatever is behind it. Unlike minimizing though, this hides all the applications windows and most importantly they all come back at once if you click the application in the dock or command-tab to the app. You can also use command-option-H to hide all other applications, though I’m not sure why you’d bother. Let me repeat myself in bigger type for those who only skim this far down:

Command-H is your friend. Minimizing is for wimps. Okay, that was a bit antagonistic. I’ll stop now.

Continuing on the “things I didn’t know” theme: I didn’t know there was a kext for OS X that let you mount ext2 and ext3 drives. Very nice to discover, now not only can Linux finally read and write HFS+ drives but OS X can read and write linux drives. Kudos to the people involved.

Well I learnt something new today. All this time that I’ve been doing some extremely funky stuff with XSLT (and I do mean funky), I never actually had the idea of the context node quite right. I discovered this because of a bug I ran into with our updating of XPath expressions that use position(). Essentially in the XML editor I’m developing, we pick out the XPath expressions in the XSLT used to lay out the data for editing so that as their input values change we can update the result like it was a spreadsheet. When the input values change we identify which XPath expressions need to be recalculated and rerun them (using Jaxen) by passing in the context node from the document we’ve been editing. The trouble hit when we started using position() because it always evaluated to 0. As it turns out the position() function counts the position of the current node in the current context, not necessarily it’s position in it’s parent node. So where I’d been passing in a single node as the context, I should have been passing in a set of all the child nodes from the parent and pointing to the appropriate node in the list as the “current” one we’re processing. As it is, it’s still not actually implemented correctly because we always use the parent nodes children as the context but the actual context may have been different. For instance given the template: <template match="item"> <value-of select="position()" /> </template> The context is actually the list of child elements of the current context node whose name is “item”. It wouldn’t include any text nodes and it wouldn’t include any elements that weren’t “items”. So for the XML snippet: <parent> <item /> <other /> <item /> <other /> </parent> The context would be the two item elements, but not the text nodes and not the other elements. Fortunately, for our purposes the approximation we’re using (all child elements, excluding text nodes) is good enough for what any of our users will need to do and significantly more accurate than anything the visual design accompanying our product will actually produce. I’ll be interested to see if the testing guys pick up on it though.

Mostly just so that I can find the recordings more easily later, here’s the recordings of DevCon VI talks I gave in Bogotá.

With the merge fast approaching, all beacon chain nodes will need to run their own local execution layer client (e.g. Besu or Geth). For validators who have just been using Infura for eth1 data so far, that means some extra setup and potentially more hardware (mostly disk). To avoid this, some people are considering going the other direction and not running their own beacon node either, just running a validator client against a public beacon node API like Infura. While this is possible, it’s extremely likely to result in reduced rewards for a couple of reasons.

Yesterday I spun up two new nodes on Prater, and sent deposits for 5000 validators to run on each. Due to the restrictions on how many new validators can activate per epoch (currently 4 for Prater), the number of active validators are currently gradually increasing giving a really nice insight into the cost of adding new validators.

This is most clearly seen by graphing the number of active validators and CPU usage on a single chart.  Here the green is CPU usage and the yellow the number of active validators.

The merge is at the point where real interop testing can begin with testnets spun up involving multiple clients. This video shows a testnet running locally with four nodes:

• Teku & Geth
• Teku & Besu
• Lighthouse & Geth
• Lighthouse & Besu

Each node has 1/4 of the validators and the network transitioned through from the initial phase0, to Altair and then then merge after which the two chains merge into one.  Also shows sending transactions with MetaMask post-merge and how the user experience is largely unchained. Ethereum keeps on being Ethereum but now with PoS instead of PoW.

Over on Twitter, there was an interesting discussion about the importance of client diversity for the beacon chain. As part of that cyber_hokie asks:

cyber_hokie – @cyber_hokie: As the majority client, isn’t it more likely that issues with Prysm would cause more severe financial penalties for minority clients voting on an alternative chain under accountable safety given Prysm validators likely hold the 2/3 threshold?

It’s important to note up front that this isn’t a Prysm specific issue. This isn’t a criticism of Prysm at all. All the major beacon chain clients are great – the issue here is any one client getting too large a share of validators. How much is too large? In an ideal world every client would have less than 1/3rd of validators. Then a bug in a single client doesn’t prevent the chain from continuing to finalise. More than 2/3rds majority is a really big problem because of the potential for those clients to finalise on an incorrect fork.

One of the most commonly overlooked steps when setting up a beacon node is to setup port forwarding on your router to ensure that other nodes can connect in to yours. This is often overlooked, or not done correctly, because there are so many different routers, all with different, fiddly interfaces that providing simple instructions is essentially impossible and because your node will generally work even without it by connecting outbound to other peers even though none can connect inbound.

I got the chance to talk with Pooja Ranjan and Alita Moore from the Ethereum Cat Herders as part of their PEEPanEIP series last week: PEEPanEIP #38: Altair in Teku with Adrian Sutton. It was a chance to talk through the actual implementation details required to support the Altair fork.

Mostly listing here for my own record, but if you haven’t seen it and you care at all about Ethereum hopefully it will be interesting.

Note: This post is deliberately high level. It doesn’t attempt to be an economic proof or go into the gory details of exactly how the difficulty adjustment works. The aim is just to see the high level effects of how the system pans out without getting lost in the nitty gritty details of what is ultimately a very complex system.

Pretty much every time the word “miner” is mentioned in an Ethereum discussion someone will claim that miners are paid too much and miners will respond saying they’re struggling to survive. Turns out, both can be simultaneously true and in fact it’s pretty much the expected case.

If you’re getting into staking ETH on the beacon chain, it’s important to know what you’re actually signing up for. There’s lots of articles about how easy it is and how awesome, intelligent and good looking stakers are, so here’s a few hard truths you should make sure you’re ok with…

Short Term Stuff

Most people know this stuff so I’m just going to skim over it.

No Withdrawals Yet

You can’t withdraw staked ETH or rewards yet. There isn’t a fixed timeline on when you will be able to.

Through the beacon chain testnets people have been gradually developing better ways to evaluate how well their validators are working. Jim McDonald made a big leap forward defining attestation effectiveness which focuses on the inclusion distance for attestations. That revealed a bunch of opportunities for clients to improve the attestation inclusion distance and attestation effectiveness scores improved quite dramatically until now scores are generally at or close to 100% pretty consistently.

But then beaconcha.in added a view of how much was earned for each attestation. Suddenly people were left wondering why some of their “rewards” were negative even though the attestation was included quickly. It became obvious that inclusion distance was only one part of attestation rewards.

Last week I put together a video explaining attestation inclusion and delays in ETH2. This is one of the keys to maximising your validator rewards and the health of the network.

It’s like something out of a Sherlock novel – the doors are all locked, windows barred and yet somehow the jewels have been stolen. Or in this case, every block created forms a single chain and yet somehow the validator client has wound up on the wrong fork. Fortunately Sherlock, or rather Cem Ozer, was on the case. This is the story behind those very annoying “Produced invalid attestation” errors that Teku would sometimes generate.

Occasionally the term “weak subjectivity period” pops up in Eth2 discussions. It’s a weird concept that you can usually just watch fly by and not miss too much. But when you’re talking about how to sync an existing Eth2 chain it becomes quite important. Probably the best resource for it is Vitalik’s post: Proof of Stake: How I Learned to Love Weak Subjectivity I’ve struggled to get my head around it and why it matters so am writing up my current understanding. There is almost certainly at least one mistake in here somewhere…

One subtle detail in the way Ethereum works is that there is a difference between Ommers and Non-Canonical Blocks. It’s common for people to use the term Ommer for both of these and most of the time the difference doesn’t matter but sometimes it does.

So what is a non-canonical block? Non-canonical blocks are ones which a client imports but which don’t wind up on the canonical chain. Maybe they were on the canonical chain for a while and then a re-org switched to a different chain or maybe they were imported to a fork and spent their entire life languishing there. Either way, they don’t form part of the current consensus chain and have absolutely no effect on the world state. It’s like their transactions have never been executed and no one gets any form of miner reward for them. Non-canonical blocks must be entirely valid blocks that could form part of the canonical chain, but weren’t because we found a better chain.

Continuing our adventures in setting up a private beacon chain… Previously we got an Eth 1 chain with the deposit contract and successfully sent a deposit to register a new validator. So far though, we have no way of telling if it actually worked. What we need is an actual beacon chain client. Enter Artemis…

Step 3 – Add a Beacon Chain Client

We’ll be using Artemis, partly because I’m incredibly biased and partly because it happens to support the exact version of the deposit contract we’ve deployed (what a coincidence!).

I’ve started a little side-project to setup a “private beacon chain”. The aim is to better understand how the beacon chain works and start to discover some of the things still required to be built or fixed in clients before it can officially launch.

So what is a private beacon chain? It’s intended to be an entirely self-contained, runs-on-my-laptop instance of the beacon chain, run as a small-scale simulation of how the real beacon chain will be fired up.

I’ve written up a simple little EIP to specify some standard names and meanings for Prometheus metrics that are common across Ethereum clients: EIP-2159.
It’s pretty simple and only defines four metrics but that’s enough to create quite powerful overview dashboards that would work across multiple clients.

I’ve had the opportunity to do some proof of concept development of the Ethereum state-rent proposal that Alexey Akhunov has been leading on the Pantheon code base. The proposal evolved as the work continued so the actual implementation is now a lot simpler than described in that PDF.

Note that the aim is to explore what it takes to implement the proposal, not to create production ready code.  The current work is all available on on my state-rent branch.

This week, the work I’ve been doing for the past 6 months, and that PegaSys has been working on for the past 18 months or so was released into the world. Specifically we’ve released Pantheon 0.8.1, our new MainNet compatible, Apache 2 licensed, Java-based Ethereum client. And it’s open source.

I’m pretty excited about it on a few fronts. Firstly I think it’s a pretty important thing for the Ethereum community. To be a healthy ecosystem, Ethereum needs to have diversity in its clients to avoid a bug in one client taking out or accidentally hard forking the entire network. Currently though, Geth and Parity dominate the Ethereum client landscape. Pantheon clearly won’t change that in the short term, but it is backed by significant engineering resources to help it keep up with the ever changing Ethereum landscape and be a dependable option.

This is the second article on things I found particularly interesting in the Ethereum yellow paper.  The first is “What’s on the Blockchain?” and the same disclaimers apply: I’m no expert and you should go verify any claims I’m making before depending on them. Comments and corrections are most welcome either via email or @ajsutton on twitter.

One of the little details in the way Ethereum works is the idea of a “nonce” attached to each account and transaction. It’s a small but important detail.

To understand the details of how Ethereum works I’ve been working my way through the yellow paper. And since there’s no better way to really understand things than to explain them, I thought I’d write up my own explanation for some of the things that stood out to me.

Fair warning: I’m no expert, there’s guaranteed to be sloppy usage of terminology and a high likelihood of mistakes. If you’re just looking for an easier to read version of the yellow paper, try the beige paper, it’s had some actually knowledgable people look it over. Comments and corrections are most welcome either via email or @ajsutton on twitter.

Yesterday I spun up two new nodes on Prater, and sent deposits for 5000 validators to run on each. Due to the restrictions on how many new validators can activate per epoch (currently 4 for Prater), the number of active validators are currently gradually increasing giving a really nice insight into the cost of adding new validators.

This is most clearly seen by graphing the number of active validators and CPU usage on a single chart.  Here the green is CPU usage and the yellow the number of active validators.

The merge is at the point where real interop testing can begin with testnets spun up involving multiple clients. This video shows a testnet running locally with four nodes:

• Teku & Geth
• Teku & Besu
• Lighthouse & Geth
• Lighthouse & Besu

Each node has 1/4 of the validators and the network transitioned through from the initial phase0, to Altair and then then merge after which the two chains merge into one.  Also shows sending transactions with MetaMask post-merge and how the user experience is largely unchained. Ethereum keeps on being Ethereum but now with PoS instead of PoW.

Over on Twitter, there was an interesting discussion about the importance of client diversity for the beacon chain. As part of that cyber_hokie asks:

cyber_hokie – @cyber_hokie: As the majority client, isn’t it more likely that issues with Prysm would cause more severe financial penalties for minority clients voting on an alternative chain under accountable safety given Prysm validators likely hold the 2/3 threshold?

It’s important to note up front that this isn’t a Prysm specific issue. This isn’t a criticism of Prysm at all. All the major beacon chain clients are great – the issue here is any one client getting too large a share of validators. How much is too large? In an ideal world every client would have less than 1/3rd of validators. Then a bug in a single client doesn’t prevent the chain from continuing to finalise. More than 2/3rds majority is a really big problem because of the potential for those clients to finalise on an incorrect fork.

One of the most commonly overlooked steps when setting up a beacon node is to setup port forwarding on your router to ensure that other nodes can connect in to yours. This is often overlooked, or not done correctly, because there are so many different routers, all with different, fiddly interfaces that providing simple instructions is essentially impossible and because your node will generally work even without it by connecting outbound to other peers even though none can connect inbound.

I got the chance to talk with Pooja Ranjan and Alita Moore from the Ethereum Cat Herders as part of their PEEPanEIP series last week: PEEPanEIP #38: Altair in Teku with Adrian Sutton. It was a chance to talk through the actual implementation details required to support the Altair fork.

Mostly listing here for my own record, but if you haven’t seen it and you care at all about Ethereum hopefully it will be interesting.

Note: This post is deliberately high level. It doesn’t attempt to be an economic proof or go into the gory details of exactly how the difficulty adjustment works. The aim is just to see the high level effects of how the system pans out without getting lost in the nitty gritty details of what is ultimately a very complex system.

Pretty much every time the word “miner” is mentioned in an Ethereum discussion someone will claim that miners are paid too much and miners will respond saying they’re struggling to survive. Turns out, both can be simultaneously true and in fact it’s pretty much the expected case.

If you’re getting into staking ETH on the beacon chain, it’s important to know what you’re actually signing up for. There’s lots of articles about how easy it is and how awesome, intelligent and good looking stakers are, so here’s a few hard truths you should make sure you’re ok with…

Short Term Stuff

Most people know this stuff so I’m just going to skim over it.

No Withdrawals Yet

You can’t withdraw staked ETH or rewards yet. There isn’t a fixed timeline on when you will be able to.

Through the beacon chain testnets people have been gradually developing better ways to evaluate how well their validators are working. Jim McDonald made a big leap forward defining attestation effectiveness which focuses on the inclusion distance for attestations. That revealed a bunch of opportunities for clients to improve the attestation inclusion distance and attestation effectiveness scores improved quite dramatically until now scores are generally at or close to 100% pretty consistently.

But then beaconcha.in added a view of how much was earned for each attestation. Suddenly people were left wondering why some of their “rewards” were negative even though the attestation was included quickly. It became obvious that inclusion distance was only one part of attestation rewards.

Occasionally the term “weak subjectivity period” pops up in Eth2 discussions. It’s a weird concept that you can usually just watch fly by and not miss too much. But when you’re talking about how to sync an existing Eth2 chain it becomes quite important. Probably the best resource for it is Vitalik’s post: Proof of Stake: How I Learned to Love Weak Subjectivity I’ve struggled to get my head around it and why it matters so am writing up my current understanding. There is almost certainly at least one mistake in here somewhere…

I’ve written up a simple little EIP to specify some standard names and meanings for Prometheus metrics that are common across Ethereum clients: EIP-2159.
It’s pretty simple and only defines four metrics but that’s enough to create quite powerful overview dashboards that would work across multiple clients.

This week, the work I’ve been doing for the past 6 months, and that PegaSys has been working on for the past 18 months or so was released into the world. Specifically we’ve released Pantheon 0.8.1, our new MainNet compatible, Apache 2 licensed, Java-based Ethereum client. And it’s open source.

I’m pretty excited about it on a few fronts. Firstly I think it’s a pretty important thing for the Ethereum community. To be a healthy ecosystem, Ethereum needs to have diversity in its clients to avoid a bug in one client taking out or accidentally hard forking the entire network. Currently though, Geth and Parity dominate the Ethereum client landscape. Pantheon clearly won’t change that in the short term, but it is backed by significant engineering resources to help it keep up with the ever changing Ethereum landscape and be a dependable option.

There’s an exceptionally valuable set of ethereum reference tests that are run by most or all of the different Ethereum clients to ensure they actually implement the specifications in a compatible way. They’re one of the most valuable resources for anyone developing an Ethereum client.

The Aleth project maintains the official test client called testeth but it’s a little cryptic to work out how to actually run things with it and then use that to debug failures happening in the client you’re actually developing. So this is what I’ve found useful:

This is the second article on things I found particularly interesting in the Ethereum yellow paper.  The first is “What’s on the Blockchain?” and the same disclaimers apply: I’m no expert and you should go verify any claims I’m making before depending on them. Comments and corrections are most welcome either via email or @ajsutton on twitter.

One of the little details in the way Ethereum works is the idea of a “nonce” attached to each account and transaction. It’s a small but important detail.

To understand the details of how Ethereum works I’ve been working my way through the yellow paper. And since there’s no better way to really understand things than to explain them, I thought I’d write up my own explanation for some of the things that stood out to me.

Fair warning: I’m no expert, there’s guaranteed to be sloppy usage of terminology and a high likelihood of mistakes. If you’re just looking for an easier to read version of the yellow paper, try the beige paper, it’s had some actually knowledgable people look it over. Comments and corrections are most welcome either via email or @ajsutton on twitter.

For many years I’ve been using the free tier of Google Workspace (previously Google Apps for Your Domain) to host my email. While I almost never use the actual web interface itself having google handle the IMAP hosting and spam filtering was an awful lot easier than hosting it myself. Of course Google are now ending their free tier and requiring all users to pay – fair enough, but the cost per user is just too much for me to justify it.

A few days ago I talked about the Email and P Myth, but didn’t explain why it’s so frustrating for editor developers that people keep wanting to use BR tags instead of P tags. It’s not actually because we’re fastidious about using the correct semantic HTML, though obviously we do recommend that, it’s because a lot of concepts that user’s expect simply aren’t possible to implement sensibly if you don’t use the correct paragraph level elements.

One of the greatest frustrations for anyone who develops an HTML editor is the constant supply of people who are convinced they want to use BR tags instead of P tags. Most of these are just people who don’t want “double spacing” and they’re happy once you give them the magical CSS incantation:

p {
  margin-top: 0;
  margin-bottom: 0;
}

The other group however are people writing HTML emails who insist that P tags are completely incompatible with some email clients and cause rendering problems. At one point it seems that Yahoo! Mail really did strip them entirely but now it just applies a different default CSS styling (ironically, the magical CSS above that removes the extra spacing). So if you naively use P without specifying the padding you want, some clients will display a blank line between paragraphs, others, notably Yahoo!, will push all the lines immediately under each other. The solution is of course the opposite magical CSS incantation:

I got the chance to talk with Pooja Ranjan and Alita Moore from the Ethereum Cat Herders as part of their PEEPanEIP series last week: PEEPanEIP #38: Altair in Teku with Adrian Sutton. It was a chance to talk through the actual implementation details required to support the Altair fork.

Mostly listing here for my own record, but if you haven’t seen it and you care at all about Ethereum hopefully it will be interesting.

This week, the work I’ve been doing for the past 6 months, and that PegaSys has been working on for the past 18 months or so was released into the world. Specifically we’ve released Pantheon 0.8.1, our new MainNet compatible, Apache 2 licensed, Java-based Ethereum client. And it’s open source.

I’m pretty excited about it on a few fronts. Firstly I think it’s a pretty important thing for the Ethereum community. To be a healthy ecosystem, Ethereum needs to have diversity in its clients to avoid a bug in one client taking out or accidentally hard forking the entire network. Currently though, Geth and Parity dominate the Ethereum client landscape. Pantheon clearly won’t change that in the short term, but it is backed by significant engineering resources to help it keep up with the ever changing Ethereum landscape and be a dependable option.

For the beacon chain MainNet, I’ve setup an instance of chaind – a handle little utility that connects to a beacon node (like Teku) and dumps out the chain data to a database so its easy to run various queries against. While you can do this with various REST API queries and custom code, having it all accessible by SQL makes ad-hoc queries a lot easier and lets you add it as a datasource for Grafana to display lots of powerful metrics.

In Ethereum, there are often short forks near the head of the chain because two miners found the PoW solution for new blocks at around the same time. Only one of those chains will ultimately wind up being considered the canonical chain and any blocks from non-canonical chains wind up having no effect on the state. Non-canonical blocks may wind up being added as ommers in future blocks, but everything in this article applies regardless of whether that happens.

There are a number of browser plugins which inject additional JavaScript APIs into the DOM so websites can take advantage of the plugin functionality.  One example of that is MetaMask which “brings Ethereum to your browser”. This allows any website the user visits to detect that the plugin is installed by checking for the presence of those APIs which may aid them in targeting attacks such as the recent spate of phishing attacks against MetaMask users. So there’s a proposal in place to require websites to get specific authorisation from the user before the APIs will be injected.  And since injecting an API to allow the website to request access would defeat the point, it uses window.postMessage:

Interesting review of the regulatory landscape around crypto-currencies. There are a lot of echo’s of issues with the over-the-counter nature of most FX trading, albeit with even less enforced regulation and uncertainty.

Bitcoin Redux explains what’s going wrong in the world of cryptocurrencies. The bitcoin exchanges are developing into a shadow banking system, which do not give their customers actual bitcoin but rather display a “balance” and allow them to transact with others. However if Alice sends Bob a bitcoin, and they’re both customers of the same exchange, it just adjusts their balances rather than doing anything on the blockchain. This is an e-money service, according to European law, but is the law enforced? Not where it matters. We’ve been looking at the details. Source: Bitcoin Redux: crypto crime, and how to tackle it | Light Blue Touchpaper Also interesting to note is that most of the regulation required is already in place and just needs to be enforced. In most cases there isn’t any need for radical rethinking of laws, just apply the current laws about treating consumers fairly and Know-Your-Customer to this new technology.

A fun thing about programming is that most days, you make progress. Maybe you fix some issues, maybe you add a feature, maybe you build towards something bigger. Your code moves ever forward. Until it doesn’t. On occasion, you will hit a Bug. Not a mundane bug, some trifle you can fix in an hour, or even a day. This is a true Bug. One that defies reason. One that evokes a “that’s not possible,” a “how could this even happen?”, or most dreadfully, a “could there be a bug in the compiler?” Hold on kids, we’re going hunting. Source: The Great Bug Hunt – Allen Pike

Essentially, the Docker approach boils down to downloading an unsigned binary, running it, and hoping it doesn’t contain any backdoor into your companies network. Feels like downloading Windows shareware in the 90s to me. When will the first docker image appear which contains the Ask toolbar? The first internet worm spreading via flawed docker images? Source: The sad state of sysadmin in the age of containers There’s certainly some truth to that.  I’m not entirely sure that the compile-from-source approach was actually that much more secure as it was practically impossible to verify the source code anyway. At which point it makes little to no difference if you’re downloading random binaries off the internet or random source code – either way you’re implicitly trusting the source. Verifying signatures for the stuff you download would be a big improvement and many of the newer deployment approaches are very lacking in this area, but it still depends on having a trustworthy way of getting the signature to verify.

After 7 years at LMAX, I’ll be winding up with them at the end of the month. It’s been a pretty remarkable journey and certainly not an easy decision to move on but ultimately another opportunity came up that I just couldn’t refuse.

I had intended to write up some reflections on the past 7 years at LMAX but I just can’t find the words. Suffice to say, it’s been an amazing journey full of wonderful people, big technical challenges and lots of innovative solutions. The early years in particular were just a fire hose of new knowledge to try and absorb. There have been many changes over the years, but it’s still an amazing place to work and leaving is sad.

Thanks to Oncle Tom’s “SSH deploys with Travis CI“, I’ve now fully automated Moolah deployment for both the client and server.

Previously the client was automated using a cronjob that pulled any changes from GitHub, built and ran the tests and if that all worked resync’d the built resources to apache’s docroot. That meant my not-particularly-powerful server was duplicating the build and test steps already happening in Travis CI and there was up to a 10 minute delay before changes went live.

I’ve recently started a little side project to create a java implementation of a CommonJS compiler: commonjs-java which is available on github under the Apache 2 license. It’s output is targeted at browsers, bundling together all the dependencies into one javascript file.

There are plenty of utilities that can do that for you on the command line or as part of a build process (e.g. browserify) and in most cases you should stick with them, but I had a couple of requirements that drove me to want a Java implementation:

In programming courses one of the first thing you’re taught is to avoid “magic literals” – numbers or strings that are hardcoded in the middle of an algorithm. The recommended solution is to extract them into a constant. Sometimes this is great advice, for example:

if (amount > 1000) {
  checkAdditionalAuthorization();
}

would be much more readable if we extracted a ADDITIONAL_AUTHORIZATION_THRESHOLD variable – primarily so the magic 1000 gets a name.

I’ve been playing around with ruby on rails recently, partly to play around with rails and partly to take a run at a web app I’ve been considering (which I’ve open sourced because why not?).

It turns out the last time I played with it was back in 2005 and slightly amusingly my thoughts on it haven’t changed all that much. The lack of configuration is still good, but the amount of magic involved makes it hard to understand what’s going on. The ease of finding documentation has improved dramatically – 10 years of blog posts really help. I’m still using TextMate and it’s still annoying that I can’t click a method name to jump to it’s definition – I hear good things about RubyMine but I’m not keen to invest that kind of money in what may be a very short-lived experiment.

A little while back I said that LMAX ran around 11,000 end to end tests in around 50 minutes. Since then we’ve deployed some new hardware to run our continuous integration on, plus continued building new stuff and are now running about 11,500 tests in under 20 minutes.

A large part of the speed boost is extra VM instances but also the increased RAM allocation available to each VM has allowed us to increase a number of limits in the system and we can now run more tests concurrently against each VM instance.

Even after a magnum opus on the DSL LMAX uses for acceptance tests, there’s one crucial feature that I haven’t mentioned: the use of aliases to allow tests to use simple, meaningful names while ensuring that uniqueness constraints are met.

Creating a user with our DSL looks like:

registrationAPI.createUser("user");

You might expect this to create a user with the username ‘user’, but then we’d get conflicts between every test that wanted to call their user ‘user’ which would prevent tests from running safely against the same deployment of the exchange.

Pre-Java 8:

ThreadLocal<Foo> foo = new ThreadLocal() {
  @Override
  protected Foo initialValue() {
    return new Foo();
  }
};

Post-Java 8:

ThreadLocal<Foo> foo = ThreadLocal.withInitial(Foo::new);

Nice.

I’ve spent the evening emailing back and forth with Chris from Sonos’ tech support about a strange issue I’ve experienced where the Sonos app can’t connect to the playbar (on both OS X and iOS). It turns out the problem is that my DLink wifi access point doesn’t handle multicast traffic properly – Chris knew that almost immediately but took me through each step carefully checking every assumption and taking the time to ask about settings in the terms DLink uses instead of generic ones. By the end of it all I had a solid understanding of what the problem was and a simple way to describe it to DLink’s support to see if it could be fixed. A little creativity on my end has even given me a good work around.

My wife and I gave each other a Sonos playbar for Christmas to improve the clarity of our TV. The initial setup was excellent – clearly stepping through each required step and very cleverly detecting the type of TV remote I have and automatically reacting to it’s volume controls so I can carry on using the TV remote as usual.

The only problem is that my Panasonic TV doesn’t provide a way to disable the internal speakers. So the playbar and the TV were both outputting sound which sounds pretty awful.  There’s two ways to solve this:

Before writing any even half serious bash script, stop and read:

• Use the Unofficial Bash Mode
• How “Exit Traps” Can Make Your Bash Scripts Way More Robust And Reliable

Any other particularly good articles on writing reliable bash scripts that should be added to this list?

In Software is sometimes done Rian van der Merwe makes the argument that we need more software that is “done”:

I do wonder what would happen if we felt the weight of responsibility a little more when we’re designing software. What if we go into a project as if the design we come up with might not only be done at some point, but might be around for 100 years or more? Would we make it fit into the web environment better, give it a timeless aesthetic, and spend more time considering the consequences of our design decisions?

Apple have released Swift, their new programming language – designed to be familiar to Objective-C programmers and work well with the existing Cocoa frameworks. It’s far too soon to make substantial judgements about the language – that can only come after actually using it in real projects for some time. However, there’s nothing that stands out as incredibly broken, so with Apple’s backing it’s extremely unlikely that it won’t become a very commonly used language. After all, there’s plenty wrong with every other programming language and we manage to make do with them.

Did you know that it's perfectly fine to enjoy programming in both static and dynamic languages?

— Honza Pokorny (@_honza)

May 26, 2014

I do a lot of coding in Java and JavaScript and it’s never bothered me that one has static types and one dynamic (and I’ve used plenty of other languages from both camps as well – it amuses me slightly that the two I use most often tend to be viewed as the worst examples of each type). I can see pros and cons of both approaches – static typing detects a bunch of common errors sooner which saves time but there are inevitably times where you wind up just fighting the type system which wastes time again. Better statically typed languages waste less of your time but at some point they all cause pain that’s avoided with dynamic languages. It winds up coming down largely to personal preference and is a little affected by what you’re building.

Writing automated tests to prove software works correctly is now well established and relying solely or even primarily on manual testing is considered a “very bad sign”. A comprehensive automated test suite gives us a great deal of confidence that if we break something we’ll find out before it hits production.

Despite that, automated tests shouldn’t be our first line of defence against things going wrong. Sure they’re powerful, but all they can do is point out that something is broken, they can’t do anything to prevent it being broken in the first place.

Kevlin Henney’s talk from YOW is a great deconstruction of the SOLID principles, delving into what they really say as opposed to what people think they say, and what we can learn from them in a nuanced, balanced way.

Far too often we take these rules as absolutes under the mistaken assumption that if we always follow them, our software will be more maintainable. As an industry we seem to be seeking a silver bullet for architecture that solves all our problems just like we’ve searched for a silver bullet language for so long. There is value in these principles and we should take the time to learn from them, but they are just tools in our toolkit that we need to learn not only how to use, but when.

Human Resource Executive Online has an interesting story related to my previous entry:

Ask yourself this question: If you were offered a new job in another city where you have no ties or networks, and you suspected that the job would probably not last more than three years (which is a good guess), how much of a raise would they have to give you to get you to move? Remember, you’d likely be trying to find a new job in a place where you don’t have any connections when that job ended. I suspect it would be a lot more than most employers are willing to pay.

Testing time related functions is always a challenge – generally it involves adding some form of abstraction over the system clock which can then be stubbed, mocked or otherwise controlled by unit tests in order to test the functionality. At LMAX we like the confidence that end-to-end acceptance tests give us but, like most financial systems, a significant amount of our functionality is highly time dependent so we need the same kind of control over time but in a way that works even when the system is running as a whole (which means it’s running in multiple different JVMs or possibly even on different servers).

One of the things we tend to take for granted a bit at LMAX is that we store the results of our acceptance test runs in a database to make them easy to analyse later.  We not only store whether each test passed or not for each revision, but the failure message if it failed, when it ran, how long it took, what server it ran on and a bunch of other information.

The key to making continuous integration work well is to ensure that the build stays green – ensuring that the team always knows that if something doesn’t work it was their change that broke it. However, in any environment with a build pipeline beyond a simple commit build, for example integration, acceptance or deployment tests, sometimes things will break.

When that happens, there is always a natural tendency to commit an additional change that will fix it. That’s almost always a mistake.

Out of the box, Fedora 19 doesn’t have support for the broadcom wifi chip in the MacBook Pro 15″ Retina. There are quite a few complex instructions for adjusting firmware and compiling bits and bobs etc, but the easiest way to get it up and running on Fedora is using rpmfusion.

You can do it by downloading a bunch of rpms and stuffing around with USB drives, but its way easier if you setup network access first via either a thunderbolt ethernet adapter (make sure its plugged in before starting up as hotplugging thunderbolt doesn’t work under Linux), or via bluetooth.  The bluetooth connection can either be to a mobile phone sharing its data connection or if you have another Mac around, it can share its wifi network over bluetooth (turn on Internet Sharing in the Sharing settings panel).

Ah Apple maps, ever the source of a good sensationalist headline. This time the Victorian police have warned people not to use Apple Maps to get to Mildura. This is definitely a bug with Apple maps, no question it should be and has been fixed.  What’s interesting though is that the Victorian police thought it would be easier to attempt to notify every iOS 6 user about the problem via the media and get them to use an alternate mapping application than it would be to call Apple and get them to fix the source data.

From Straights Times:

A flier aboard Sunday’s Swiss airline flight – a 57-year-old Chinese man […] felt disturbed during his meal when the passenger in front of him reclined his chair.[…]

“The older of the two felt disturbed during his dinner. When the younger did not respond to his protests, he hit him on the head with the flat of his hand. It was a real slap,” said the guide, Ms Valerie Sprenger.

CSS3 makes it trivial to have alternating row colours for tables, but when the table is in a fixed height scroll panel, it’s much more difficult to have those alternating row colours extend beyond the bottom of the table content to fill the available space. Here’s the approach I use.

First let’s start with a simple table with alternating colours in a scroll panel:

That gets us alternating row colours where there is table content. To get it to extend beyond the table content we need to add a filler row:

Following on from my last post, Jason Adam Young has some excellent advice to help you continually get better and better and building software and perhaps more importantly, be more and more valuable to whoever you happen to be working for.

This afternoon during an interview, a potential new hire at LMAX asked what LMAX could offer in the way of career progression. Since I’ve been thinking a fair bit about what comes next in my career now that I’m moving back to Australia I thought I had a pretty good answer and that it might be worth sharing here2{#footlink2:1323196996971.footnote}.

In most industries career progression is signified by ever fancier sounding job titles, but for a software developer, the fancier the title sounds the less likely you are to actually do any development. Essentially, career progression often amounts to career migration – into “architecture”1{#footlink1:1323196219398.footnote} or managements. If you really love developing software though you need to look at it a different way. If I spend the rest of my career as a “Senior Software Engineer” I’ll be pretty happy, so long as I can keep honing my skills and becoming more valuable to the companies I work for. Ideally I’d like the pay I take home to grow in line with that value.

After four years living in the UK, my wife and I have decided to move back to Australia mid-February 2012 to be closer to family. As such, I’m now starting to look at job opportunities back in Oz. If you’re hiring, I’d love to hear from you and discuss how we might work together. My CV is available online to give you an idea of my experience and of course the backlog of this blog shows some of my thinking and learning throughout the years1{#footlink1:1322426906768.footnote}.

James Turner:

Unneeded interfaces are not only wasted code, they make reading and debugging the code much more difficult, because they break the link between the call and the implementation. The only way to find the implementing code is to look for the factory, and see what class is being provisioned to implement the interface. If you’re really lucky, the factory gets the class name from a property file, so you have to look another level down.

This is the 1001th post to this blog. Fittingly, the 1000th post was actually reasonably technical. Being late to the blogging craze I started on 25th January 2004, 2448 days ago. A average posting rate of roughly every 2 and a half days is pretty good – much better than I would have expected actually.

It’s already been mentioned on my travel/personal stuff blog but news this good should be shared as widely as possible. My wife and I are expecting a baby in April of next year. We’re very excited and pleased that Mum and baby are both doing well. If you’re interested in how things progress, we’ll be posting that over at “The Suttons”.

Despite all the hype about how everyone’s making advertising more targeted and therefore more useful to people, you still get gaffs like this one:

The advertising industry is booming and ads are popping up in more and more places, but the quality of ads are still really bad. It’s no wonder ad blocking technology has become so popular.

Normally applets run in a security sandbox, much like JavaScript, but with signed applets those restrictions are lifted and the applet can do anything. At least, that’s the theory – in practice most implementations leave a security manager instance installed but with significantly reduced restrictions (such that most programs will never be affected by them). To get around these remaining restrictions, applets can remove or replace the security manager:

System.setSecurityManager(null);

Having set a null security manager though, you will start to get NullPointerExceptions in Firefox on OS X when you try to call Java methods from JavaScript (via LiveConnect). Basically, Firefox is assuming that there will always be a security manager in place. To get LiveConnect working again, rather than setting the security manager to null, simply create a new security manager that allows everything:

I’ve been trying to solve this problem pretty much since Aperture 3 came out with it’s Places/GPS support. Every time I added location information to a photo, it wound up being tagged as where I really took the photo but also a completely incorrect, but consistent location (for me it was always The House of Binns in Scotland). Location information pops up in so many different places in Aperture and displayed in so many different ways that it was really hard to track down what was going on – sometimes it would have the first part of the place name right, but then had the country incorrectly shown as Scotland.

Anyone who’s done much work with JavaScript has probably discovered that the selection APIs are completely different in Internet Explorer vs the rest of the world, what comes as a bit more of a surprise once you start using them in anger is that FireFox is actually quite different to all the other W3C compliant browsers in an important way as well.

If all you ever want to do is retrieve the selection or you only want to work with ranges rather than selection, you’ll probably never encounter this, but as soon as you use window.getSelection().addRange() you’re going to get bitten. The difference is that FireFox will preserve whatever range you give it precisely as you original created it. The other browsers won’t.

This has been said before but apparently even really smart people aren’t listening. So here’s my top two tips for job applications:

1. Tailor your résumé. If you’re applying for a JavaScript job, highlight the JavaScript experience you have first and foremost. Even something as simple as the order you list skills in makes a difference – if JavaScript is at the end of a list of skills, it’s probably not a priority, if it’s first it makes it seem more important to you, so you seem more qualified.
2. Get a blog, make it look good, fill it with good content.

The corollary to number 2 is that you should set your FaceBook page to private – it never seems to be what you want to have turn up. It’s surprising how many blogs have relatively good technical content but look terrible. Even if you’re not going to be applying for design jobs, having a decent layout without broken images is always helpful.

From yield thought – On The Fear of Reading Code:

When I was learning to program someone told me that I should try to read as much code as possible. Budding genius that I was, I thought this was advice for stupid people who needed to learn from their betters. I thought the only reason to read code was to pick up tricks from it.

How wrong I was. The real reason to read code has nothing to do with learning from the programmer who wrote it.

Thanks to a tweet from @bdelacretaz I discovered Apache Pivot today. It does indeed have a nice website, and the library itself looks great. It’s pitched as an RIA framework and mostly seems focussed on browser based deployment via applets. However, pivot apps can also be deployed as standalone applications which is where I think it’s most promising.

The killer feature as far as I can see, is the much richer set of components that are available, combined with a better set of customisation points such as effects, transitions, web queries and data binding. There are libraries to do most or all of this with Swing, but since it wasn’t designed for it up front, they tend to feel pretty clunky and “bolted-on”. Hopefully Pivot can avoid that.

Chris J Davis in Lessons for the Newly Minted:

As you can clearly see, when you have tight deadlines, and mountains of work, refactoring existing code that works is highly unadvisable. As a developer you must make strategic decisions about where your time is spent, and this must be informed and balanced by the needs of the company. Should you strive to craft breathtakingly beautiful code? Yes, but not at the expense of the overall velocity of your development schedule. Sometimes good enough is beautiful. Amen to that.

I’ve been doing a bit more Googling and seem to have hit onto a few key articles that tie into a web of articles around build systems. There’s certainly a lot more options that I’d originally thought.

Build Tools

• Gant – Not really a build tool itself but an interesting library for scripting ant tasks in Groovy.
• Gradle – Came out of the work on Gant and provides a full build tool with Groovy scripting and leveraging Ant tasks quite heavily under the hood. Uses Ivy for dependency management and promises good things for multi-project builds. Most interestingly though it has transitive dependency support without the need for remote repositories or pom/ivy files.
• Schmant – Aims to be comparable to Ant in features but nicer and easier to work with. Uses Java 6 scripting to let you use a wide range of languages to script the build, but the sample build files look a little complex still. Most interesting is the TaskExecutor support for running different build tasks in parallel threads – not sure if it’s easier to use than ant’s parallel task though.
• Apache Buildr – rake for Java I guess. Could also be described as maven done right – the build files are kind of POM like, but are actually full ruby classes. I played with this one a bit and it’s very impressive, though its transitive dependency support is still a bit immature.

Maven Info

Also stumbled across some good Maven articles:

For ages now I’ve been keeping the EditLive! installer for my demo environment in subversion. Having the ability to roll back quickly and easily if something goes wrong has given me the confidence to track the development branch reasonably closely and then be able to show people the features that have just finished being developed which is fantastic market feedback. I’ve never actually had to rollback before, but today I found an alternative use: reproducing bugs.

Dave Walker – Proper Care and Feeding of Computing Consultants: Excellent set of things you should do to get the most out of a consultant visit. I haven’t been consulting very long and it’s not the main part of my job, but I’d add:

• Let them know the hours you want them to be on site. It’s no fun for a consultant to be sitting outside the building at 8am when you start at 9:30. Similarly if you have an awesomely laid back European culture and finish at 4 they may be able to get an earlier plane home so it’s good to know in advance.
• Make sure the consultant knows the exact name of your department (and how to pronounce it if it’s a foreign country to them)
• Make sure you’ve lined up the important people your consultant will need to work with so that they are available. You don’t want expensive consultants wasting time waiting for people to come back from a meeting or trying to find the right people in the maze of your company.
• When they arrive show them where the bathroom is – where to get water from is important too.
• Recommended travel details if the place is hard to get to. Of course, flying into a different country and driving works out well if you just so happen to be upgraded to a convertible for free…

Or as Reuters put it:

One in 7 who leave Guantanamo involved in terrorism Lies, damned lies and statistics.

Since my last question was so quickly solved thanks to Kaushal Sinha, I thought I’d try a simpler one. I need to set up a page in WebSphere Portal that doesn’t appear in any of the navigation structure but it does need a known URL. It’s used only as a landing point in the portal, has a portlet that asks a couple of questions and then sends the user on to the right place. It just doesn’t make any sense to navigate to that portal via the links at the top of the standard portal theme, but I can’t just remove permissions from the page because it needs to be available when they come in via external links (where it does make sense).

Our office in Windsor has a huge window at the front which makes a nice atmosphere on clear blue-sky days like today. Occasionally,it also provides entertainment, from the odd kid knocking on the window to watching the occasional drunk singing as he goes by.

When the Queen is staying just up the hill at Windsor castle, we often get a couple of the Queen’s guard on horse back come down for a coffee next door. Usually this means we look out the window at a horse’s backside, but sometime’s they’re nice enough to turn them round. The horses are absolutely huge and I’d always wondered how much of a run up they took to get on.

JavaScript libraries are awesome – they make it possible to develop cross browser coolness without going completely insane working around all the incompatibilities. The trouble is, we’re talking about JavaScript libraries. Instead of working around differences in browsers, now you work around differences and incompatibilities in JavaScript libraries.

While the simple response is to just pick one and stick with it, there are some problems with that:

The new banner on the Ephox.com homepage is a good example1. The site was really simple so I went ahead and picked the JavaScript library I knew best – Scriptaculous and Prototype. I developed the script to get a nice cross fade between the two images and went to put it up on the live site. At that point, I discovered someone else at one point had decided to use a lightbox library. As Murphy would have it, it depended on JQuery.

Most developers think of software bugs as annoying but ultimately harmless. Sadly, as I found out today, that isn’t always true. It turns out that due to a bug in ScreenFlow, occasionally it will suddenly and inexplicably start playing extremely loud random noise instead of the soft, dulcet tones of the screen cast you were aiming for1.

I was wearing headphones at the time and even though the volume was turned down to just one click above mute, it didn’t seem to pay any attention to that and just about blew my eardrums out. I still have a headache from it.

From the Maidenhead Advertiser:

Look out for a giant pink fluffy testicle gracing the nation’s top sporting events this summer. Raising awareness of testicular cancer – good on him and best of luck to him:

“I’m going to go to whatever I can, but no doubt most of it will be sport-related. Tennis, football, cricket – whatever I can get to. I’ll be using public transport so I’m sure I’ll get well-noticed.” I’m sure he will…

I hadn’t realized this until Rob Weir pointed it out, but Windows XP moves to the “extended support” phase of it’s life cycle (also Office 2003) after today. So for most consumers, that means XP is now dead – though still more popular than Vista1{#footlink1:1239695774478.footnote}.

I have a lot of demo VMs set up with XP and I really don’t want to migrate them to Vista (let alone pay for that), but a demo on Linux just isn’t going to cut it with clients who use Windows exclusively. Demoing on OS X works but is too hard to control the environment since it can’t be a virtual machine. I do a lot of webinars so it’s really useful to have the VM set to 1024×768 without having to actually resize the screen down and unplug the second monitor.

Recently John Gruber started an uproar against the Diggbar – an URL shortening service that also adds frames. Frames are one of the most annoying inventions ever to come to HTML, even when used by the original site author. When used by third party’s they have a major impact on the usability for readers who can no longer bookmark, copy URLs or see where they actually are. Back in the 90s when frames were accepted practice, nearly every site would have a link saying something like “stuck in frames? click here” to help users get out.

Gianugo Rabellino, among a bunch of interesting stuff on Agile and Open-source development:

do-ocracy and his French-speaking twin JFDI I really must remember that phrase – just brilliant. Doug, you’d probably be interested in Gianugo’s thoughts as well.

Ok seriously, we’ve got to stop this tabs on top thing before it spreads any further.  That top bit of a window – it’s for dragging the window around and I really don’t want to have to watch carefully to see where I’m clicking in case I accidentally change tabs or grab a tab and throw it headlong into it’s own window. With Chrome I told myself it was justifiable because all the tabs are separate processes so clearly each process rendered it’s own toolbar and UI so essentially the tabs had to be at the top.  This justification is of course rubbish but it made me feel better.  Besides, Chrome is a Windows-only browser and they do all kinds of stupid things like that over there. Clearly that would never fly on Mac…

CMSWire – Web Content Management and Recession — Unlikely Duo?

Forrester’s Stephen Powers in the Q4 2008 Web Content Management Survey predicts that Web CMS industry will continue to grow despite the global economical difficulties, because it is an investment companies must make to remain competitive. Interesting survey results, particularly notable that 72% of respondents plant to increase their Web Content Management investment. The feeling I get right now, is that people are uncertain and are delaying many purchasing decisions or simply taking longer to make decisions, but not outright cancelling purchasing plans.

Handy little tool for checking how sites render in various versions of IE.  Must remember this for next time I’m redesigning a site.  It looks like they’re running VMWare and having a slight issue at the moment though…

It occurs to me that for the past two years I’ve written at least one post a week (not as much here, mostly over there). In the next week or two I’ll be spending most of my time writing demo scripts, white papers, various other technical documentation and a couple of speeches. I’m pretty sure if I counted it all up I’d have written way more English than I have Java or any other programming language. That’s not even counting email, which I’ve been writing tons more of.

Dylan posted a good blog post at a ridiculous time of night last night discussing software architecture, his role in it and more. Firstly, it’s really good to see these kinds of blog posts appearing – Ephox has gotten really slack at blogging and I think that’s a shame because we have a lot of good stuff to say. Actually publishing things makes you actually think it through and it allows people to build on those ideas and improve them. It’s always hard to find the time and energy to blog, but it’s worth it.

If you haven’t heard already, we’re spending our first Christmas away up in the Swiss Alps so we’ll be sure to get a white Christmas.  We actually found a self guided tour from an Australian company which includes the hotel, Christmas lunch, a sleigh ride and a suggested itinerary for exploring Switzerland. The train ticket they give us includes most of the travel around the country and discounts off the optional parts so we should wind up seeing quite a lot of the country.

James Duncan Davidson did the math and found he was better off without a land line phone:

Once the numbers have been looked it, however, it makes more sense to just go all the way. Of course, if $2400 per year was the only way to have decent access to the Internet, it’d be a different story. But it doesn’t need to cost that much anymore. So I’m cutting the twisted pair lines. Because, you know, they are historic and quant. Sure, a land line still sounds better than most mobile phone calls, but 3G calls are getting there in terms of clarity and on a good day sound as good to me as a land line. In a few years, they’ll be even better. Interestingly, I just got my phone bill from BT and am more glad than ever that I have a land line:

IWWCM has a component specifically to provide “inline editing” – in other words, adding links to content items so you can quickly edit, accept, reject or delete if you’re logged in as a user who has permission to do such things.  I have two problems with this approach:

1. You have to be logged in to see the links.
2. The links affect the way the page looks so you don’t see the site the same way as normal visitors.

Within a portal environment or even an intranet, neither of those problems really apply, but when you’re publishing out a public website they can be an issue.

You’re sitting on the train listening to music on your iPhone. Your wife rings, you answer it and the girl sitting cross from you suddenly gives you an odd look. What went wrong? When you have headphones in the iPhone makes no audible ringing noise. So people around you have no idea that you just took a call. Not that Id change this but it does generate some odd looks. Especially when you answer with “hey sweety”. Oh well.

For those who are interested, I’ve put together the collection of links that I found on getting MathML to render in browsers on LiveWorks! I’m a little unsure about the status of Safari and Opera so if anyone familiar with MathML in those browser could provide any info that would be greatly appreciated.

Also, as a bonus tip that I’ve picked up while researching XHTML modes in browsers, if you save a file with a .xhtml extension (or .xhtm or even .xht I think) browsers will actually use their XML parser to read and render the file. Much simpler than reconfiguring your web server to send the right mime type if you’re just testing stuff. Web servers with up to date extension to mime type mappings will also serve the file as application/xhtml+xml which is handy if you are serving static files too.

A couple of years back I went on a crusade against manual deployment of pretty much everything in Ephox. It was originally driven by a desire to “dog food” our products so we needed to automatically deploy each nightly build to our internal systems. From there it went out to automatic releases (still problematic due to a dodgy FTP server at our hosting provider) and got a fancy webapp front end that let you pick which version to deploy.

There is so much effort put into coming up with complex ways of filtering spam, yet somehow the blatantly obvious posts still get through. Google Blog search for sites linking to my blog is 100% spam and they are all the most common form of splog these days:

%NAME% wrote an interesting post today on %SITENAME%
Here’s a quick excerpt

%COPIED POST CONTENT%

Get more information from %SITENAME% %LINK%

The copyright of the above excerpt belongs to %SITENAME%. We welcome any comments from the copyright owner.

CMSWire – News You Can Use:

When the homepage is dominated by news you are not necessarily communicating more. In many situations, you are damaging your reputation as a quality news source. Forcing news into people’s faces just annoys them. The fact is, most intranets really don’t have that much news that’s globally interesting, so most intranets need to focus on the commonly used resources or the targeted information that’s specific to smaller groups – often that’s not PR written news or even news at all, more just status updates.

I’ve quickly come to realize that the mobile worlds has a huge dependency on synchronization tehnology to make things work smoothly. Toucan read your email on the phone and reply from your laptop. Read rss items should be synced and just about everything else on your phone should be synced with somewhere else. The problem is that generally synchronization support is lousy. NetNewsWire is too slow syncing feeds, Mail.app doesn’t seem to notice if a message changes from unread to read and the WordPress iPhone app doesn’t seem to download drafts that you created in the browser interface. Sync is the killer requirement that goes unsaid on mobile devices. You can spend as long as you like polishing he UI but if your synchronization isn’t seamless your app will be a chore to use. If you get it right users won’t notice at all.

I’ve been looking forward to the iPhone 2.0 OS and the App Store for a fair while, not because there was new functionality I desperately wanted on my iPhone but because having access to native resources should make the few common webapps I use better. I was particularly impressed by Apple’s notification system and looked forward to say having a flag on NetNewsWire showing me how many unread items I had or being able to be notified me when someone sent me an instant message.

Jeff Atwood posted a rant about the iTunes Music Store requiring iTunes to be installed in order to access it. In particular, he didn’t like how a link to something on the iTMS resulted in an error page if you didn’t have iTunes installed.

Is it so unreasonable to expect links in your browser to resolve to, oh, I don’t know, web pages containing information about the thing you just clicked on? Is there anything more anti-web than demanding users install custom software to display information that could have just as easily been delivered through the browser? It’s an exceptionally compelling argument but there’s one small flaw. If you’re not sure what it is, just drop me an email.

One of the challenges with many content management systems, and IBM’s is no exception, is creating short, clean URLs. As part of structuring and managing your content, the URL segments tend to build up to very long URLs. While most systems have a way to provide shorter aliases they need to be manually created and tend to just redirect to real URL rather than being the one canonical URL for content.

Some time ago now, James Robertson blogged about the poor state of Mac support in CMS products. Quite rightly he identified the WYSIWYG editor as the most common problem area which of course got my attention. It’s over six years ago now that Ephox switched over to Java from ActiveX to get support for Mac and it’s probably the smartest thing we’ve ever done. Not because we have vast numbers of Mac users, but because it only takes one Mac user to sink a deal.

Jef Atwood has discovered the implicit variable type declaration in C#:

BufferedReader br = new BufferedReader (new FileReader(name));

Who came up with this stuff?

Is there really any doubt what type of the variable br is? Does it help anyone, ever, to have another BufferedReader on the front of that line? This has bothered me for years, but it was an itch I just couldn’t scratch. Until now.

Actually, there is a question about the type of br – it could be a BufferedReader or it could be any superclass of BufferedReader and sometimes that distinction is important.  The most common example of this is in the collections API:

I got just got home from a very entertaining evening with some folk from the Web Content 2008 conference watching, or rather largely ignoring, an overall boring game of basketball between two teams I didn’t know from a bar of soap (for the record, the Celtics won and were premiers or something). Anyway, I found in my email an entire conversation within Ephox around this article on CMS Wire about the talk I gave today. It’s actually a very good summary of what I said and I hadn’t realized there was anyone from CMS Wire even at the conference (Rachelle, please do say hello tomorrow if you get this, I don’t know what you look like).

At the risk of becoming a link blog, anyone thinking about social software should go and read Dare Obasanjo’s latest post: Participation as Social Capital: The Fundamental Flaw of Social News Sites. I think the most telling part for me was:

Although turning participation in your online community into a game complete with points and a high score table is a good tactic to gain an initial set of active users, it does not lead to a healthy or diverse community in the long run. Digg and Slashdot both eventually learned this and have attempted to fix it in their own ways. The trouble with starting a new social site is always in bootstrapping. How do you get enough people using the site to start making the social aspects actually pay off? Ranking users is basically the default answer because it works so well with even a few users – it’s easy to get to the top and you feel just as special about getting there so you hang around and try and stay there so it helps to build the initial users of the site.

I’ve had this article by Seth Gottlieb open for a while now but quite frankly don’t have much I could add except to say go read it. The idea that content is more than just data because of how real people perceive and work with it leads to a huge difference in how you design user interfaces for content management systems. A content management system should be more than just a web interface to some great big database, it shouldn’t make users think the way the database is laid out, it should let them focus on the content and the true messages that it conveys.

For the first few years that I worked at Ephox there was a regularly recurring problem: how to let our clients know exactly what’s changed between versions. We were good at showing off the new features but never had an accurate list of which bugs were fixed and every so often a client would ask for that. It seems simple enough – just keep a changelog – but there were some challenges.

Ephox’s fearless leader, Andrew Roberts turned up in an interview with Randal Leeb-du Toit. Some good stuff in there, definitely worth a listen.

Reuters:

“Later this year, Vodafone customers in Australia, the Czech Republic, Egypt, Greece, Italy, India, Portugal, New Zealand, South Africa and Turkey will be able to purchase the iPhone for use on the Vodafone network,” Britain-based Vodafone said in a statement.

The plans for the iPhone in Australia will be very interesting – mobile internet is ridiculously expensive there at the moment and the iPhone is basically dependant on an unlimited data plan. While I was over in Australia recently with data roaming turned off my iPhone was pretty much reduced to a big, somewhat ineffective phone rather than a mobile internet sensation…

Business Week on the increasing number of companies that have at least some Macs:

In a survey of 250 diverse companies that has yet to be released, the market research firm Yankee Group found that 87% now have at least some Apple computers in their offices, up from 48% two years ago.

What’s interesting about this is that it’s one of the few surveys I’ve seen that doesn’t focus on raw market share. While hardware manufacturers might be interested to know that Apple has X% of the total market, software developers shouldn’t care.

It’s been echoing all over the blogosphere today that NYTimes.com “hand codes everything” instead of using a WYSIWYG editor. It all extrapolates from a fairly offhand reference by the design director, Khoi Vinh:

It’s our preference to use a text editor, like HomeSite, TextPad or TextMate, to “hand code” everything, rather than to use a wysiwyg (what you see is what you get) HTML and CSS authoring program, like Dreamweaver. We just find it yields better and faster results.

The server migration to a new data center last night went horribly wrong when the new IP didn’t (and still doesn’t) have a reverse DNS lookup. Sadly that means the secondary DNS where all the actual requests go to refused to pull the updates and the site was effectively down.

I think I’ve managed to sort out the DNS issues for all the domains hosted here now by making one of the secondary DNS’s the primary DNS and generally juggling things around. This is probably a smarter set up anyway so I’ll probably leave it this way in future. Still, annoying….

Arrived safely in the UK early this morning with no hassles from customs thanks to my shiny new visa. Somewhat ironically I also got my first serendipitous travel event from Drupal:

The problem of course being that this was my reminder to change my home location from Brisbane to London so I actually miss Andrew by a week. Never mind, he’s heading over here next anyway.

I woke up this morning to discover that I can no longer access my GMail account (using Google apps for your domain) via IMAP. It just returns an error saying that IMAP is not enabled for your account. The IMAP option has even disappeared from the web interface, now only allowing you to enable POP access. The same is true for a second Google Apps for your domain account that work uses.

Symptoms

• You have a NetGear FVS124G router providing VPN.
• You use IPSecuritas to let Mac users log into that VPN.
• VPN has worked in the past.
• VPN is no longer working and you haven’t changed any settings (in our case there was a power outage and the modem configuration didn’t come back as expected initially then was fixed so all the settings were the same as they used to be).
• In the IPSecuritas log you get a message stating “inappropriate sadb message passed.” and  Phase 1 never actually connects (thus Phase 2 times out because of a lack of Phase 1 connection).

Solution

1. Log into the admin interface of the NetGear router and go to Management->Settings Backup.
2. Click “Back Up” and save the backed up settings file to your local hard drive.
3. Under “Restore saved settings from file” click “Choose file…” and select the file you just downloaded.
4. Click Restore.

If you’re having the same problem I was, your VPN will now work.

I had a very interesting conversation with our product manager yesterday centred around ensuring the development team is always working on the highest value functionality for our clients (and thus the business). One of the key messages I took away from the conversation was the distinction between estimates of cost for development work and estimates of value.

So what exactly do I mean by cost and value? Cost is generally fairly simple to measure, it’s the amount you have to pay for the resources used to actually develop a function. That includes developer time, office space for those developers, hardware and software etc etc. There’s a lot involved to get an accurate assessment of the cost but it’s fairly straight forward to measure it as you spend. Just ask accounts for the figures. It only gets difficult once you start taking into account opportunity cost which is generally ignored anyway.

You know if there’s one thing more confusing and less documented in programming than character set encodings, it has to be time zones. For example, here’s the package tracking log for my new monitor:

      <td>
        <strong>Time</strong>
      </td>
      
      <td>
        <strong>Location</strong>
      </td>
      
      <td>
        <strong>Event Details</strong>
      </td>
    </tr>
    
    <tr>
      <td nowrap="nowrap">
        Feb 26, 2008
      </td>
      
      <td nowrap="nowrap">
        04:29:00 AM
      </td>
      
      <td nowrap="nowrap">
        SWANSEA WEST GLAM GB
      </td>
      
      <td nowrap="nowrap">
        Arrival Scan
      </td>
    </tr>
    
    <tr>
      <td nowrap="nowrap">
        Feb 25, 2008
      </td>
      
      <td nowrap="nowrap">
        10:30:00 PM
      </td>
      
      <td nowrap="nowrap">
        SWANSEA WEST GLAM GB
      </td>
      
      <td nowrap="nowrap">
        Departure Scan
      </td>
    </tr>
  </table>
</td>