How To Round-Trip Data Via SSH

I keep forgetting this so for the record, you can use SSH to round trip data out to a remote server and back to your own box. This is most useful for adding latency and bandwidth limitations to a connection when doing website testing. The trick is to use both local and remote port forwards:

ssh -L 9090:localhost:9091 -R 9091:localhost:80 remote.server.com

The -L argument starts listening on port 9090 locally and sends all that traffic to port 9091 on the remote server (the domain name, localhost, is resolved by the remote server so it refers to the remote server, not your local machine). Then the -R argument listens on port 9091 of the remote server and forwards all that traffic back to your machine’s port 80 (here localhost is resolved on your local machine).

You don’t have to use localhost as the domain name. For example, if the site you want to test is deployed on your intranet at testmachine.intranet which remote.server.com doesn’t have access to, you could use:

ssh -L 9090:localhost:9091 -R 9091:testmachine.intranet:80 remote.server.com

Or if the test site is publicly available you can do it all without the -R argument:

ssh -L 9090:testmachine.com:80 remote.server.com

In all these cases, you connect to localhost:9090 to utilise the tunnel.

Development Mode – Concatenating Scripts and CSS

HTML 5 Boilerplate reminded me of an old-school tool which is extremely useful for concatenating JS and CSS files on the fly – server side includes:

<FilesMatch ".*\.combined\.(js|css)$">
  Options +Includes
  SetOutputFilter INCLUDES
</FilesMatch>

Then you have a main scripts.combined.js (or css) which contains:

<!--#include file="libs/backbone-min.js" -->
<!--#include file="libs/underscore-min.js" -->
<!--#include file="libs/jquery-1.7.1.min.js" -->

Plus any other files you need, in the order you specify. This works great for development mode so you can change a file on the fly and just refresh the browser without running any kind of build script. When it comes time to push to production, it’s easy for a build script to process the file ahead of time and be confident that you’ll get exactly the same result.

Cross Pairing

This evening I stumbled across two interesting posts about alternate layouts for pairing rather than sitting side by side. Firstly Tom Dale talking about Tilde’s corner desk pairing setup (and some of the software setup they use) and that was inspired by Josh Susser’s face to face pairing setup at Pivotal Labs.

Both approaches require more floor space which makes them difficult to setup but I would expect the face to face pairing to be a heck of a lot better if done well. I’ve always preferred having separate screens in mirror configuration as well as separate keyboards and mice to allow the developers to sit a little further apart to be comfortable and to be able to look straight ahead at the screen. That said, I quite like having a second monitor for spreading out windows as we have at LMAX so it’s not clear cut which is better.

It’s also interesting to note the popularity of the flat screen iMacs as opposed to Mac Pros or laptops. The former being too expensive for extra power and extensibility that generally isn’t required and the latter often being a bit too individualised to be good as pairing machines. Plus laptops, while amazingly powerful these days still have less bang for the buck and the reduction in performance is just enough to matter for development.

Bottlenecks in Programmer Productivity

Yan Pritzker recently posted “5 ways to get insane productivity boosts for coders” which provides some good tips on how to improve your usage of tools when doing technical work. To summarise:

Never look when you can search
Don’t repeat yourself (by setting up shortcuts and command line aliases)
Learn a scripting language
Learn an editor (and use that editor everywhere)
Learn regular expressions

However, nearly all of these tips really boil down to how to be more productive at writing text and the mechanics of writing code – editing actual source code files, jumping to different places in those files, executing commands more efficiently etc. Are these really the tasks that consume the vast majority of a developers time?

While often it feels like they are, after all we spend all day working with a text editor or the command line, are we really spending all that time struggling to keep up with the stream of code our brains are trying to output? Personally, I spend a lot more time thinking about what the best algorithm is, what direction we should be pushing the design in and how to do that and so on. Rarely do I need to type at full speed for extended periods of time.

Given that, I suspect the real bottlenecks in developer productivity are more along the lines of:

Comprehending an area of code and its design
Identifying, evaluating and selecting potential solutions to a problem (be that choice of algorithm, design choices or choice of classes/libraries etc)
Understanding and sharing a design vision within the team
Understanding and sharing requirements within the team

This isn’t an exhaustive list but it should give an idea for the types of things that really limit programmer productivity. In other words, someone who types with two fingers can be much more productive than someone who knows every shortcut in their editor if they are able to identify a simpler solution to the problem.

For example, recently LMAX needed to make a large and potentially quite risky change to how our system worked. The change affected a core concept in the system and so could have knock-on effects to a wide range of components and be very hard to test effectively. At the start of our two week iteration we already had a potential solution planned out but the developers were uncomfortable with the amount of risk involved with that plan. So we spent a full week discussing options, exploring the code and experimenting, plus talking with business experts to get a better understanding of the domain model. All that discussion led us to find a much simpler solution which, if something was missed, would be sure to cause failures in our existing acceptance tests. Even counting that initial week of discussion, finding the right solution meant we could deliver the change in about half the time we had expected.

So by all means, spend time learning to use your tools better, it avoids a lot of tedious boring work, but if you really want a step change improvement in productivity, work to improve you communication, design and thinking skills.

Simple Long Poll Patterns in JavaScript

A couple of little patterns for writing long-poll based JavaScript applications have evolved in some internal apps I’ve been working on lately so I thought I’d document them for posterity.

Simple Long Poll Loop

(function($) {
    var longPollUrl = '/longpoll';

    var lastReceivedSequence;

    function poll() {
        $.ajax(longPollUrl, {
            data: { 'lastSequence': lastReceivedSequence },
            dataType: 'json',
            timeout: 15000,
            success: function(data) {
                if (!data) return;
                lastReceivedSequence = data.sequenceNumber;
                $(window).trigger('onLongPollEvent', data);
            },
            complete: poll
        });
    }
})(jQuery);
 view raw basic.js This Gist brought to you by GitHub.

There’s very little special about this – it basically just uses jQuery to do a standard long poll. The pattern that I like is that instead of calling a particular function directly to handle events it just uses the jQuery trigger function to fire an event that can be handled by any interested code, without introducing tight coupling between the two bits of code. Listeners would receive events with:

$(window).bind('onLongPollEvent', function(eve, longPollData) { ... });
 view raw listener.js This Gist brought to you by GitHub.

Automatically Reload JavaScript Changes

One of the internal tools I’ve been working on is our “big feedback” page for reporting build results. This page is displayed on a large monitor the whole team can see, so refreshing the page to load JavaScript changes requires actually getting up and walking over to it. Fortunately, there’s a simple pattern to get the JavaScript to reload itself:

var serverVersion;
...
if (data.serverVersion != serverVersion) {
    window.location.reload();
    return;
}
...
 view raw reload.js This Gist brought to you by GitHub.

Each long poll event sent down to the browser includes the server version – when it changes, the JavaScript reloads the page. There are two nice little details here:

Deploying a new version actually requires some downtime while the server restarts. Naive implementations will reload the page when the long poll request returns an error but that usually results in the page attempting to reload and also getting an error or no response from the server and now there’s no JavaScript running to recover. Instead, this will wait for the server to come back up and be answering requests again before reloading.
So I don’t have to remember to update the server version, the server actually sends the UNIX time stamp from when it started up. Anytime the server restarts, that value changes and the JavaScript will reload. This can lead to some false positives but it’s extremely simple and effective. Another option would be for the build scripts to insert the SVN revision but that doesn’t work as well for development.

Usually I insert this reload code directly into the long poll loop so that it is guaranteed to run first and avoid any potential problems with data formats changing. Even the serverVersion data format can change since it doesn’t try to parse it in any way – any change, including data.serverVersion no longer being present, will cause the page to reload.

Put together the code comes out something like:

(function($) {
    var longPollUrl = '/longpoll';

    var lastReceivedSequence;
    var serverVersion;

    function poll() {
        $.ajax(longPollUrl, {
            data: { 'lastSequence': lastReceivedSequence },
            dataType: 'json',
            timeout: 15000,
            success: function(data) {
                if (!data) return;
                lastReceivedSequence = data.sequenceNumber;
                if (data.serverVersion != serverVersion) {
                    window.location.reload();
                    return;
                }
                $(window).trigger('onLongPollEvent', data);
            },
            complete: poll
        });
    }
})(jQuery);
 view raw full.js This Gist brought to you by GitHub.

Nothing earth-shatteringly new, but handy little tools to have up your sleeve.

Symphonious

Living in a state of accord.

Category Archives: Code and Geek Stuff