Those familiar with EC2 won't be surprised to hear that we won't be going with the service for three reasons:

1. It's at least as expensive as the dedicated server we'd need.
2. The filesystem gets reset everytime the server reboots (S3 provides a REST API to store and retrieve data, not a filesystem)
3. The server gets a new IP address every time it reboots.

Since then Amazon have rolled out new services that solve problems 2 and 3 and reserved instances to help with 1. What surprises me after a couple of years running a single EC2 instance with an app that’s using S3 for storage though is just how stable it has been.

Remember that EC2’s original point in life was scalability, not running one single instance for a really long time. They’ve done tons of upgrades to their infrastructure over the last couple of years as well which normally would mean down time and migrations. You can imagine my surprise when I checked how long it’s been since the instance rebooted:

e2wiki:~# uptime
 04:28:45 up 499 days, 19:14,  1 user,  load average: 0.33, 0.10, 0.03

Just shy of 500 days since a reboot for any reason. I can’t say that about any other hosting service I’ve ever used so even if EC2 is more expensive, it’s seriously reliable.

Now we just need to fix the memory leak in the app we’re running on that server – it up and dies a couple of times a week. That said, the script that automatically restarts it is so effective that the external monitoring tools don’t even notice so it’s probably not worth the effort.

The latest in my deployment-caused frustrations is CouchDB. I have a few use cases that I think CouchDB would be perfect for and it would save me heaps of development effort and headaches. The trouble is, while CouchDB may be of the web, it really isn’t of the enterprise IT architecture.

That’s not to say it wouldn’t fit in perfectly well. It’s not to say there’s anything wrong with CouchDB. It doesn’t even mean that it would be hard to deploy or hard to maintain or anything to do with CouchDB at all. It’s just not part of the enterprise plan for “stuff we put on our servers”. Database stuff goes in Oracle or DB2 and we really don’t need a second database instance running. The fact that CouchDB is an entirely different type of database and has completely different strengths and weaknesses making it perfect for this particular use case doesn’t get a hearing.

When you have big enterprise as your clients, the cool toys always seem just out of reach1.

I’d wish that I worked in an environment that was more relaxed and we could deploy tons of different systems based on what was the best fit for this particular job, except that I have that problem within Ephox and it’s not so much fun either.

1 – like Java versions above 1.4…↩

If you’re returning the page as requested, use 200.

If you’re returning a server error, use 500.

If the requested page doesn’t exist, use 404.

If you can follow even just those three rules, you’ll make life so much easier for user agents. Anything about that is great too, but whatever you do, please get these basics right.

Epic fail to IIS and IBM Portal on these points.  Something is seriously wrong with the internet…

My latest use for it was to add name based virtual host support to two completely separate virtual machines.  One machine runs IBM WCM and the other runs Quickr. Both use the same port, and in the future there will be more VMs with different versions as well, so while it would be possible to assign different port numbers, I’d prefer to not have to remember which VM is using which port etc.  The firewall however can only forward connections on a given port to one VM.

The solution then is to forward the traffic to Apache and configure mod_proxy within name based virtual hosts. Effectively Apache is acting as an intelligent HTTP router and I can add any number of VMs by adding a new CNAME and virtual host entry.

Exporting and Importing a Web Content Library

It should let you move web content (minus drafts and previous versions unfortunately) from one IWWCM server to another.

1. The scp problem is almost certainly caused by using ant 1.7.0 or below and jsch 0.1.30 or above. You could upgrade to the latest nightly of ant1 but it's probably easier to just drop back to jsch 0.1.29 which is what ant was developed against and works nicely. Bug 41090 contains the gory details.
2. If the command you're executing with the ssh task starts a background service or otherwise leaves a process running, that may be the cause of the problem. You can add 'shopt -s huponexit' to your /etc/profile, .bashrc or somewhere like that. I must admit, I'm somewhat vague on the exact details of what that does but the basic idea seems to be to signal any background processes that bash is exiting and then not wait for them to complete (which allows your ssh connection to close). If you're starting a server they'll probably ignore the hup signal it sends and if not, use the nohup command.

Hopefully that will be the last I'll see of that issue.

1 – which seems to mean compiling from source at the moment, since the nightly build directory the Ant website links to is empty↩

It usually starts, though it can take over half an hour to do so and on a couple of occasions it's just flat out sat there and done nothing for multiple hours on end. At startup it outputs the log message:

Aug 20, 2007 3:08:56 PM org.apache.coyote.http11.Http11AprProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080

and then nothing until all of a sudden 5-45 minutes later it suddenly comes back to life, finishes starting up and works perfectly. There's no CPU usage while it's out, it's just sitting there waiting for something to happen (network lookup??).

Sigh. I'm sure the world is out to get me….

I still wish Amazon had a preconfigured solution that acted as the web front end and load balancer with a static IP, but it appears that it's quite feasible to just point your DNS at EC2 and your server seems to stay put.

I've also done a bunch of development against S3 with some pretty fantastic results. With a little bit of simple caching it's actually feasible to run the server here in Australia and store the data on S3 without too much pain. Having very simple APIs is nice because it allows you to build a mock S3 quite easily to use for testing without having to jump through a lot of hoops.

Overall, I'm very impressed – building a web app entirely on S3 is not only feasible, it's fairly simple and can actually speed up development. There's even a couple of personal projects of mine that S3 may be a good fit for.

It is of course always possible to reindex the original documents, but that would take a fair while, so should you keep a backup of the search index? If you restored the backup, how would you identify which documents needed updating?

Solr seems to support replication – should you just use that as a constant backup that you can swap over to using if something goes wrong?

So many questions…

And it looks like there are answers out there somewhere – Solr has a bunch of tools related to backups and stuff. Looks promising, though the documentation doesn't really say what the best practices are at least there are tools that look like they help you manage the search indexes.

Those familiar with EC2 won't be surprised to hear that we won't be going with the service for three reasons:

1. It's at least as expensive as the dedicated server we'd need.
2. The filesystem gets reset everytime the server reboots (S3 provides a REST API to store and retrieve data, not a filesystem)
3. The server gets a new IP address every time it reboots.

The cost only applies to us because we don't need scalability – our needs are really quite consistent so we're not avoiding purchasing large amounts of redundant hardware. We also have the ability to just pay a hosting company to set up one dedicated server for us instead of setting up our own server farm. If you were offering software as a service however, Amazon's offering is likely to save you a lot of money.

The filesystem resetting is a challenge for deploying most existing web applications, but not for software designed to run with S3. For instance, it's pretty easy to imagine a wiki implementation that uses S3 as it's "database" for storing data directly (probably with some local caching etc). Wikis are somewhat ideal for this because search is about the only query you perform on the data – otherwise you just retrieve pages by name which S3 is perfectly suited for. The fact that so many wikis use flat files instead of databases is an indication that it'd work pretty well. There would be a few hurdles but nothing insurmountable.

The dynamic IP however is a real pain. There are examples of using dynamic DNS to work around it but the lag in DNS updates seems like a problem to me. The better solution would of course be to have a load balancer in front of your EC2 instances – the load balancer would have a static IP address and the EC2 instances would just register with it when they start up. Unfortunately this means you have to have a server outside of EC2 to do the load balancing which means another hosting provider to work with and it just seems odd to have the load balancing server in a different data center to the rest of the servers. If Amazon added an option to build an EC2 machine that could only ever have one instance but had a guaranteed IP address it would be the perfect solution.

It's certainly something interesting to play with – I'll have to chase up a corporate credit card and see if I can get access to do some experimentation some time.