Tomcat Startup Issues
I was so close to having everything working… EC2, S3, automatically pulling down the latest build and deploying it, Tomcat 5.5 with the native APR libraries, SSL support and using iptables to forward ports 80 and 443 directly over to Tomcat. Everything ready to go. Except Tomcat isn't so keen on starting.
It usually starts, though it can take over half an hour to do so and on a couple of occasions it's just flat out sat there and done nothing for multiple hours on end. At startup it outputs the log message:
Aug 20, 2007 3:08:56 PM org.apache.coyote.http11.Http11AprProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
and then nothing until all of a sudden 5-45 minutes later it suddenly comes back to life, finishes starting up and works perfectly. There's no CPU usage while it's out, it's just sitting there waiting for something to happen (network lookup??).
Sigh. I'm sure the world is out to get me….
August 21st, 2007 at 7:32 pm
Perhaps the kernel is running out of entropy and is desperately trying to generate it? See if /proc/sys/kernel/entropy_avail is more than 0. If it’s 0, your system is out of entropy.
In that case you might want to install rngd, at least in Gentoo it can be found in rng-tools package. rngd generates semi-real entropy by fetching numbers from /dev/urandom and feeding them to /dev/random. Another option is to switch Tomcat to use /dev/urandom instead of /dev/random. How that can be done, I have no idea and of course if you really need that uber-secure tamper-proof entropy /dev/random creates, then you just need to generate more real entropy…
And with my luck your problem isn’t about this issue at all. Well, at least I tried. :-)
August 21st, 2007 at 7:33 pm
… and the actual path for checking the entropy is /proc/sys/kernel/random/entropy_avail, and even that only under Linux …
August 21st, 2007 at 8:24 pm
Jaba, worth a look as best I can tell Tomcat *should* be using /dev/urandom but having APR in the mix perhaps it’s not. The other complication is that it’s the HTTP listener that takes forever and the HTTPS listener starts immediately. I’ll make sure rngd is installed though.
For the record, it’s a Debian Etch server.
August 21st, 2007 at 9:24 pm
Jaba,
You’re right, it was the SSL libraries not getting enough entropy. rng-tools failed to start the daemon on this particular Debian install but the Tomcat users list pointed me to http://marc.info/?l=tomcat-user&m=118209169008472&w=2 which lets you generate a random seed with openssl. I wound up having to set the RANDFILE environment variable to get it to be found but it solved the problem.
Thanks for your help.
August 21st, 2007 at 9:54 pm
No problem!
August 21st, 2007 at 10:05 pm
You can actually go one step better - set RANDFILE to /dev/urandom and ignore the openssl step. Works like a charm.
November 16th, 2007 at 7:43 am
If you are using the Sun JDK 1.5.x (or 1.6 I think), you may also be encountering an issue where you think you’ve configured the JDK to use /dev/urandom, but it’s really not. For details on this see http://bugs.sun.com/view_bug.do?bug_id=6202721 . I’ve had success with the workaround of “-Djava.security.egd=file:/dev/./urandom” as a Java option. To clarify: I wasn’t using Tomcat’s SSL but rather direct application of the SecureRandom class.
March 20th, 2008 at 1:03 am
I’m really interested to know how you use iptables to forward connection on port 80 and 443 to tomcat
I’m looking for a solution to redirect port to another IP address (in EC2), but I couldn’t figure out how ?