Rostamizadeh.Blog

A place for me to write about interesting technology topics.

Cron Job on Ubuntu Taking Down Rails Application Almost Nightly

With the help of New Relic, I recently discovered one of my more taxed Ubuntu servers was having Rails downtime almost nightly averaging about six minutes each time. This was a bit puzzling because if you’ve read my other blog posts, you’ll know I’m big on monitoring processes and making sure nothing dies without a plan in place to bring it back up; I also test these plans to make sure they work, and setup email alerts for certain failures. In the end, I found a cron job as the culprit, and made a change to the process it was kicking off (update-apt-xapian-index) which fixed my issues with downtime.

Who Watches the Watchers: Using Upstart to Manage Monit Which Manages Unicorn

So I finally have a spiffy new web application deployed on my VPS, and I’m ready to flip the switch and share it with the world…but what happens when my server reboots? Or what happens when Unicorn shuts down and doesn’t restart? Or maybe Unicorn is consuming too many system resources and I need to kill workers or restart the master process? What should I do? How should I set all this up?

In short, my solution consists of using Upstart to manage Monit and ensure it is always running, while Monit is configured to manage my Unicorn worker and master processes. I installed Monit and set it up with Upstart over SSH to my server, while Monit service tests get pushed to my server when I do Capistrano deployments. We’ll dive into the details below…

Capistrano Deployment Ending in Bundler::GemfileNotFound

I recently made a gemfile change, committed and pushed my changes, and ran a Capistrano deployment…which exploded in an enormous ball of fire on the server. Here’s what I saw in my deployment output:

1
2
3
4
** [out :: IP] Original PID:  21444
** [out :: IP] USR2 sent; Waiting for .oldbin
** [out :: IP] .
** [out :: IP] .

… “Throw more dots, more dots, more dots, come on more dots. Ok, stop dots.” (Congrats if you get that reference.) …

1
2
3
4
5
** [out :: IP] .
** [out :: IP] .
** [out :: IP] Waiting for new pid file
** [out :: IP] 
** [out :: IP] New master failed to start; see error log

Following the sage recommendation to view the error log, shown above, I opened my Unicorn error log and was greeted with this:

1
2
3
INFO -- : executing ["/path/to/site/shared/bundle/ruby/1.9.1/bin/unicorn", "-D", "-c", "/path/to/site/shared/config/unicorn.rb", "-E", "staging", {7=>#<Kgio::UNIXServer:fd 7>}] (in /path/to/site/releases/20120521061632)
INFO -- : forked child re-executing...
/home/{user name}/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/bundler-1.1.4/lib/bundler/definition.rb:15:in `build': /path/to/site/releases/20120520044313/Gemfile not found (Bundler::GemfileNotFound)

Precompiling Assets Locally for Capistrano Deployment

I’ve had a goal of fast Capistrano deployments to my VPS for a while now, but I’ve constantly been plagued with asset precompilation taking anywhere from four to 15 minutes on my little server (I’m using Rackspace’s smallest offering, a VPS with 256MB RAM). When I precompile assets locally, it always finishes in under a minute…so I came up with an approach to leverage my local machine for precompilation and upload the assets to the server. I’ve also avoided any shenanigans with adding assets to my git repository (Ew! Don’t do that!).

To take my newly found performance boost one level further, I integrated local asset precompilation with Ben Curtis’ approach of skipping asset precompilation unless any assets have changed.

Wrangling Unicorn USR2 Signals and Capistrano Deployments

I recently had an issue with my Capistrano deployments on a Rackspace server (their smallest offering at 256MB RAM) where Unicorn would receive an upgrade signal (USR2), but a new master process wasn’t started. This ended up being a problem between the example Unicorn init script and the lack of horsepower on my VPS. Hopefully the information below will aid other people in their quests to get zero downtime (or close to it) deployments with Capistrano and Unicorn.