nginx, rails and ubuntu - 502 bad gateway

We were getting tons of 502 errors under load, but then I stumbled across a posting in a news group.

cat /proc/sys/net/core/somaxconn

Will show you how many connections you can have. This should be 1024, because Phusion Passenger is hard coded for this value. Mine was 128!

Do this:

sudo sysctl -w net.core.somaxconn=1024

And then restart nginx.

get-flash-videos and osx (off topic)

I travel a lot and want to watch tv shows from the UK while I am doing so. There is get_iplayer that works nicely to get BBC shows, but on occasion there are things on itv (!) I watch. For this I use 'get-flash-videos'.

I had a tough time getting this working on OSX, so in a nutshell here is what I did:

1) Get 'get-flash-videos'
The home page is here. You pull that down somewhere onto your mac (I have it in a Downloads/get-flash-videos directory).

2) Update perl!
This is the key! I know nothing about perl, but here is what I did:

perl -MCPAN -e shell

This brings up a perl shell. Perl seems to have a package manager called CPAN. You will need to update this:

install Bundle::CPAN

Then you need to install Digest:SHA and Compress:ZLIB:

install Digest::SHA1
install Compress::Zlib

3) FLVStreamer and rtmpdump
Installing FLVStreamer is non-trivial and there are other guides about that.
I download rtmpdump from here
Then

chmod +x rtmpdump
sudo cp rtmpdump /usr/local/bin/.

4) Grab video!
now you should be able to do something like:

./get_flash_videos http://www.itv.com/itvplayer/video/?Filter=228293

And have it pull down an mp4. I use Handbrake to convert it for playing on the ipad.

Toygaroo on Rails

Seeing as there has been a lot of press recently about my company Toygaroo I thought I might throw out some tech info for those of you who care.
For those who don't know, Toygaroo is America's biggest toy rental company (think netflix for kid's toys). Recently - March 25th, 2011 - we appeared on the season premiere of Shark Tank - a national television show on ABC. About 4.6 million people watched the show.

The Platform
Toygaroo is a Ruby on Rails 3 application. It is based heavily on the code we wrote for FilmAmora.com, Spain's leading DVD rental company. We are running on Ubuntu 10.04 LTS. We're using Passenger 3 and Nginx (doesn't everyone?!). We're using Dalli in front of MemCached (though we've had some issues with this that I really should blog about!).

The Host
Right now we are running on Amazon EC2 service - though with Mark Cuban coming on board that might change. I am a big fan of EC2, though I think the machines are a little underpowered for what you pay.

The numbers!
In the 2 hours after the show aired we received around 70,000 page views. The basic architecture is a load balancer sitting in front of a whack of app servers. We are not using a scaling solution right now - hey, we're a start up! - so we wind up more servers if we feel we need them. It is a pretty simple process - I have a script for us to follow to get an Ubuntu server up and running in no time.

Caching
I looked into other solutions, like Varnish, but decided that Rails could handle the job with a combination of page, action and fragment caching. And I haven't been wrong so far. Even under heavy load we are getting great response time. The key - as I found out with FilmAmora - is what level to cache on. We cache 'blocks'. i.e. if you look at an index page with lots of toys we cache each toy block. That block can appear on many different pages, so it is a nice solution I find.

As time goes on I'd like to post more about how Toygaroo is coded and running as I think it will provide a nice real world example for what you can do with Rails 3. If you have any questions drop me a line (comment on here).

Background File Processing Daemon in Ruby

I am writing this up because I scoured the net and could not find what I would have thought would have been a common thing to do.
We have an application that needs to watch several directories (on the server) and parse files that are placed there (via scp) by a third party. FWIW, these files represent sports betting prices.

Requirements

A background task that could be
* monitored
* run forever!
* process files instantly - parse them into ruby objects and store them into our database for use by the rails app

Our Rails app is written in 2.3.x (its been running for a while) and uses Bundler.

The Solution

After some poking around I decided to use a combination of the Daemons gem, EventMachine and FSSM.

The Daemon

This was inspired heavily by a posting on StackOverflow.

1) Install what you need
I tried to get this working successfully with Bundler, but it was a no go. So I needed to install daemons and eventmachine 'normally':
sudo gem install daemons eventmachine fssm

2) Setup the Daemon:

Setup

Usual stuff for a ruby file:

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'

We have multiple directories that need watching. So have an array:

watch = [
  "/Users/smyp/development/wl/xtf/horse", 
  "/Users/smyp/development/wl/xtf/sport",
  "/Users/smyp/development/wl/xtf/live",
  "/Users/smyp/development/wl/xtf/alpha"
]

if ENV['RAILS_ENV'] == 'production'
  watch = ["/home/mcdata/horse", "/home/mcdata/sport", "/home/mcdata/live", "/home/mcdata/alpha"]
end

We launch a separate daemon for each directory as we don't want a huge file in the horses directory to slow down processing in the live directory.

Daemon Config

With the daemons gem you can set things like what the process will be called. And where the pid file will reside, etc etc.

dir = File.expand_path(File.join(File.dirname(__FILE__), '..'))

daemon_options = {
  :app_name   => "xturf_file_monitor",
  :multiple   => false,
  :dir_mode   => :normal,
  :dir        => File.join(dir, 'tmp', 'pids'),
  :backtrace  => true
}

3) The Actual Daemon

Cue spooky music!

class PriceDaemon
  attr_accessor :base_dir
  def initialize(base_dir)
    self.base_dir = base_dir
  end

  def dostuff
    logger.info "About to start job for #{base_dir}"
    EventMachine::run {
      # Your code here
      xhj = PriceFileJob.new(base_dir)
      xhj.clear_backlog
      FSSM.monitor(base_dir) do
        create {|base, relative| xhj.clear_backlog}
        update {|base, relative| xhj.clear_backlog}
      end
    }
  end

  def logger
    @@logger ||= ActiveSupport::BufferedLogger.new("#{RAILS_ROOT}/log/price_file_monitor.log")
  end
end

What this does is:
a) create a class that takes the directory to watch as an initialize parameter
b) do an EventMachine run that first clears out any backlog files then fire up an FSSM monitor. The FSSM monitor gives us events on create, update (and delete, but we don't care about that). As a safety measure I simply trawl through the entire directory every time a file is created or updated. This ensures that anything we missed will get caught.
We delete files ourselves after processing, so the directory should only have a few files in it anyway.

4) Spawn the Daemon

Bring on Mia Farrow!

watch.each_with_index do |base_dir, i|
  Daemons.run_proc("price_daemon_#{i}", daemon_options) do
    Dir.chdir dir
    PriceDaemon.new(base_dir).dostuff
  end
end

This will go through our array and file up a daemon for each directory. There are downsides to doing it this way - its not so easy to start and stop one (but then they shouldn't ever die, so if they do we just start and stop them all).

5) The File Processor

This of course will be specific to your operation, but, here's an outline of ours:

class PriceFileJob
  attr_accessor :base_dir
  def initialize(base_dir)
    self.base_dir = base_dir
    logger.info "watching #{base_dir}"
  end

  def logger
    @@logger ||= Logger.new("#{RAILS_ROOT}/log/price_file_job_#{base_dir.split("/").last}.log", "daily")
  end

  def clear_backlog
    files = Dir.new(base_dir).entries.sort_by{|c| File.stat(File.join(base_dir, c)).ctime}
    files.each do |file|
      process_file(file)
    end
  end
  
  def process_file(file)
  end
  
  private
end

6. Capistrano

We use Capistrano to deploy, so I included some tasks in our deploy.rb

before "mc:release", "file_processors:stop"
after "mc:release", "file_processors:start"

namespace :file_processors do
  desc "start processors"
  task :start, :roles => :db do
    run "cd #{current_path}; RAILS_ENV=#{fetch :rails_env} ruby ./script/price_file_monitor.rb start"
  end

  desc "get status of processors"
  task :status, :roles => :db do
    run "cd #{current_path}; RAILS_ENV=#{fetch :rails_env} ruby ./script/price_file_monitor.rb status"
  end

  desc "stop processors"
  task :stop, :roles => :db do
    run "cd #{current_path}; RAILS_ENV=#{fetch :rails_env} ruby ./script/price_file_monitor.rb stop"
  end
end

That's it! I hope you found this interesting.

I should also write up how we monitor these processes... maybe next time!

will_paginate and ajax in rails 3

After googling and stackoverflowing around I couldn't find anything that explicitly said how do to this.

I have a page that has 'pagable' areas on it, and want to load these using ajax. And I am in rails 3!

I found it actually was quite easy and only needed a few lines of code.

1 - In the page view (users/show.html.erb)
To enable the data-remote attribute on will_paginate's links

<div id="queue"><%= render :partial => 'queue' %></div>
<script>
$(document).ready(function() {
    $('.pagination a').attr('data-remote', 'true');
});</script>

2 - In the controller (users_controller.rb)
To allow a js response

def show
  @user = User.find(params[:id])
  @queue_items = @user.queue_items.with_state(:pending).paginate(:page => params[:queue_page] || 1, :per_page => 1)
  respond_to do |format|
    format.html 
    format.js
  end
end

3 - A show.js.erb template
You need the second line to reapply the data-remote to the new links

$('#queue').html('<%=escape_javascript render :partial => "queue" %>');
$('.pagination a').attr('data-remote', 'true');

That's it! Your new dataset should load into the 'queue' div.

Customizing Devise to a pseudo multi-stage Signup

The Requirements!
On our new site we want to have a kind of multi-stage signup. The first page the user chooses a type of subscription, the second page they create their account, with address information, and the third page they enter their billing info. If they don't do the third stage that's ok as we will warn them that their account is incomplete (and they won't get any services until they give us the billing info)

We are doing this new site in Rails3 and I thought I'd use Devise as the authentication engine. In our old site we use Restful Authentication, which has been awesome. But Devise seems to be the thing all the kids are into today!

So here's how we did it:

Step 1:
Create a New Controller
We created a new controller to handle the first stage of the signup. It's very simple. It only is a "new" method (for now) as we do no saving or updating of the subscription.

rails g controller subscriptions new

      create  app/controllers/subscriptions_controller.rb
       route  get "subscriptions/new"
      invoke  erb
      create    app/views/subscriptions
      create    app/views/subscriptions/new.html.erb
      invoke  rspec
      create    spec/controllers/subscriptions_controller_spec.rb
      create    spec/views/subscriptions
      create    spec/views/subscriptions/new.html.erb_spec.rb
      invoke  helper
      create    app/helpers/subscriptions_helper.rb
      invoke    rspec
      create      spec/helpers/subscriptions_helper_spec.rb

The controller looks like this:

class SubscriptionsController < Devise::RegistrationsController
  def new
    @subscription_plans = SubscriptionPlan.visible
  end
end

Pretty basic stuff!

The view is:

<% title "Sign up" %>

<div>Pick a subscription</div>
<% semantic_form_for :subscription, :url => users_sign_up2_path, :html => { :method => :get }  do |form| %>
  <%= form.input :plan_id, :as => :radio, :collection => @subscription_plans.map{|sp| ["#{sp.description}", sp.id]} %>
  <% form.buttons do %>
    <%= form.commit_button "Continue" %>
  <% end %>
<% end %>

Step 2:
Routes!
We needed to add some custom routes to the users block to let Devise know what was going on:

  devise_for :users do
    get "/users/sign_up" => "subscriptions#new"
    get "/users/sign_up2" => "registrations#new"
  end

The first line says use the subscriptions controller as the first stage of the sign up. That way we don't need to change any helpers and can do things in the standard Devise way. The second adds the second stage - the actual user new-ing and creation:

Step 3:
Override the Registrations Controller.
This is where 'the good stuff' happens! Note that we don't need to override the create method. Devise has a hook in it for building the model, so we override that instead.
It should be noted that this only works if the model can't be saved - i.e. your validations are complete. In our case we are creating 3 models in 1: subscription, address and user. The subscription and address will get saved if the user gets saved. So, we have validations in the user requiring a subscription and address. Then validations on the address to make sure it is good. That way if the validation of the address fails the validation of the user fails and we get chucked back out the the new user form.
The code might say it better!

class RegistrationsController < Devise::RegistrationsController
  def new
    @address = Address.new
    begin
      @subscription_plan = SubscriptionPlan.find(params[:subscription][:plan_id])
    rescue Exception => e
      redirect_to users_sign_up_path and return
    end
    @user = User.new
  end
  
  protected

    # Build a devise resource passing in the session. Useful to move
    # temporary session data to the newly created user.
    def build_resource(hash=nil)
      address_info = params[:user].delete(:address) rescue {}
      begin
        sub_info = params[:user].delete(:subscription_plan)
        @subscription_plan = SubscriptionPlan.find sub_info["id"]
      rescue Exception => e
        redirect_to users_sign_up_path and return
      end
      
      subscription = Subscription.new(:subscription_plan_id => @subscription_plan.id)
      
      @address = Address.new(address_info.merge(:country => "US"))
      
      @user = User.new(params[:user].merge(:first_name => @address.first_name, :last_name => @address.last_name))
      @user.subscription = subscription
      @user.current_shipping_address = @address
    end

    def after_sign_up_path_for(resource)
      new_user_billing_detail_path(resource)
    end
  
end

Also note that we override 'after_sign_up_path_for'. That way we move the user onto the third stage of the process - the billing info - if they sign up.

This is still in early stages, and not yet live, but the process seems to work.

testing Iridium gateway and active merchant with Rspec

Recently I rewrote the billing section of a website. The site uses Iridium as the gateway.
I thought people might be interested in seeing how I spec'd this:

      context "iridium" do
        before do
          BillingDetail.gateway = 
            ActiveMerchant::Billing::IridiumGateway.new(
                       :login => "Casdasdasd",
                       :password => "asdasdasd",
                       :enable_3d_secure => true)
        end
        
        it "should work" do
          bd = Factory.build(:unsaved_billing_detail)

          bd.should_receive(:ready!)

          credit_card = Factory.build(:iridium_good_no_3ds)
          address = Factory.build(:iridium_good_no_3ds_address)
          bd.address = address

          bd.authorize(100, credit_card)
        end

        it "should fail" do
          bd = Factory.build(:unsaved_billing_detail)

          bd.should_receive(:fail!)

          credit_card = Factory.build(:iridium_card_declined)
          address = Factory.build(:iridium_card_declined_address)
          bd.address = address

          lambda {
            bd.authorize(100, credit_card)
            }.should raise_error(TGR::GatewayError)
        end
      end

I use FactoryGirl, so I set up some factories:

Factory.define :iridium_good_no_3ds, :class => ActiveMerchant::Billing::CreditCard do |cc|
  cc.number "4976000000003436"
  cc.last_name "Watson"
  cc.first_name "John"
  cc.verification_value "452"
  cc.month "12"
  cc.year "2012"
end

Factory.define :iridium_good_no_3ds_address, :class => Address do |address|
  address.last_name "Watson"
  address.first_name "John"
  address.address_1 "32 Edward Street"
  address.city "Camborne"
  address.state "Cornwall"
  address.zipcode "TR14 8PA"
  address.country "GB"
end


Factory.define :iridium_card_declined, :class => ActiveMerchant::Billing::CreditCard do |cc|
  cc.number "4921810000009076"
  cc.last_name "Lewis"
  cc.first_name "Jack"
  cc.verification_value "875"
  cc.month "12"
  cc.year "2012"
end

Factory.define :iridium_card_declined_address, :class => Address do |address|
  address.last_name "Lewis"
  address.first_name "Jack"
  address.address_1 "4 Wing Road"
  address.city "Leighton Buzzard"
  address.state "Bedfordshire"
  address.zipcode "LU7 0JB"
  address.country "GB"
end

This seems to work a treat. I can do all my other tests using the bogus gateway, but to make sure I am actually providing the right info for Iridium this context, with its before block to set the right gateway, does the job nicely!