RoR Ramblings

installing sunport_rails on ubuntu production

2011-10-30T14:13:00.002+01:00

We've added search functionality to Toygaroo with Solr via sunspot_rails.

If you are going to install it on Ubuntu then make sure you do:

sudo apt-get install openjdk-6-jdk

First!

Using the html5 boilerplate and rails 3

2011-10-01T15:15:00.003+02:00

I am experimenting with using the html5 boilerplate with rails 3.0x. It is an interesting setup. I am using Russ Frisch's html5 boilerplate template, but updating it to use the latest boilerplate code.

It seems to be a pretty nice way of getting your css and js ducks in a row. Over the next few days I need to see how it handles having jquery mobile in the mix. I am also not 100% sure how to use a some sort of grid layout template (like 960 or 114opx) with it. Or even if you should!

We shall see.

Suffering Rails3 slowdown

2011-09-28T11:44:00.003+02:00

We've just pushed our Rails 3 upgraded app to production... and are suffering a massive slowdown in insert/update speed over Rails 2.

At the moment I am not sure of the exact cause of this.

It *might* be mysql inserts, though I can't quite see why that would be.

It *might* be because this new version we are using vestal_versions to track changes.

It might be because the moon is in the house of Mars for all I know!

I hate getting stung by unknowns. The speed on our test environment is tolerable, slightly slower than the rails 2 version, but I was willing to accept that because the new version is doing so much more.

Bench-marking is one thing.. but know why the bench marks are slower is the key!

Expiring fragments from daemons

2011-09-26T19:44:00.002+02:00

We have an application that gets its data from a series of daemons that go out and read in data. This works great, except, we are caching pages. And I'd like to expire those pages based on an update.
It turns out that an Observer doesn't have access to expire_action or fragment. And a Sweeper is not called from data-only (i.e non-controller) updates! Buggers!

But there is a solution. You can call the sweeper directly from your importer:

MySweeper.instance.clean_up(model_instance)

This works, except I couldn't get it reliably to expire the actions. So, I used direct calls to Rails.cache.delete to do this.

Thinking about it, I guess I could then have just written an observer! As those do get called from controller-less updates.

meta_search sort_link helper and associations

2011-07-21T10:45:00.005+02:00

It took me a while to find this, so, for my own memory I am going to quickly write this up.

I have a view that shows a table of objects (a pretty standard index view). The only issue was I want to sort on one of the columns that actually has data coming not from the main object but an association.
What I discovered is a line in a posting here that says:

You can define your own custom sort scopes. Define scopes named “sort_by__asc” and “sort_by__desc” and sort_link @search, :name will work as you might expect.

So, I have an object of 'info' defined like this:

class Info < ActiveRecord::Base
  belongs_to :region
  scope :contains_string, lambda {|str| where(:name.matches % "%#{str}%")}
  search_methods :contains_string

  scope :active, lambda{where(:active => true)}
  search_methods :active, :type => :boolean
end

What I found is that you can put this at the end:

scope :sort_by_region_name, lambda{joins(:region).order("regions.name asc")}
search_methods :sort_by_region_name

and then in my view I can do:

<table>
    <thead>
        <tr>
            <td width="25%"><%= sort_link @info_search, :name, "Info" %></td>
            <td width="25%"><%= sort_link @info_search, :region_name, "Region" %></td>
            <td><%= sort_link @info_search, :active %></td>
        </tr>
    </thead>
        ...body info...
</table

And in my controller:

class Admin::InfosController < Admin::BaseController
  def index
    search = params[:search] || {"meta_sort" => "name.asc"}

    @info_search = Info.search(search)
    @infos = @linfo_search.paginate(:page => params[:page]||1, :per_page => 15)   # load all matching records
  end
end

And presto I have an index that
a) default to sorting by name (see the first line of the index method)
b) let's me sort on an associated value

Cool!

I'm sorry, but POW sucks

2011-07-01T14:50:00.003+02:00

I have been (trying to) use Pow, and the Powder gem, for a few weeks now. At the outset it looked good: you get a local domain for you test on, you don't have to worry about deployments, etc etc.
But it sucks.
If you get an error in rendering (and we are using this in development, so of course this is going to happen) pow goes into a tail spin and on my MacPro it takes almost a minute to come out of it as it retries the request dozens of times.

So you end up with a huge log that looks pretty much like this:

  SQL (2.2ms)  SHOW TABLES
  SQL (1.2ms)  SHOW TABLES
  SQL (0.9ms)  SHOW TABLES
  SQL (1.2ms)  SHOW TABLES
  SQL (0.9ms)  SHOW TABLES
  SQL (1.2ms)  SHOW TABLES
  SQL (1.2ms)  SHOW TABLES
  SQL (1.2ms)  SHOW TABLES
  SQL (1.0ms)  SHOW TABLES
  SQL (1.3ms)  SHOW TABLES
  SQL (0.9ms)  SHOW TABLES
  SQL (1.2ms)  SHOW TABLES
  SQL (0.9ms)  SHOW TABLES
  SQL (1.4ms)  SHOW TABLES
  SQL (1.2ms)  SHOW TABLES
  SQL (1.4ms)  SHOW TABLES
  SQL (1.1ms)  SHOW TABLES
Rendered /Users/smyp/.rvm/gems/ree-1.8.7-2011.03@toygaroo_r3/gems/actionpack-3.0.3/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (8663.2ms)
Rendered /Users/smyp/.rvm/gems/ree-1.8.7-2011.03@toygaroo_r3/gems/actionpack-3.0.3/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (8728.0ms)
Rendered /Users/smyp/.rvm/gems/ree-1.8.7-2011.03@toygaroo_r3/gems/actionpack-3.0.3/lib/action_dispatch/middleware/templates/rescues/template_error.erb within rescues/layout (22759.6ms)
Rendered /Users/smyp/.rvm/gems/ree-1.8.7-2011.03@toygaroo_r3/gems/actionpack-3.0.3/lib/action_dispatch/middleware/templates/rescues/template_error.erb within rescues/layout (22758.7ms

I am going back to using Passenger standalone. Ok, you loose the pretty local domain, but it works much more reliably.

nginx, rails and ubuntu - 502 bad gateway

2011-05-02T15:10:00.002+02:00

We were getting tons of 502 errors under load, but then I stumbled across a posting in a news group.

cat /proc/sys/net/core/somaxconn

Will show you how many connections you can have. This should be 1024, because Phusion Passenger is hard coded for this value. Mine was 128!

Do this:

sudo sysctl -w net.core.somaxconn=1024

And then restart nginx.

get-flash-videos and osx (off topic)

2011-05-01T10:24:00.003+02:00

I travel a lot and want to watch tv shows from the UK while I am doing so. There is get_iplayer that works nicely to get BBC shows, but on occasion there are things on itv (!) I watch. For this I use 'get-flash-videos'.

I had a tough time getting this working on OSX, so in a nutshell here is what I did:

1) Get 'get-flash-videos'
The home page is here. You pull that down somewhere onto your mac (I have it in a Downloads/get-flash-videos directory).

2) Update perl!
This is the key! I know nothing about perl, but here is what I did:

perl -MCPAN -e shell

This brings up a perl shell. Perl seems to have a package manager called CPAN. You will need to update this:

install Bundle::CPAN

Then you need to install Digest:SHA and Compress:ZLIB:

install Digest::SHA1
install Compress::Zlib

3) FLVStreamer and rtmpdump
Installing FLVStreamer is non-trivial and there are other guides about that.
I download rtmpdump from here
Then

chmod +x rtmpdump
sudo cp rtmpdump /usr/local/bin/.

4) Grab video!
now you should be able to do something like:

./get_flash_videos http://www.itv.com/itvplayer/video/?Filter=228293

And have it pull down an mp4. I use Handbrake to convert it for playing on the ipad.

Toygaroo on Rails

2011-03-28T15:58:00.003+02:00

Seeing as there has been a lot of press recently about my company Toygaroo I thought I might throw out some tech info for those of you who care.
For those who don't know, Toygaroo is America's biggest toy rental company (think netflix for kid's toys). Recently - March 25th, 2011 - we appeared on the season premiere of Shark Tank - a national television show on ABC. About 4.6 million people watched the show.

The Platform
Toygaroo is a Ruby on Rails 3 application. It is based heavily on the code we wrote for FilmAmora.com, Spain's leading DVD rental company. We are running on Ubuntu 10.04 LTS. We're using Passenger 3 and Nginx (doesn't everyone?!). We're using Dalli in front of MemCached (though we've had some issues with this that I really should blog about!).

The Host
Right now we are running on Amazon EC2 service - though with Mark Cuban coming on board that might change. I am a big fan of EC2, though I think the machines are a little underpowered for what you pay.

The numbers!
In the 2 hours after the show aired we received around 70,000 page views. The basic architecture is a load balancer sitting in front of a whack of app servers. We are not using a scaling solution right now - hey, we're a start up! - so we wind up more servers if we feel we need them. It is a pretty simple process - I have a script for us to follow to get an Ubuntu server up and running in no time.

Caching
I looked into other solutions, like Varnish, but decided that Rails could handle the job with a combination of page, action and fragment caching. And I haven't been wrong so far. Even under heavy load we are getting great response time. The key - as I found out with FilmAmora - is what level to cache on. We cache 'blocks'. i.e. if you look at an index page with lots of toys we cache each toy block. That block can appear on many different pages, so it is a nice solution I find.

As time goes on I'd like to post more about how Toygaroo is coded and running as I think it will provide a nice real world example for what you can do with Rails 3. If you have any questions drop me a line (comment on here).

Background File Processing Daemon in Ruby

2011-02-12T11:08:00.011+01:00

I am writing this up because I scoured the net and could not find what I would have thought would have been a common thing to do.
We have an application that needs to watch several directories (on the server) and parse files that are placed there (via scp) by a third party. FWIW, these files represent sports betting prices.

Requirements

A background task that could be
* monitored
* run forever!
* process files instantly - parse them into ruby objects and store them into our database for use by the rails app

Our Rails app is written in 2.3.x (its been running for a while) and uses Bundler.

The Solution

After some poking around I decided to use a combination of the Daemons gem, EventMachine and FSSM.

The Daemon

This was inspired heavily by a posting on StackOverflow.

1) Install what you need
I tried to get this working successfully with Bundler, but it was a no go. So I needed to install daemons and eventmachine 'normally':
sudo gem install daemons eventmachine fssm

2) Setup the Daemon:

Setup

Usual stuff for a ruby file:

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'

We have multiple directories that need watching. So have an array:


watch = [
  "/Users/smyp/development/wl/xtf/horse", 
  "/Users/smyp/development/wl/xtf/sport",
  "/Users/smyp/development/wl/xtf/live",
  "/Users/smyp/development/wl/xtf/alpha"
]

if ENV['RAILS_ENV'] == 'production'
  watch = ["/home/mcdata/horse", "/home/mcdata/sport", "/home/mcdata/live", "/home/mcdata/alpha"]
end

We launch a separate daemon for each directory as we don't want a huge file in the horses directory to slow down processing in the live directory.

Daemon Config

With the daemons gem you can set things like what the process will be called. And where the pid file will reside, etc etc.

dir = File.expand_path(File.join(File.dirname(__FILE__), '..'))

daemon_options = {
  :app_name   => "xturf_file_monitor",
  :multiple   => false,
  :dir_mode   => :normal,
  :dir        => File.join(dir, 'tmp', 'pids'),
  :backtrace  => true
}

3) The Actual Daemon

Cue spooky music!

class PriceDaemon
  attr_accessor :base_dir
  def initialize(base_dir)
    self.base_dir = base_dir
  end

  def dostuff
    logger.info "About to start job for #{base_dir}"
    EventMachine::run {
      # Your code here
      xhj = PriceFileJob.new(base_dir)
      xhj.clear_backlog
      FSSM.monitor(base_dir) do
        create {|base, relative| xhj.clear_backlog}
        update {|base, relative| xhj.clear_backlog}
      end
    }
  end

  def logger
    @@logger ||= ActiveSupport::BufferedLogger.new("#{RAILS_ROOT}/log/price_file_monitor.log")
  end
end

What this does is:
a) create a class that takes the directory to watch as an initialize parameter
b) do an EventMachine run that first clears out any backlog files then fire up an FSSM monitor. The FSSM monitor gives us events on create, update (and delete, but we don't care about that). As a safety measure I simply trawl through the entire directory every time a file is created or updated. This ensures that anything we missed will get caught.
We delete files ourselves after processing, so the directory should only have a few files in it anyway.

4) Spawn the Daemon

Bring on Mia Farrow!

watch.each_with_index do |base_dir, i|
  Daemons.run_proc("price_daemon_#{i}", daemon_options) do
    Dir.chdir dir
    PriceDaemon.new(base_dir).dostuff
  end
end

This will go through our array and file up a daemon for each directory. There are downsides to doing it this way - its not so easy to start and stop one (but then they shouldn't ever die, so if they do we just start and stop them all).

5) The File Processor

This of course will be specific to your operation, but, here's an outline of ours:

class PriceFileJob
  attr_accessor :base_dir
  def initialize(base_dir)
    self.base_dir = base_dir
    logger.info "watching #{base_dir}"
  end

  def logger
    @@logger ||= Logger.new("#{RAILS_ROOT}/log/price_file_job_#{base_dir.split("/").last}.log", "daily")
  end

  def clear_backlog
    files = Dir.new(base_dir).entries.sort_by{|c| File.stat(File.join(base_dir, c)).ctime}
    files.each do |file|
      process_file(file)
    end
  end
  
  def process_file(file)
  end
  
  private
end

6. Capistrano

We use Capistrano to deploy, so I included some tasks in our deploy.rb

before "mc:release", "file_processors:stop"
after "mc:release", "file_processors:start"

namespace :file_processors do
  desc "start processors"
  task :start, :roles => :db do
    run "cd #{current_path}; RAILS_ENV=#{fetch :rails_env} ruby ./script/price_file_monitor.rb start"
  end

  desc "get status of processors"
  task :status, :roles => :db do
    run "cd #{current_path}; RAILS_ENV=#{fetch :rails_env} ruby ./script/price_file_monitor.rb status"
  end

  desc "stop processors"
  task :stop, :roles => :db do
    run "cd #{current_path}; RAILS_ENV=#{fetch :rails_env} ruby ./script/price_file_monitor.rb stop"
  end
end

That's it! I hope you found this interesting.

I should also write up how we monitor these processes... maybe next time!

will_paginate and ajax in rails 3

2010-11-24T10:34:00.004+01:00

After googling and stackoverflowing around I couldn't find anything that explicitly said how do to this.

I have a page that has 'pagable' areas on it, and want to load these using ajax. And I am in rails 3!

I found it actually was quite easy and only needed a few lines of code.

1 - In the page view (users/show.html.erb)
To enable the data-remote attribute on will_paginate's links

<div id="queue"><%= render :partial => 'queue' %></div>
<script>
$(document).ready(function() {
    $('.pagination a').attr('data-remote', 'true');
});</script>

2 - In the controller (users_controller.rb)
To allow a js response


def show
  @user = User.find(params[:id])
  @queue_items = @user.queue_items.with_state(:pending).paginate(:page => params[:queue_page] || 1, :per_page => 1)
  respond_to do |format|
    format.html 
    format.js
  end
end

3 - A show.js.erb template
You need the second line to reapply the data-remote to the new links

$('#queue').html('<%=escape_javascript render :partial => "queue" %>');
$('.pagination a').attr('data-remote', 'true');

That's it! Your new dataset should load into the 'queue' div.

Customizing Devise to a pseudo multi-stage Signup

2010-10-18T10:46:00.005+02:00

The Requirements!
On our new site we want to have a kind of multi-stage signup. The first page the user chooses a type of subscription, the second page they create their account, with address information, and the third page they enter their billing info. If they don't do the third stage that's ok as we will warn them that their account is incomplete (and they won't get any services until they give us the billing info)

We are doing this new site in Rails3 and I thought I'd use Devise as the authentication engine. In our old site we use Restful Authentication, which has been awesome. But Devise seems to be the thing all the kids are into today!

So here's how we did it:

Step 1:
Create a New Controller
We created a new controller to handle the first stage of the signup. It's very simple. It only is a "new" method (for now) as we do no saving or updating of the subscription.

rails g controller subscriptions new

      create  app/controllers/subscriptions_controller.rb
       route  get "subscriptions/new"
      invoke  erb
      create    app/views/subscriptions
      create    app/views/subscriptions/new.html.erb
      invoke  rspec
      create    spec/controllers/subscriptions_controller_spec.rb
      create    spec/views/subscriptions
      create    spec/views/subscriptions/new.html.erb_spec.rb
      invoke  helper
      create    app/helpers/subscriptions_helper.rb
      invoke    rspec
      create      spec/helpers/subscriptions_helper_spec.rb

The controller looks like this:

class SubscriptionsController < Devise::RegistrationsController
  def new
    @subscription_plans = SubscriptionPlan.visible
  end
end

Pretty basic stuff!

The view is:

<% title "Sign up" %>

<div>Pick a subscription</div>
<% semantic_form_for :subscription, :url => users_sign_up2_path, :html => { :method => :get }  do |form| %>
  <%= form.input :plan_id, :as => :radio, :collection => @subscription_plans.map{|sp| ["#{sp.description}", sp.id]} %>
  <% form.buttons do %>
    <%= form.commit_button "Continue" %>
  <% end %>
<% end %>

Step 2:
Routes!
We needed to add some custom routes to the users block to let Devise know what was going on:

  devise_for :users do
    get "/users/sign_up" => "subscriptions#new"
    get "/users/sign_up2" => "registrations#new"
  end

The first line says use the subscriptions controller as the first stage of the sign up. That way we don't need to change any helpers and can do things in the standard Devise way. The second adds the second stage - the actual user new-ing and creation:

Step 3:
Override the Registrations Controller.
This is where 'the good stuff' happens! Note that we don't need to override the create method. Devise has a hook in it for building the model, so we override that instead.
It should be noted that this only works if the model can't be saved - i.e. your validations are complete. In our case we are creating 3 models in 1: subscription, address and user. The subscription and address will get saved if the user gets saved. So, we have validations in the user requiring a subscription and address. Then validations on the address to make sure it is good. That way if the validation of the address fails the validation of the user fails and we get chucked back out the the new user form.
The code might say it better!

class RegistrationsController < Devise::RegistrationsController
  def new
    @address = Address.new
    begin
      @subscription_plan = SubscriptionPlan.find(params[:subscription][:plan_id])
    rescue Exception => e
      redirect_to users_sign_up_path and return
    end
    @user = User.new
  end
  
  protected

    # Build a devise resource passing in the session. Useful to move
    # temporary session data to the newly created user.
    def build_resource(hash=nil)
      address_info = params[:user].delete(:address) rescue {}
      begin
        sub_info = params[:user].delete(:subscription_plan)
        @subscription_plan = SubscriptionPlan.find sub_info["id"]
      rescue Exception => e
        redirect_to users_sign_up_path and return
      end
      
      subscription = Subscription.new(:subscription_plan_id => @subscription_plan.id)
      
      @address = Address.new(address_info.merge(:country => "US"))
      
      @user = User.new(params[:user].merge(:first_name => @address.first_name, :last_name => @address.last_name))
      @user.subscription = subscription
      @user.current_shipping_address = @address
    end

    def after_sign_up_path_for(resource)
      new_user_billing_detail_path(resource)
    end
  
end

Also note that we override 'after_sign_up_path_for'. That way we move the user onto the third stage of the process - the billing info - if they sign up.

This is still in early stages, and not yet live, but the process seems to work.

testing Iridium gateway and active merchant with Rspec

2010-10-14T12:26:00.003+02:00

Recently I rewrote the billing section of a website. The site uses Iridium as the gateway.
I thought people might be interested in seeing how I spec'd this:


      context "iridium" do
        before do
          BillingDetail.gateway = 
            ActiveMerchant::Billing::IridiumGateway.new(
                       :login => "Casdasdasd",
                       :password => "asdasdasd",
                       :enable_3d_secure => true)
        end
        
        it "should work" do
          bd = Factory.build(:unsaved_billing_detail)

          bd.should_receive(:ready!)

          credit_card = Factory.build(:iridium_good_no_3ds)
          address = Factory.build(:iridium_good_no_3ds_address)
          bd.address = address

          bd.authorize(100, credit_card)
        end

        it "should fail" do
          bd = Factory.build(:unsaved_billing_detail)

          bd.should_receive(:fail!)

          credit_card = Factory.build(:iridium_card_declined)
          address = Factory.build(:iridium_card_declined_address)
          bd.address = address

          lambda {
            bd.authorize(100, credit_card)
            }.should raise_error(TGR::GatewayError)
        end
      end

I use FactoryGirl, so I set up some factories:


Factory.define :iridium_good_no_3ds, :class => ActiveMerchant::Billing::CreditCard do |cc|
  cc.number "4976000000003436"
  cc.last_name "Watson"
  cc.first_name "John"
  cc.verification_value "452"
  cc.month "12"
  cc.year "2012"
end

Factory.define :iridium_good_no_3ds_address, :class => Address do |address|
  address.last_name "Watson"
  address.first_name "John"
  address.address_1 "32 Edward Street"
  address.city "Camborne"
  address.state "Cornwall"
  address.zipcode "TR14 8PA"
  address.country "GB"
end


Factory.define :iridium_card_declined, :class => ActiveMerchant::Billing::CreditCard do |cc|
  cc.number "4921810000009076"
  cc.last_name "Lewis"
  cc.first_name "Jack"
  cc.verification_value "875"
  cc.month "12"
  cc.year "2012"
end

Factory.define :iridium_card_declined_address, :class => Address do |address|
  address.last_name "Lewis"
  address.first_name "Jack"
  address.address_1 "4 Wing Road"
  address.city "Leighton Buzzard"
  address.state "Bedfordshire"
  address.zipcode "LU7 0JB"
  address.country "GB"
end

This seems to work a treat. I can do all my other tests using the bogus gateway, but to make sure I am actually providing the right info for Iridium this context, with its before block to set the right gateway, does the job nicely!

upgrading from ruby 1.8.6 to 1.8.7

2010-02-26T19:54:00.003+01:00

We recently updated filmamora.com to use the latest passenger (2.2.10) and Ruby Enterprise Edition 1.8.7. This was a move from 2.2.4 and 1.8.6 respectively. (We also upgraded to Nginx 0.7.4)

So far it seems to be a very worthwhile move! I had looked into using ruby 1.9, but ran into may issues with gems.

Here are some Munin charts.

Mysql... this I don't even understand! What a difference!

Memory - also seems to be a nice drop.

Individual Interupts

Interupts

The webapp itself was not changed at all. it is still using Rails 2.2.2.

SSL_Requirement

2009-12-20T10:22:00.002+01:00

I use the ssl_requirement plugin to let me specify what actions need to be secure.

But - I found it limiting. I needed to:

Specify what the secure domain was - we only have a certificate on the base domain, not all the subdomains

Be able to easily turn it off in different environments

So, I forked a version off that does that. If you have the same needs (and, like me, prefer a plugin over a gem) check mine out!

Apache + Passenger + SSL + OSX

2009-12-19T20:32:00.003+01:00

Recently I wanted to do some work on parameter passing and also how to keep parameters passed to SSL pages through redirects.
The first stumbling block was to get passenger and apache to play nice with SSL on OSX

I am running Leopard (not SNOW Leopard) and here is what I did to get it to work. I couldn't find anything specific on the net about this, so I thought I'd chuck this up here. There may be nicer, better ways of doing this.

1 - Create the cert

Apple has a page on creating a cert. This all worked fine except the locations aren't right. This page must be for Tiger.
For Leopard apache is in /etc/apache2 (or /private/etc/apache2 depending on your installation).
It all seemed to work fine for me as written apart from that.
You end up with a ssl.key directory in your apache2 directory. You may wish to rename this domain.ssl.key if you are doing multi domain development. I am, but this is the only domain I wanted to check ssl on.

2 - Apache and SSL

This turned out to be easier than expected, but everything's easy when you know how!
Edit /etc/apache2/httpd.conf
All I needed to do is put this line around line 40:

Listen 80
Listen 443

You add the Listen 443

3 - Passenger files

I did this by hand. As far as I know you can't do this via the prefpane,
I have used the PrefPane to create the vhosts file.
sudo vi passenger_pane_vhosts/mydomain.local.vhost.conf

Then add:

<VirtualHost *:443>
  ServerName mydomain.local
  ServerAlias mydomain.local es.mydomain.local en.mydomain.local
  DocumentRoot "/Users/smyp/development/mydomain/public"
  RailsEnv development
  <directory "/Users/smyp/development/mydomain/public">
    Order allow,deny
    Allow from all
  </directory>

# SSL Configuration
SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP
SSLOptions +FakeBasicAuth +ExportCertData +StdEnvVars +StrictRequire

#Self Signed certificates
    SSLCertificateFile /private/etc/apache2/ssl.key/server.crt
    SSLCertificateKeyFile /private/etc/apache2/ssl.key/server.key
    SSLCertificateChainFile /private/etc/apache2/ssl.key/ca.crt

SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0

</VirtualHost>

Basically what you do is copy all the stuff from the area and then add in the extra SSL config.
Just point the crt, key and ca.crt files to the ones you created in step 1 from the apple doc.

That's it! You should be ready to go!

Let me know if there are any errors in this and I'll correct them... I wasn't making notes as I went along, so this is done from looking back, so maybe I've left something out.

redirect_to : pass all parameters

2009-12-16T12:05:00.003+01:00

On filmamora we redirect people to the language version we think they want. This is done with a subdomain for the language.
As if that wasn't painful enough, it now seems I was stripping out additional parameters - like the parameters passed in from our mailings that triggered Google Analytics campaigns.
Doh!

So this:

  def language_redirect
    redirect_to :subdomain => session[:language] unless RAILS_ENV == 'staging'
  end

Needed to become this:

  def language_redirect
    p2 = params.merge({:subdomain => session[:language]})
    redirect_to p2 unless RAILS_ENV == 'staging'
  end

Such a small change - such a big difference! Marketing types appeased!

rotating nginx logs

2009-10-06T15:01:00.002+02:00

Since moving most of my systems over to plain Ubuntu on EC2 (as opposed to using ec2OnRails) I've needed to figure out certain things - like rotating the nginx logs (using Nginx instead of apache).

Simply create a file in /etc/logrotate.d/nginx that looks like this

/opt/nginx/logs/*.log {
  daily
  missingok
  rotate 30
  compress
  delaycompress
  sharedscripts
  postrotate
    /etc/init.d/nginx restart
  endscript
}

move from ec2onrails

2009-07-24T15:05:00.002+02:00

Recently the performance on FilmAmora.com has been plummeting. And I had no idea why. Often apache was timing out waiting for a mongrel. Very little development has happened recently on FilmAmora, so I was stumped.
So - like a good developer - I took this as a chance to make a drastic move from ec2onrails machines to straight ubuntu, with everything else installed by hand.

It has made a HUGE difference. So much so that I could actually drop a machine from the configuration and still have better response times.

I followed these instructions to get started: rails-nginx-passenger-ubuntu. I also switched to nginx and ruby enterprise edition - just to make following the instructions easier.

So far so good. It took a couple hours and presto a leaner meaner installation. My NewRelic Apdex score went from .5 to .9.

expiring action_cache

2009-07-17T12:10:00.001+02:00

I was having a bugger of a time expiring my cache. The problem was the cache_path.
I have index actions that take filters. So, I created a cache_path for my action caching:

caches_action :index, :layout => false, :cache_path => Proc.new { |c| "#{c.params[:league_id]}/index/#{c.params.values.sort.to_s}" }

Which means that all my caches end up in a nice tree of 'views/[league name]'. But, how to expire a whole part of the cache tree when there could be dozens of nodes as people could be searching on last names too.

What I found was that expire_action was NOT the way to go. expire_fragment lets you pass in a regular expression. Now in my sweeper I have this:

  def expire_cache_for(player)
    expire_fragment(%r{views/#{player.league.to_param}.*})
  end

Which will expire all the cached code for the player's league.

installing rmagick on osx leopard

2009-06-23T22:36:00.002+02:00

I've put up a gist of a bash file that works (as of today) to do this from source.
You can clone it:

git clone git://gist.github.com/134823.git gist-134823

nerdy but funny

2009-06-21T20:24:00.000+02:00

Ruby Toolbox

2009-06-07T17:53:00.002+02:00

Just found this site (via a tweet)... seems to have a comprehensive list of rails extensions, broken down by category.
If only they linked to them!

Ruby Toolbox

Google's Page Speed

2009-06-06T19:29:00.001+02:00

Google releases Page Speed - get it here - an open-source Firefox plugin similar to Yahoo's YSlow

i18n

2009-06-06T13:55:00.003+02:00

We've (finally!) moved FilmAmora up to rails 2.2.2 - and not without some headaches.
The biggest ones seem to revolve around translations. We were using po/mo files and that worked great. Moving to 222 required some changing of gems (to fastgettext) and other things. But we were left without standard translations of all the ruby messages for some reason.

The guys at rails i18n.org came to the rescue!

Their Github project

Has config files for most locales that will cover all the standard messages. A real life saver!

installing the mysql gem under leopard

2009-05-24T20:17:00.001+02:00

I am building a new dev laptop (an MSI Wind u100 running Leopard!) and got everything going... but getting the mysql gem to install was tricky. I had installed MySQL from a binary DMG install.
The trick was to install the gem like so:

sudo gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config

Forcing a different render format

2009-05-24T16:15:00.002+02:00

I am writing an OAuth system for FilmAmora for the iphone app (and anything else I guess) to use.
I want to always output xml from the controller that is handling the catalog output... but how to do that!

I came up with this:

  def list
    render :template => "/catalogs/list.xml.builder", :content_type => "text/xml"
  end

This will always output xml (with the right mime type) no matter what the requested format was.

Creating a new rails app with an old version

2009-05-18T17:55:00.001+02:00

After much hunting I found how to do this

gem list rails

to see what versions you have. I need to create a 2.1.2 rails app. you can do that by issueing this command:

rails _2.1.2_ my_app

cookies and hosts / domains

2009-05-10T15:09:00.003+02:00

I have been struggling for weeks with a problem of users being logged in to mysite.com, but not en.mysite.com and es.mysite.com. I posted on Ruby Groups and even tried to hire someone to do the work... but to no avail.
Finally I think I cracked it today.

Two things. First, for some reason doing this didn't work:

  if RAILS_ENV == 'production'
    config.action_controller.session = {
      :session_key => '_my_session',
      :session_domain => '.mysite.com',
      :secret      => 'xxx'
    }
  else
    config.action_controller.session = {
      :session_key => '_my_session',
      :session_domain => '.mysite.local',
     :secret      => 'xxy'
    }
  end

So, I substituted this into each environment file:

config.after_initialize do
  ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS[:session_domain] = '.mysite.local'
end

That got my session's having the correct domain set (.mysite.com) meaning that the session was saved across subdomains.

Cookies

But then I encountered a second problem. Cookies.
After switching to hide and seek / show and tell methods to enable me to page cache even pages that had logged in status all over them (more on this in another post) I needed some user info in a cookie. But my cookies were tied to the subdomain!
I was setting the cookies like this:

      cookies[:current_user] = {
        :cart_items_count => cart_items_count,
        :nothing => 1
      }.to_json

I thought, ok I just need to add the domain in:

      cookies[:current_user] = {
        :cart_items_count => cart_items_count,
        :nothing => 1,
        :domain => ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS[:session_domain]
      }.to_json

No...this didn't work. Everything was being chucked into the cookies 'value' and not the domain field (same happened if I added in an expires_at key). What was going on?
Aha! It is the json-izing.

So, I moved that out to a seperate variable:

      value = {:cart_items_count => cart_items_count, :nothing => 1}.to_json
      cookies.delete :login, :domain => ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS[:session_domain]
      cookies[:current_user] = {
        :value => value,
        :domain => ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS[:session_domain]
      }

Now my cookies had the proper domain info and the value was still proper json. And now my users are logged in across sub domains!

backgroundrb by hostname

2009-03-21T13:19:00.003+01:00

I love backgroundrb - the way to run scheduled tasks and async workers. BUT on my setup I wanted to run certain backgroundrb tasks on one machine and certain ones on another (for memory reasons and also for testing).

I couldn't find something that did that so I forked backgroundrb into my own repository on GitHub. You can find it here.

You can now have a config file like this:

# A Sample YAML configuration file
---
:backgroundrb:
  :ip: 0.0.0.0 #ip on which backgroundrb server is running
  :port: 11006 #port on which backgroundrb server is running
  :environment: production # rails environment loaded, defaults to development
  :debug_log: true # whether to print debug logs to a seperate worker, defaults to true
  :log: foreground # will print log messages to STDOUT, defaults to seperate log worker
  # :result_storage: memcache # store results in a mecache cluster, you also need to specify location of your memcache clusters in next section
  :persistent_disabled: true # turn this off if your application doesn't use backgroundrb's persistent/enqueued tasks system
  :persistent_delay: 10 # the time (seconds) between each time backgroundrb checks the database for enqueued tasks
  :per_environment: true

# You specify your worker schedules here
:whatever.local:
    :backgroundrb: 
        :environment: test
    :schedules:
      :test_worker: # worker name
        :test: #worker method
          :trigger_args: */5 * * * * * * #worker schedule
          :data: whatever

:phils-mac-pro.local:
    :backgroundrb: 
        :environment: development
    :schedules:
      :test_worker: # worker name
        :test: #worker method
          :trigger_args: */5 * * * * * * #worker schedule
          :data: phil

You can launch backgroundrb with a specific hostname like this:

./script/backgroundrb --hostname=whatever.local

I've created Capistrano tasks that fire up an instance on 2 servers and now they act differently based on hostname.

Sweet!

Generating sitemaps and xml catalogues

2009-03-13T16:23:00.002+01:00

One of the things we have to do at FilmAmora is generate a catalogue of our films. This is used in a few ways, but principally to drive our recommendations system.

I stumbled across this great blog entry:
Generating-Sitemaps-With-Rails that not only tells you a very nice way of generating sitemaps but also gave us great ideas for using the same technique to generate our catalogues.

Awesome!

ec2onrails EU

2009-02-13T20:39:00.003+01:00

Well, necessity is the mother of invention. We needed new FilmAmora servers to deal with the increasing load. I also wanted to move to EU-based EC2 instances.
That meant rolling our own because I believe there are no publicly available ec2onrails instances that run in the eu-west region.

This was not as easy as I would have liked, so here's a little step by step.

1. Go to Alestic and get the id of the proper eu instance. (At this writing it is ami-ac032bd8)

2. Fire up an instance of this. I use Elastic Fox and you need to set the region to europe then choose this instance.

3. copy your pem's into the instance:

scp -i IDENTITY {cert,pk}-*.pem root@HOSTNAME:/mnt/

4. ssh into the instance

ssh -i IDENTITY root@HOSTNAME

5. Get the build script

wget http://ec2ubuntu-build-ami.notlong.com

6. update so you can get git

apt-get update && apt-get upgrade -y

7. get git

apt-get install git-core

8. get ec2onrails and use the latest branch

git clone git://github.com/pauldowman/ec2onrails.git
cd ec2onrails
git checkout --track -b 0.9.9.1 origin/0.9.9.1

9. Run the build script!

 bash ec2ubuntu-build-ami \
 --script /root/ec2onrails/server/build-ec2onrails.sh \
 --user YOUR_ACCOUNT_NUMBER \
 --access-key YOUR_ACCESS_KEY \
 --secret-key YOUR_SECRET_KEY \
 --bucket BUKCET_NAME \
 --prefix PREFIX \
 --location EU

Your account number is the 12 digit number from your amazon access information page.

10. Register the instance

ec2-register bucket/manifest.xml --region eu-west-1

That's it!
You should now be able to go to Elastic Fox and launch and instance of this that will be in Europe!

DropDown menu for Blueprint tabbed menus

2009-02-11T13:12:00.001+01:00

I love blueprint css! And I also love the tabs menu plugin for it. But, we neede drop down menus on FilmAmora. So I wrote a little extension that gives you that. It's not perfect, but, it works for now.

Find it here:
Blueprint_dd

BackgroundRb and Synchronous calls

2009-02-06T12:05:00.002+01:00

BackgroundRb lets you also call the workers synchronously. This is quite neat as you can use it as a form of pseudo-proxy and all other kinds of goodies.
But, the documentation is lacking, so I am posting this for all you frustrated Googlers out there. Here is what you need to do.

class SampleWorker < BackgrounDRb::MetaWorker
  set_worker_name :sample_worker

  def create(args = nil)
    # this method is called, when worker is loaded for the first time
  end
  
  def test_me args
    puts "test_me 1 (puts)"
    logger.debug("test_me hit (log)")
    return "of course it works"
  end
end

Now, the BackgroundRb site tells you to call this like so to get a response back:

MiddleMan.worker(:sample_worker).test_me(:arg => "1")

This, in fact, won't work.

You need to do this:

MiddleMan.worker(:sample_worker).test_me({:arg => "1"}, true)

The second parameter (and make sure you put {} around your first set of args) is some freaky boolean to say, "no, seriously, I want a result back". Otherwise you get nil.

Bonus Tip!
When you define a method in a worker it HAS to accept arguments.
This won't work:

def test_me
end

This will:

def test_me(something = 1)
end

Rails and eTags

2009-01-16T20:28:00.001+01:00

In the never ending search for speed I've been doing a lot of stuff with 304 Conditional get's. Here is a good article about etags - which is a vital component to it!

Rails-ETags

MySQL and Leopard

2008-12-12T17:19:00.003+01:00

I had such problems on my macbook (though not on the macpro) in getting mysql running.

This article seems to have helped.

checkboxes for multiple items

2008-12-08T13:00:00.002+01:00

I am doing an interface where I want to list a bunch of things (index) and have a checkbox next to each of them. Then, using a button, edit or show all those selected items.
I am always running into this problem of checkboxes (maybe I am just stupid) so I spent some time and figured out a way that works for me. There are probably other solutions, but here is what I do:

First, in the index.html I have this javascript code:

<% content_for :javascript do %>
<script type="text/javascript" charset="utf-8">
/* <![CDATA[ */
    function editDevices() {
        var device_count = document.getElementsByName('device[]').length; 
        var total="";
        for(var i=0; i < device_count; i++){
            if(document.forms['group_edit'].elements['device[]'][i].checked) {
                if (total.length > 0) {
                    total += ","
                }
                total +=document.forms['group_edit'].elements['device[]'][i].value;
            }
        }

        new Ajax.Request('<%= group_edit_devices_path %>', { 
            parameters: { 
                authenticity_token: encodeURIComponent('<%=form_authenticity_token%>'),
                device_ids: total
            } 
        } 
        );
    }
    
    function showDevices() {
        var device_count = document.getElementsByName('device[]').length; 
        var total="";
        for(var i=0; i < device_count; i++){
            if(document.forms['group_edit'].elements['device[]'][i].checked) {
                if (total.length > 0) {
                    total += ","
                }
                total +=document.forms['group_edit'].elements['device[]'][i].value;
            }
        }

        new Ajax.Request('<%= group_show_devices_path %>', { 
            parameters: { 
                authenticity_token: encodeURIComponent('<%=form_authenticity_token%>'),
                device_ids: total
            } 
        } 
        );
    }
    /* ]]> */
</script>
<% end %>

Then I have the table of devices:

<table border="0" cellspacing="5" cellpadding="5">
    <thead>
        <tr>
            <th>&nbsp;</th>
            <th>Uid</th>
            <th>Name</th>
            <th>Ip</th>
            <th># Displays</th>
        </tr>
    </thead>
        <tfoot>
            <tr>
                <td colspan="5">
                    <ul class="tabs">
                        <li><%= link_to_function "Show", "showDevices()"  %></li>
                        <li><%= link_to_function "Edit", "editDevices()" %></li>
                        <li><%= link_to "Delete" %></li>
                    </ul>
                </td>
            </tr>
        </tfoot>
        <tbody>
            <% for device in @devices %>
            <tr>
                <td><%= check_box_tag("device[]", device.id, false, :id => "device_#{device.id}") %></td>
                <td><%=link_to_remote h(device.uid), :url => formatted_device_path(device, :js), :method => "get" %></td>
                <td><%=h device.name %></td>
                <td><%=h device.ip %></td>
                <td><%= device.displays.count %></td>
            </tr>
            <% end %>
        </tbody>
</table>

The result is that my ajax function will be called with a comma separated list of the ids of the selected devices.

&@$%^£& IE 7 and insecure content!

2008-11-27T12:33:00.002+01:00

What a morning!

A user kindly reported that he couldn't log in to FilmAmora using IE7. IE6 worked, FireFox worked. Safari worked.

IE7 (for me) was coming up first with the 'there is insecure content on this page do you wish to show it?' (the login in page is under https). Clicking yes or no had the same result: the page would flash but then IE7 reported that it couldn't show the page. (It should have, it turns out, said 'I am choosing not to show the page').

The issue was that insecure content. For me, specifically it was the statcounter call. But your issue may be other calls. Anyway, I removed all calls to anything outside of my domain and that seems to have fixed it!

unfuddle

2008-11-03T11:56:00.001+01:00

I use unfuddle as a git host and bug tracking, etc service. But, because I have so many projects on the go I ran into this problem where you can only have an ssh key in ONE account on unfuddle. So, I wrote to the support guys and asked them how to handle multiple accounts from one machine.
Here is their response:

The easiest way to deal with need multiple keys for multiple accounts is to actually just create them using different filenames in your ~/.ssh/ directory. For instance, if you have 2 accounts, "account1" and "account2", you would want to run "ssh-keygen -t rsa" twice, each time specifying a different filename:

~/.ssh/unfuddle_account1_id_rsa
~/.ssh/unfuddle_account2_id_rsa

Now, by default, ssh will use the file ~/.ssh/id_rsa when connecting to a remote server. As neither of your keys is in this file, you must tell SSH (not git) which private keyfile to use when connecting to each account. This is done using the SSH config file.

For instance, create or edit a file named .ssh/config with the following contents:

Host account1.unfuddle.com
User git
IdentityFile ~/.ssh/unfuddle_account1_id_rsa

Host account2.unfuddle.com
User git
IdentityFile ~/.ssh/unfuddle_account2_id_rsa

I have followed this model for my own accounts (of which I have many) and it has worked perfectly. I hope this helps you as well.

Creating a summary

2008-10-16T19:00:00.002+02:00

I have an app that has rows of statistics. We wanted to create summaries of arbitrary rows.
This method seems to work:

  def self.create_summary_row(array)
    unless array.blank?
      klazz = array.first.class
      sum_stat = klazz.new
      array.each do |line|
        if line.is_a?(klazz)
          line.attributes.each do |k,v|
            unless v.nil? || v.is_a?(Time) || v.is_a?(Date)
              if sum_stat[k].nil?
                sum_stat[k] = v 
              else
                sum_stat[k] += v
              end 
            end
          end
        end
      end
      sum_stat
    end
  end

ec2onrails

2008-09-29T18:56:00.004+02:00

This is so great - goodbye other hosting companies that can't compete!

Amazon has hosting facilities known as ec2 (elastic compute cloud). You pay by the hour ($.10 for a small installation). That works out to around $70/month. But, the power you get is much better than your average hosting.

Paul Dowman has released a great ec2 instance called ec2onrails that can get your rails app up and running on ec2 in about 30 minutes. I am now using it for FilmAmora as well as for another client.

It is awesome!

One thing I discovered today was the sweet built-in support for cron jobs. Write a script and plop it in your app's script directory. Call it hourly, daily or weekly (perhaps monthly? I didn't check). This script will get run automatically by an already existing cron job! No need to figure out those silly cron settings. Very cool!

I highly recommend it. FilmAmora is running much much better on it than on the old hosting (where I had to restart it every day just to hopefully have enough memory to run 2 mongrels).

secure forms

2008-08-15T09:52:00.002+02:00

One thing I struggled with in RoR was creating forms that submitted to https. So I ended up writing a couple of helpers:

  def secure_form_for(record_or_name_or_array, *args, &proc)
    unless RAILS_ENV == 'production'
      url_options = {}
    else
      url_options = {:protocol => 'https://', :only_path => false }
    end
    
    options = args.last.is_a?(Hash) ? args.pop : {}
    if !options[:url].nil?
      options[:url] = url_options.merge options[:url]
    else
      options[:url] = url_options
    end

    return form_for(record_or_name_or_array, options, &proc)
  end

and

  def secure_form_tag(*args, &proc)
    logger.debug("secure_form_tag args #{args.inspect}")
    unless RAILS_ENV == 'production'
      url_options = {}
    else
      url_options = {:protocol => 'https://', :only_path => false }
    end

    options = args.last.is_a?(Hash) ? args.pop : {}
    if !options[:url].nil?
      options[:url] = url_options.merge options[:url]
    else
      options[:url] = url_options
    end

    logger.debug("secure_form_tag options #{options.inspect}")

    return form_tag(options, &proc)
  end

The should be simple replacements for form_for and form_tag. Now, I have to admit I wrote these when I was using Rails 1.2.5, so there might be some things in 2.x that make these redundant, but, they work for me.
The only caveat is that if you are doing a form where you are not passing in the action, just the object e.g

<% form_for @object do %>

Then you are going to have to supply the actions. I am looking into this now, and I might rewrite this post if I sort it out!

xml_hidden plugin

2008-07-26T15:08:00.003+02:00

Recently I've been investigating the steps needed to open up FilmAmora's API to the outside world. That means supplying xml data so people can do what they want with it.
But I didn't want sensitive data in models to be exposed in the xml. And I also am too lazy to write custom to_xml methods for everything.

So I wrote a plugin call xml_hidden That lets you set something on the class to hide certain attributes from xml output.

class Film < ActiveRecord::Base
  attr_xml_hidden :acquiring_url, :id, :created_at
end

Now whenever I output a film to xml those values can't be seen.

Escaping for JavaScript

2008-07-20T10:30:00.001+02:00

Recently I've started to use the most excellent Prototip2 for doing sexy tooltip stuff. It works a treat.
The only problem I have had is that some of the things I want to put into the tips have single quotes in them (e.g Bob's Team). I was surprised to find that Rails doesn't (as far as I could find) have a handy dandy way of making strings JS friendly. So I whipped up this extention to Erb::Util.
I hope some of you find it useful. It will escape single quotes and as an added bonus it also does the html escaping so you only have to make one call.

class ERB
  module Util
    def js_escape(str)
      h(str.gsub(/[']/, '\\\\\''))
    end

    alias js js_escape
    module_function :js
    module_function :js_escape
  end
end

I put this into an initializer.

You can then call it from your views like this:

<%=js team.name%>

Combating XSS

2008-07-14T13:01:00.001+02:00

This info is available elsewhere, but as I always forget, this is a good place to repeat it.

Install the White List plugin. Get it here

Once installed you'll need to add the following line to the init.rb of the white_list plugin:

ActionView::Base.send :include, WhiteListHelper

In the Application.rb (Application Controller)

include HtmlFilterHelper
before_filter :sanitize_params

Then I have a class called HtmlFilterHelper

module HtmlFilterHelper
  def sanitize_params(params = params)
    params = walk_hash(params) if params
  end

  private
  def walk_hash(hash)
    hash.keys.each do |key|
      if hash[key].is_a? String
        hash[key] = white_list(hash[key])
      elsif hash[key].is_a? Hash
        hash[key] = walk_hash(hash[key])
      elsif hash[key].is_a? Array
        hash[key] = walk_array(hash[key])
      end
    end
    hash
  end

  def walk_array(array)
    array.each_with_index do |el,i|
      if el.is_a? String
        array[i] = white_list(el)
      elsif el.is_a? Hash
        array[i] = walk_hash(el)
      elsif el.is_a? Array
        array[i] = walk_array(el)
      end
    end
    array
  end
end

For what it is worth, I also do stuff like this:

<%=white_list synopsis.synopsis[0,100].gsub(/<\/?[^>]*>/, "")%>

Which removes any html-style tags from the text and then white_list's it before outputting.

Becoming a Git

2008-07-09T23:28:00.000+02:00

Recently the SVN repository service we were using for FilmAmora had a lengthy outage. So, being a real Rails-er I decided to switch over to GitHub. Getting the source up into it was pretty easy.

So, I went ahead and started making changes and doing git commit's and all that good stuff. Well, today I went to the projects homepage on GitHub and you can imagine my surprise and horror when I saw that it looked like nothing had been checked in since my original commit.

Wot de hell?!

I then scoured the net for Git for dummies without success. So here it is:

1) Do all the stuff they tell you do to to get your code in.
2) Do your commits... but remember that for some strange reason you are committing into your LOCAL Git repository.
3) here's the key - do a git push! that will chuck your code up onto the server.

Seems perhaps obviously, but coming from an svn or cvs background it confused the hell out of me.
At a later date (probably this weekend) I will write a little Git note about how this works from the perspective of someone used to using one of the more traditional source code systems.

Euro-friendly permalinks

2008-07-05T11:51:00.000+02:00

Permalink_fu is rather an awesome thing. The only drawback is that it doesn't really handle accented characters very well. I've looked around and seen a bunch of solutions that attempt to be 'clever' and do the substitution on one line, etc. But I decided that I'd rather go for brute force.
I created an extension to String with one method called 'no_accents':

class String
  def no_accents

Then I do a whack of these:

    str = self
    #Spanish letters
    str = str.gsub("á", "a")
    str = str.gsub("é","e")

For upper and lower case.
Obviously there is a big list. Right now I am just doing Spanish, French and German as they are the ones I am most interested in.

I had to make a change to Permalink_fu to call my method first:

    def escape(str)
      str = str.no_accents
      s = ((translation_to && translation_from) ? Iconv.iconv(translation_to, translation_from, str) : str).to_s

But that's all.

Here is the file. What I did is put it in my initializers directory.

If you add to it maybe you could post your additions. I am sure some clever-clogs out there will come up with a more slick solution. But... mine works!

Rails 2.1 caching - nothing is ever easy!

2008-06-28T13:10:00.000+02:00

Last night I watched the new Railscast episode that talked about the new caching features in Rails 2.1. I thought it looked cool and would add it to the new round of FilmAmora changes.

But... nothing is every easy!

I may look simple, but I encountered several problems

1. Conflicts with GetText

We use GetText for the translations on FilmAmora. We like it because there are free poEditor apps on every platform and we can easily send off the files to whomever to be translated.

The problem comes with this line:

require 'gettext/rails'

That you need to have to fire up some of the Rails-specific GetText stuff. This is all fine - it has been working for quite some time. But, when I tried to cache our Genres like this:

  def self.all_cached(language)
    key = "genres_#{language}"
    Rails.cache.fetch(key) {Genre.find(:all).sort_by {|genre| genre.get_description}.reject {|g| g.get_films_count == 0}}
  end

Here is the result:

undefined method `cache' for GetText::Rails:Module

What?!?! Yes, it seems that GetText::Rails will hide Rails. This is, quite frankly, SHIT. So, after a long time poking around I have discovered that you need to do this:

  def self.all_cached(language)
    key = "genres_#{language}"
    ::Rails.cache.fetch(key) {Genre.find(:all).sort_by {|genre| genre.get_description}.reject {|g| g.get_films_count == 0}}
  end

Yippee! It will all work now, right?

Wrong.

2. Class != Class

I hit refresh the first time and wow was I excited! I saw lines like this in the log:

Cache write (will save 0.59562): genres_es

Woo hoo! Look at all the time I will save!

So I hit refresh.

undefined method `get_description' for #<Genre id: 1, description: "Action and Adventure", order_by: 2>

What?!?!

I am using the 'default' memory store. Something is going funny with retrieving objects from it. I never solved this problem.

If I do this in the console:

>> @genres = Genre.all_cached("es")
>> @genres = Genre.all_cached("es")

I see this in the log:

Cache write (will save 0.51974): genres_es
Cache hit: genres_es ({})

And I can do this:

>> @genres[0].get_description("es")

So what is going on in my web app? I have no idea. It seems that the class retrieved from the memory cache is incomplete in some way. get_description is not an accessor, it does go off and get the translation, but... so what? it is still a method.
This had me stumped!

3. File Store no worky
So I added this to development.rb:

config.cache_store = :file_store, '/cache_store'

I ended up getting this:

undefined method `get_description' for #<String:0x5459f0c>

Ok, I am tired of this now.

4. MemCached
Everyone is talking about using MemCached for this kind of thing. Now, I know developers and we are a lazy bunch. My guess is that this whole caching stuff has been written with MemCache in mind and screw anything else (see points 2 and 3 above).
So I installed MemCache and changed the line in the development.rb to be this:

config.cache_store = :mem_cache_store

I restarted the server and hit refresh. Trying to contain my excitement I saw a properly rendered page.
I hit refresh again.

undefined class/module Subgenre

For CHRIST'S SAKE!
After another web-scouring exercise I discovered a solution.

My method now looks like this:

  def self.all_cached(language)
    key = "genres_#{language}"
    Subgenre
    ::Rails.cache.fetch(key) {Genre.find(:all).sort_by {|genre| genre.get_description}.reject {|g| g.get_films_count == 0}}
  end

And guess what? It works. Of course it now means running memcache on my local machine and installing it on the production box. But I've saved .52 seconds! It took 3 hours to get to the solution, so I figure in only 350 web page hits I will make it back.

Oh, of course I was already caching the html on the server for most things, but this is a little tidier.

Linking offsite

2008-06-27T09:27:00.000+02:00

When you get into something like Ruby on Rails you want to use it for everything - the old adage about getting a hammer and everything becomes a nail rings true in programming!

I wanted to link to Wikipedia. I thought I could use link_to for something like that, but it turns out that really link_to (and it's variations) are meant to link to something inside your RoR app, not outside of it.

So I wrote this handy method that generates a link off to Wikipedia.

def wikipedia_link(text, language = current_language.to_s)
  "<a href=\"http://www.wikipedia.org/search-redirect.php?search=#{h(text)}&amp;language=#{language}\" target=\"new\">Wikipedia</a>"
end

It's fairly simple - pass in the text you want to find on Wikipedia and your language.

IE and form spacing

2008-06-19T10:22:00.000+02:00

One has to wonder how Microsoft got it so wrong, but anyway.

One thing I love about RoR is the ajax side of it and how easy it makes it. For instance adding something to a cart with ajax:

<% form_remote_tag :url => {:controller => :store, :action => :add_to_cart, :id => film} do %>
    <%=submit_tag(_('Buy &raquo;'), "title" => _("add to cart @ %s") % number_to_currency(film.price, :unit => "€"), :class => "buy_button_big buy_button")%>
<% end %>

That is great. On FireFox it looks like this:

But on IE6 it looks like this:

Insane quatities of extra space. Seems that IE loves to add space after forms. You can't stop it!

Or can you!?

I have found that adding this in my css is the trick:

form {
  margin:0;
  padding:0;
}

But, since I don't need it in ff... why not use the very sweet CSS Browser Selector plugin.

Now you can do this:

.ie6 form {
  margin:0;
  padding:0;
}

Add all shall be well in the garden.

pull from another site

2008-06-17T21:12:00.000+02:00

Recently a site I was working on needed to forward all unknown requests to another website (their old website). This turned out to be shockingly easy to do!

In the Application Controller:

  def default
    logger.debug("request.env[\"REQUEST_URI\"] #{request.env["REQUEST_URI"]}")
    render :text => Net::HTTP.get_response(
      "www.oldsite.com",
      request.env["REQUEST_URI"]
      ).body
  end

And at the very bottom (so it is only done if nothing else catches the route):

  map.connect '*path' , :controller => 'application' , :action => 'default'

Easy peasy!

Format My Source Code for Blogging

2008-06-14T13:10:00.000+02:00

Format My Source Code for Blogging

Absolving your views of responsibility

2008-06-14T09:39:00.000+02:00

One thing I struggled with when I started out with RoR was how to create a site that was 'current looking'. In all the books they told you about views and layouts, but not how to use the layout to its most effective end.
What I wanted was a simple 3 column website. But instead of 'static' information in the left and right column I wanted it to change. In some cases it was ads, in some cases a list of items, etc. The only thing I could think of was to hold a lot of layout information in the view. So, I would have the header and footer in layouts/application.html.erb but in the middle there was a big <%= :yield %> that would handle the middle (vertically) of the page, including the columns.
(Hold on, I'm getting to the code!)
The problem with this is that if your view is wrong, or if you don't include it, then your site goes all to hell.

The solution, I found, was named :yields.

To illustrate I am going to build a funky 3 column site in 5 minutes.

Create the project

$ rails threecol

To set up a new project

Neat tip: open TextMate and drag the threecol folder onto the icon in the doc. That way it will open up the whole structure (i.e. the rails project).

Add a layout

Matthew James Taylor has a great website with some free to use CSS templates. So, we're going to use his 3 column one.

Go here. View source, copy the CSS.

Create a new file in your public/stylesheets directory called site.css and paste the CSS from Matt's page in.

Include the CSS in your Layout

Create a file in your app/views/layouts directory called application.html.erb. This is the name for the default layout, so you don't need to change anything in the controller we'll create in a moment.

Go back to Matt's page, view source and paste the whole thing into application.html.erb. Delete the styles definitions from here. (lines 12 - 170 as of this writing)

Knock up a scaffold

(Best to setup the database.yml here first)

$ script/generate scaffold product name:string description:string

note: in Rails 2 you don't have the same scaffolding as before. doing the above, where you define some fields, will give you a controller, model, views and migration

Get everything setup:

$ rake db:drop db:create db:migrate test

Get your CSS on

Delete products.html.erb from the layouts directory. For our example we don't need it.

Back in your application.html.erb paste this line in the header:

<%= stylesheet_link_tag 'site' %>

If you start the server (script/server) now and go to localhost:3000/products you should see Matthew's page all nice and purty because we haven't included any of our content.

Get your content in

Delete the center of Matthew's code (lines 40 - 59 here) and replace it with

<%= :yield %>

Refresh now and you will see your scaffold in the middle. Nice.

Sidebars

Ok, we could have done all that easy, so let's get to the fun stuff.
Delete Matthew's code for either sidebar.
The center of your application.html.erb should now look like this:

            <div class="col1">

                <!-- Column 1 start -->
                    <%= yield %>
                <!-- Column 1 end -->
            </div>
            <div class="col2">
                <!-- Column 2 start -->
                <!-- Column 2 end -->
            </div>
            <div class="col3">
                <!-- Column 3 start -->
                <!-- Column 3 end -->
            </div>

inside column 2 (the left nav in Matthew's layout) put this:

<%= yield :left_nav_content %>

In views/products/index.html.erb do this:

<% content_for :left_nav_content do %>
    <h1>Here it is!</h1>
    <div>Exciting left-nav content</div>
<% end %>

What is going on?

We all know about <%= yield %>, but adding a name to it says to look in your view (or any views including, even through partial rendering) and see if there is something that wants to write out to this space. <% content_for :left_nav_content do %> says that the stuff in here, please write there.

The power is - you can do anything in there. Include partials, for instance.

The advantage is that now you can not screw up your layout. You can still put things whereever you want, but the layout is out of the view's hands.

I hope this is clear. It made a big difference to how I construct views!

Ultrasphinx: ' != "

2008-06-11T12:17:00.001+02:00

Just ran into a wee ultrasphinx problem.

I had been doing this:

@query = "actor:'#{@person.name}' OR director:'#{@person.name}'"

but it turns out that did nothing! You need to surround query segments in double quotes for it to actually parse them as a single unit:

@query = "actor:\"#{@person.name}\" OR director:\"#{@person.name}\""

Legitimate job posting?

2008-06-10T13:52:00.001+02:00

I recently responded to a job posting I saw on jobs.rubynow. The company seemed to like me:

"Thank you for your interest in ***. After our first round of resume reviews, we are interested in moving you forward to our next round of evaluations. This is a significant accomplishment considering the competitive nature of this position, and the small number of candidates for a position that go to on – so congratulations! "

Wow! Great! Not exactly sure of the English at the end of the sentence there, but cool.

They then proceeded to ask me a list of questions... all of which I had already answered on the resume they loved so much! But I humored them and answered them again.
A few days later I got this back:

"A mini-test is a round between resume reviews and interviews, that helps the company get a sense of your real interest in the job, as well as your skills and abilities. Receiving a mini-test is a pretty significant accomplishment considering the competitive nature of this position, and the small number of candidates for a position that will receive one – so congratulations! "

Hmmm surprisingly similar to the first email.

They wanted me to write a test. Not really what I like to do, but perhaps I'd be interested. But here is the test:

3. Code as much of the system as you can within a time-box, which is a maximum of 10 hours in this case.

Uh, they want me to spend 10 HOURS???? on their test!?!

The real kicker for me was that they sent a spec for the test. It was to write them a new system for their HR department to track resumes! They didn't want a test, they wanted a free application!

They've made it sound like competition is fierce, so you'd better really do a good job. Now, most developers are proud of what they write and it is a remote test, so why not spend 20 hours or more and create an awesome system!

Or am I just being paranoid?

Oh, the also never answered any of my emails. I only got the bulk emails going out to candidates.

BTW, the company was Rich Applications Consulting,

Ultrasphinx cheatsheet: indexing habtm relationships

2008-06-10T09:27:00.000+02:00

In the previous post I quickly summarized how I am doing the indexing of fields of a model. Quite simple I know.

I struggled a little bit with getting the other models I wanted indexed to come through.

Before, as I mentioned, I was using Ferret. Now, despite all of its memory-hog bugginess it is simple to understand because you can modify stuff at the post-sql stage. So before, for instance, I had a method on the film model that did this:

def participants_name
    participants.collect{|a| a.name}.join(' ')
  end

That gave me a big long string and I could pass that to acts_as_ferret and it would use that to find all the films with Johnny Depp in them.

The problem came with Sphinx not doing any post-sql processing. I wanted the same results, but how?
The model looks like this:

class Film < ActiveRecord::Base
  has_and_belongs_to_many :participants
...
end

At first I thought I could index participants and then step through all of their films, but that seemed too much work.

So... here comes the cheatsheet part!

How to index HABTM relationships

Again, maybe this is in the documentation but I had a hard time piecing it together.

I do this:

is_index :concatenate => [
    {:class_name => "Participant", 
     :field => "name", 
     :table_alias => "p1",
     :as => "actor", 
     :association_sql => "LEFT OUTER JOIN films_participants ON (films.`id` = films_participants.`film_id`) LEFT OUTER JOIN participants AS p1 ON (p1.`id` = films_participants.`participant_id`)"}
]

So, where as the :fields array of is_indexed is used for the fields of a model, :concatenate is where you set up the habtm relationships.
To break it down
1 - :class_name => the name of the class that this model relates to. We need to set this because you can reference by a different name as we'll see in a moment

2 - :field => same as the fields table on this model, what field do we want to index on. Not it is called 'field' and not 'fields'! You can only do one.

3 - :table_alias => i.e. GODSEND! I have 2 Participant habtm in my film model - one for actors and one for directors. The directors :concatentate definition is almost exactly the same as the actors one. If we didn't have :table_alias Sphinx wouldn't be able to tell them apart. Remember, Sphinx uses all this cleverness to generate one (very clever!) bit of sql.

4 - :as => what you want to call this in your search. I can then write a query that says "actor:'Johnny Depp'" and it will search only on that field.

5 - :association_sql => yes, Virginia, I am sorry but you have to write your own sql. It will ALWAYS look like this, so just copy and past mine and replace the table and column names.

That's it.

You can now do a query like this:

@films = Ultrasphinx::Search.new(:query => "Johnny Depp", :class_names => ['Film']).run

And it will search for any film titles with Johnny Depp in them AND any actors in the film. Sweet!

Now, didn't I say something about directors?

I have another habtm relationship:

 has_and_belongs_to_many :directors, :class_name => "Participant", 
                            :join_table => "directors_films"

Same other model (Participant). So now my :concatenate will look like this:

:concatenate => [              
                {:class_name => "Participant", :field => "name", :table_alias => "p1",
                :as => "actor", :association_sql => "LEFT OUTER JOIN films_participants ON (films.`id` = films_participants.`film_id`) LEFT OUTER JOIN participants AS p1 ON (p1.`id` = films_participants.`participant_id`)"},
                {:class_name => "Participant", :field => "name", :table_alias => "p2",
                  :as => "director", :association_sql => "LEFT OUTER JOIN directors_films ON (films.`id` = directors_films.`film_id`) LEFT OUTER JOIN participants AS p2 ON (p2.`id` = directors_films.`participant_id`)"}]

Two references to Participant, but :table_alias saves the day.

Ultrasphinx cheatsheet: indexing fields you need!

2008-06-09T16:28:00.000+02:00

The Sphinx/Ultrasphinx combination is great for searching. Very fast and much less load on your database than other options. It also takes a lot less memory and is more reliable.

But, the documentation is maddening. Here is what I have learned about implementing it.

To index a field on a model:

is_indexed :fields => [{:field => :field_name, :sortable => true}]

The sortable lets you sort on it later in searches. The thing to remember is that sphinx knows nothing except what you explicitly tell it. So it won't know about fields you don't mention, or what you might want to do with those fields. Don't include the :sortable flag and you can't sort on it.

To that end, we have a field called 'available' that we set/unset if an item in the catalogue shouldn't be displayed online. Now, we never really want to search on that, but, we do want to include it in searches as a filter. So, the first line of our model is really this:

is_indexed  :fields => [{:field => :title, :facet => true, :sortable => true}, :available]

That means we can do this:

@films = Ultrasphinx::Search.new(:query => "title:Foo", :class_names => ['Film'], :filters => {"available" => 1}).run

And we'll just get a list of available films named Red (or with Red in the title).

Ultrasphinx & UTF-8 data

2008-06-08T08:40:00.000+02:00

On filmamora.com we obviously have a lot of data that is not your standard ascii - Spanish film titles and Actor's name especially.

After switching from Ferret to Sphinx as a search engine (more about this in another post) everything was going great using the ultrasphinx plugin. That is until I tested searching on a name like 'Belén'. No films were found with an actress with that name - though I knew there were loads of actresses with that name and therefore films.

The problem was the data settings in the database. Even though I was displaying the names correctly this was due to a transliteration between the characters in the database and the webpage. The data was not being stored correctly.

You MUST create tables as default charset utf8 to get the correct results:

Step 1) Create the table in a migration


create_table(:films, :options => 'DEFAULT CHARSET=utf8') do |t|
      t.string :title, :null => false
      ....
    end

Step 2) Tell rails that the database is in utf-8 by putting this in your database.yml:


development:
  adapter: mysql
  database: filmamora_development
  encoding: utf8

Step 3) Make sure the data going IN is in utf8. Filmamora gets a lot of data from scraping our wholesales site (using scrapi), so I do this:

film_info_es = dvdgo_movie.scrape(URI.parse("#{server}/#{page}"),
:parser_options => {"char-encoding" => "latin0",
"output-encoding" => "utf8", "quote-marks" => "true" },
:user_agent => "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12")

Step 4) search!
@films = Ultrasphinx::Search.new(:query => @query, :class_names => ['Film'],
:page => params[:page]||1, :per_page => 12,
:filters => {"available" => 1}).run

Now, if you were (like us) unlucky enough to have NOT done this from the start and can't re-run your migrations (we have a catalogue of 35,000 items!), you can do this to your tables:

ALTER TABLE mytablename DEFAULT CHARACTER SET utf8

We then deleted all our character data (film titles, actor's name, synopsis) and reimported. We kept our id's (which we needed to make sure our existing orders and wishlists, etc didn't go wrong) but got the new data.

In theory you don't need to set the charset on each table. In MySQL you can create a database with a default charset:

CREATE TABLE tablename DEFAULT CHARACTER SET utf8;

And then all your tables should use it too.

But, I don't think that rake db:create does this... (maybe it does now that I have set the character set in the database.yml... hmmm). I also am not sure if the rake db tasks to create tables explicitly set the charset or not. Anyone?

I'm going to write a few posts on sphinx/ultrasphinx because it is so awesome and I had some issues finding clear info on how to use it, so I might as well write it up.