Delayed_job for background processing in Rails

*update: delayed-job is now available on gemcutter*

The first thing to do obviously is to install DelayedJob. There are plenty of forked versions available on git-hub. I chose collectiveidea beacuse it was recommend on railscasts. I did refer to this site extensively for setting up delayed_job.

$ sudo gem install collectiveidea-delayed_job

Job half done. I followed instructions on the github page and ensure that I have my environment setup properly. I added the following to the environment.rb:

 config.gem 'collectiveidea-delayed_job', :lib => 'delayed_job',
                     :source => 'http://gems.github.com'

and the following to the Rakefile:

begin
      require 'delayed/tasks'
rescue LoadError
      STDERR.puts "Run `rake gems:install` to install delayed_job"
end

Then issue the command:

./script/generate delayed_job
rake db:migrate

To start the delayed_job server in production mode, issue:

$ RAILS_ENV=production ./script/delayed_job start

To start it in development mode, issue:

$ rake jobs:work

Interesting find – obvious but cost me a lot of time: Nothing stops one from running BOTH the above commands. Infact, in production mode I ran the delayed_job as a daemon AND also using rake. Silly me – I forgot that if I change any code I would need to restart both. I wrongly assumed that jobs:work only ‘showed’ the console — it starts the delayed_job server. So, if you do it this way, you will have twice the number of background jobs floating around 😉

Well, anyway, once I had this properly configured as a daemon, I set about changing code. This was the really aweome part of DelayedJob — no dependencies!! I changed the code from:

user = User.find_by_name('xyz')
user.get_daily_call_statistics

user = User.find_by_name('xyz')
user.send_later(:get_daily_call_statistics)

AND IT WORKS !! Awesome. While digging around a little more, I realized the there is not enough scope for debugging in the default way. I looked up github and google for some help and found this useful tip:

1. Create a config/initializers/delayed_job_config.rb and add:

Delayed::Job.destroy_failed_jobs = false
Delayed::Worker.logger = Rails.logger

This logs all delayed job output to the environment log files and ensures that failed jobs are not destroyed. There are other settings to reduce the failure attempts and the time for the delayed job but I was too excited to try them out immediately.

2. Suppose I have some global variables in the helpers, the are not accessible in the model methods called via delayed_job. Maybe a bug in delayed_job – I do plan to dig deeper into this and figure this one out — either way. I had to break my head trying to figure this one out.

To conclude, what I had earlier was:

Processing DashboardController#explicit_refresh_daily_statistics (for 121.247.65.47 at 2009-10-29 13:16:33) [GET]
Parameters: {"action"=>"explicit_refresh_daily_statistics", "controller"=>"dashboard"}
last seen ...........
Redirected to http://acemoney.in/dashboard
Completed in 74414ms (DB: 16912) | 302 Found [http://acemoney.in/dashboard/explicit_refresh_daily_statistics]

Now after adding the send_later, I have:

Processing DashboardController#explicit_refresh_daily_statistics (for 121.247.65.47 at 2009-10-29 15:16:41) [GET]
Parameters: {"action"=>"explicit_refresh_daily_statistics", "controller"=>"dashboard"}
last seen ...........
Redirected to http://acemoney.in/dashboard
Completed in 420ms (DB: 99) | 302 Found [http://acemoney.in/dashboard/explicit_refresh_daily_statistics]

This means my response time fell from from 74 seconds to 0.5 seconds

Now, I already had backgrounDrb tasks configured earlier and want to migrate them ‘somehow’ to DelayedJob with minimal code. Stay tuned, this post will be updated.