Uploading multiple files with nginx upload module and upload progress bar

Uploading files ‘efficiently’ has also been a pain point in most apps. I finally got to experiment with:

There are resources available for handing these individually but not together. I used references from here and here. It is advisable to read these 2 links before proceeding. The nuances of the directives are explained there and I don’t want to repeat it.

Nginx installation via passenger

When you are using the passenger-install-nginx-module, make sure you use the customize build with at least the following custom arguments:

--add-module=<path>/nginx_upload_module-2.2.0/
--add-module=<path>/masterzen-nginx-upload-progress-module-ac62a29/

FYI, I downloaded v2.2.0 for nginx upload module and v.0.7 for nginx upload progress module. Nginx upload module ensures that the file is uploaded by nginx and stored in a temporary location and the nginx upload progress module tracks this download and provides progress reports for it. Its important to remember that both can be used independently but its more fun to use them together!

Here is my nginx.conf:


http {
 ...
 upload_progress proxied 1m;

 server {
   listen 80;
   server_name zooppa;
   root /Users/gautam/work/showmetv/public;

   # Match this location for the upload module
   location /videos/fast_upload {
     proxy_pass http://127.0.0.1;
     upload_pass @fast_upload_endpoint;
     upload_store /Users/gautam/work/showmetv/tmp/upload_store 1;

     # set permissions on the uploaded files
     upload_store_access user:rw group:rw all:r;

     # Set specified fields in request body
     # this puts the original filename, new path+filename and content type in the requests params
     upload_set_form_field fast_asset[][original_name] "$upload_file_name";
     upload_set_form_field fast_asset[][content_type] "$upload_content_type";
     upload_set_form_field fast_asset[][filepath] "$upload_tmp_path";

     upload_pass_form_field "^X-Progress-ID$|^authenticity_token$";
     upload_cleanup 400 404 499 500-505;
     track_uploads proxied 30s;
   }

   location ^~ /progress {
     # report uploads tracked in the 'proxied' zone
     upload_progress_json_output;
     report_uploads proxied;
   }

   location @fast_upload_endpoint {
     passenger_enabled on;
     rails_env development;
   }

   location / {
     rails_env development;
     passenger_enabled on;
   }
}

Some important aspects of the above code which require some explanation:

upload_set_form_field upload[fast_asset][][original_name] "$upload_file_name";

Here the parameter being passed to the controller is

Parameters: { "action" => create,
"fast_asset"=>[{"content_type"=>"application/octet-stream",
"original_name"=>"1.flv",
"filepath"=>"/Users/gautam/work/showmetv/tmp/upload_store/5/0025383985"},
{"content_type"=>"image/jpeg", "original_name"=>"1_5575099.jpg",
"filepath"=>"/Users/gautam/work/showmetv/tmp/upload_store/6/0025383986"}],
"controller"=>"videos", "X-Progress-ID"=>"de03d98d4408e2f0d9b5eb34dd82322c" }

Notice that the upload[fast_asset] is an Array of Hashes for each file uploaded. Be sure to set X-Progress-ID in the list of fields to be passed through and ensure that track_uploads is in the same location which is uploading the files. Here is the HTML I used:

...
<head>
<%= javascript_include_tag 'plugins/jquery.uploadProgress' %>
<%= stylesheet_link_tag 'upload' %>
</head>
...
<script type="text/javascript">
 $(function() {
   $('form').uploadProgress({
     /* scripts locations for safari */
     jqueryPath: "/javascripts/jquery.js",
     uploadProgressPath: "/javascripts/plugins/jquery.uploadProgress.js",
     /* function called each time bar is updated */
     uploading: function(upload) {$('#percents').html(upload.percents+'%');},
     /* selector or element that will be updated */
     progressBar: "#progressbar",
     /* progress reports url */
     progressUrl: "/progress",
     /* how often will bar be updated */
     interval: 2000
    });
 });
</script>
...
<%= form_tag video_fastupload_path, :multipart => true, :method => :post, :id => 'upload' %>
<dl>
 <dt for="firstname">Video</dt>
 <dd>
   <fieldset>
      <%= file_field_tag :video_file %>
   </fieldset>
 </dd>

 <dt for="video_image">Video Image</dt>
 <dd>
   <fieldset>
      <%= file_field_tag :video_image %>
   </fieldset></dd>
 </dd>
</form>

<div id="uploading" style="float:left;">
  <div id="progress">
    <div id="progressbar">&nbsp;</div>
    <div id="percents"></div>
  </div>
</div>

Note that the file_field name is irrelevant. A POST request with file fields which matches the location for nginx upload module would replace the file fields with its own — in the case above with upload[fast_asset] array. The above code ensures that the file is uploaded and stored in the temporary location via the nginx upload module and it is tracked by the nginx upload progress module.  I have used the jquery.uploadProgress.js from here – its simple and sweet.

Its also important to know how to actually transfer the uploaded files using paperclip. Its simple enough:

class Video < ActiveRecord::Base

# I stored files in CloudFiles but you could use S3 buckets.
has_attached_file :video,
  :storage => :cloud_files,
  :cloudfiles_credentials => "#{RAILS_ROOT}/config/rackspace_cloudfiles.yml",
  :container => "#{ApplicationConfig['cloudfiles']['videos']['container']}"

has_attached_file :image,
  :styles => { :small_thumb => [ "50x50", :jpg ],
               :large_thumb => [ "100x100", :jpg ],
               :medium_thumb => [ "170x126", :jpg ],
               :detail_preview => [ "608x336", :jpg ]
  }

def fast_asset=(files)
 if files &&  files.respond_to?('[]')
   files.each do |item|
     self.tmp_upload_dir = "#{item['filepath']}_1"
     tmp_file_path = "#{self.tmp_upload_dir}/#{item['original_name']}"
     FileUtils.mkdir_p(self.tmp_upload_dir)
     FileUtils.mv(item['filepath'], tmp_file_path)

     if file.index(item) == 0
      self.video = File.open(tmp_file_path)
     else
      self.image = File.open(tmp_file_path)
     end
   end
 end
end

This is simplistic code to just save the uploaded files as the paperclip assets. Now another issue presents itself. We have uploaded the file to a temporary location and now want to transfer it to S3 or CloudFiles. However, the tracking continues to be in force till the request completes. After uploading to the nginx temporary store, we should update the paperclip assets asynchronously via delayed job. If this is not done, the HTTP upload request will complete after the files are uploaded to S3 or Cloudfiles, so the progress bar will show 100% (completed) and still wait till we upload to the remote store!

Using a delayed job to create this resource is a perfect solution – the user sees the progress bar complete and once the files are uploaded, the request is complete from the users perspective — the rest is server side processing.

The controller code is simpler:

class VideosController < ActionController::Base
  def create
    ...
    Delayed::Job.enqueue CreateVideo.new(params)
    ...
  end
end

And the delayed_job worker:

class CreateVideo < Struct.new(:params)
  def perform
      ...
      video = Video.new(params[:video])
      video.fast_asset = params[:fast_asset]
      ...
  end
end

The performance — 10 fold! I found files uploading faster and keeping the Rails webserver on lesser resources.Here are a few images of uploading in progress. This is a 90MB file on a LAN – uploaded in 15 seconds. Earlier, with no nginx upload module and just direct upload to a paperclip asset, this took about 1.5 minutes.

About these ads

About Gautam Rege

Rubyist, Entrepreneur and co-founder of Josh-Software - one of the leading Ruby development shops in India.
This entry was posted in Ruby on Rails and tagged , , . Bookmark the permalink.

33 Responses to Uploading multiple files with nginx upload module and upload progress bar

  1. Pingback: Tweets that mention Uploading multiple files with nginx upload module and upload progress bar « Josh Software – Where Programming is an Art! -- Topsy.com

  2. sean says:

    What does your routes file look like?

  3. Gautam Rege says:

    The routes file just requires a route for ‘/videos/fast_upload’! Eg.

    map.video_fastupload ‘/videos/fast_upload’, :controller => ‘videos’, :action => ‘create’

    The create method will get ALL the params, including the injected fast_asset.

  4. Jai says:

    Does the progress bar work for you in chrome? It doesn’t seem to work for me.

  5. Jai says:

    Also, what is your suggestion for updating (re-uploading) a file?

  6. Gautam Rege says:

    Jai,
    Progress bar works fine in Chrome. Here are some things you may have not configured:

    1. Check if the request is indeed reaching the server — the /progress?X-Progress-ID=. Check nginx access.log to see if you are seeing the request come in.

    2. You can also use curl if do have the X-Progress-ID with you.

    Does the upload succeed at all? If it does, it could even be the positioning of block in the nginx.conf. Do paste relevant information or code-snippet. It would help.

  7. Gautam Rege says:

    @Jai Regarding re-upload of a file, it depends on your Model not the upload progress bar. If you wanted to know about ‘Cancellation’ of a file upload in progress, its different.

    I found that you can simply navigate away from the page (or reload the page) — nginx detects broken connection and aborts the upload process.

    Hope this helps.

  8. chief says:

    I used this for my routes:
    match ‘/images/create/fast_upload’ => ‘images#create’, :via => [:post]

    And in my controller :
    @image = @resource.images.build(params[:image])

    That gives me an error:

    NoMethodError (undefined method `images’ for nil:NilClass):
    app/controllers/images_controller.rb:58:in `create’

    It throws this error at the very end of the upload. Also there is no difference in time in the upload with this configuration, so the upload module must not be taking the file.
    Thanks..

    • Gautam Rege says:

      @chief I don’t think the error you are getting is related to the upload progress bar — it seems that you have not initialized @resource.

      Check your code – or paste the snippet of how @resource was created.

  9. chief says:

    Started POST “/images” for …. at Sat Dec 11 07:26:49 +0000 2010
    Processing by ImagesController#create as HTML
    Parameters: {“commit”=>”Create Image”, “authenticity_token”=>”ConM/EvSKBRDSjPX8Z3AmvyYFJ7SkeMkOedEl5rz0R4=”, “utf8″=>”\342\234\223″, “image”=>{“photo”=>#<$
    Completed in 86ms

    • Gautam Rege says:

      @chief

      It seems that your nginx rules may not be functioning at all – and the request is being processed by your webserver.

      I am pretty sure you are not seeing a progress bar, as you dont have the X-Progress-ID in your parameters.

      Check the nginx.conf file to see if the “location /images/create/fast_upload ” is correct.OR paste the contents of your nginx.conf here.

      Cheers!

  10. chief says:

    At this point I am just working on the upload module and not the progress module.

    nginx.conf
    http://pastie.org/1368622

    My route is:
    match ‘/images/create/fast_upload’ => ‘images#create’, :via => [:post]

    controller:
    @image = Image.new(params[:image])

    Model
    http://pastie.org/1368633

    This uploads through the ruby server and not the nginx module.

  11. Gautam Rege says:

    Just a curious check — I hope the form is being posted to “/images/create/fast_upload” and not to images_path.

    Would also recommend you add proxy_pass http://127.0.0.1; to the location.

  12. chief says:

    true, :method => :post do %>

    Description

    “30×15″ %>

    Started POST “/images/fast_upload” for 68.122.69.57 at Wed Dec 15 06:59:47 +0000 2010
    Processing by ImagesController#create as HTML
    Parameters: {“upload”=>{“fast_asset”=>{“original_name”=>”cessna.jpg”, “content_type”=>”image/jpeg”, “filepath”=> /path/shared/uploads_tmp/9/0032387699″}}, “Location”=>””, “authenticity_token”=>”DFOrANYgXhk0rNsT3ou6AFLCsDnEbwHiYtjPnf00+B8=”, “Description”=>””}
    [paperclip] Saving attachments.

    Rails creates a new record, but all the parameters I input are nil.

  13. chief says:

    new.html.erb

    true, :method => :post do %>

    .
    .
    .
    “30×15″ %>

  14. Gautam Rege says:

    @chief — you file is getting uploaded by nginx upload module — otherwise the fast_asset fields would not have been there.

    Add the fields you want to be ‘passed through’ in your upload_pass_form_field and you should get the params as you entered in your form.

    I saw the pastie and if you keep a standard prefix (or suffix) to your field names you could simply add that as part of the regex for upload_pass_form_field

  15. Pingback: Pro tip: Increase webpage performance using sprite, data-uri and jammit « Josh Software – Where Programming is an Art!

  16. karl says:

    How do the nonfile parameters get passed? Looking at the image I also see a title which is probably a string and a description, probably text data type, on the form. Are the data in these fields uploaded by nginx also?

    • Gautam Rege says:

      Karl,
      Nginx does not add any more fields. These extra fields (from the images) are passed via upload_pass_form_field configuration.

      In the nginx.conf above, you can configure the regex for upload_pass_form_field and allow custom fields to be passed-thru.

      In the webportal above, we have this config:

      upload_pass_form_field “^X-Progress-ID$|^contest_id$|^ad|^video|^authenticity_token$|^format$”;

      So all form fields starting with ‘ad’ or ‘video’ like ‘ad[title]‘, ‘video[tags]‘ etc. gets passed.

  17. @Gautam: This is a great write up, thanks for putting it together.

    Just wondering if you’ve noticed any issues with Firefox 3.6? On my local development machine, the progress bar works pretty well in both Safari and Chrome, updating quite smoothly every two seconds.

    But with Firefox, I’m lucky to see one update of the progress bar before it completes. I see this behaviour with Firefox 3.6 when using the same files I just tested uploading in Safari and Chrome and saw the progress bar working fine.

    • Gautam Rege says:

      Hi Gaelian,
      I have tested with FF 3.6 and its working fine. Can you check with Firebug -> Net and see what is the response of the /progress calls?

      Also, (of course) if there are any JS errors :) Or paste a link to a pastie — I can check it out.

      • Hi Gautam,

        Thanks for the reply.

        Funny you should mention Firebug, I spent some time trying to debug this issue using Firebug to view the XHR responses. They appeared to be happening but very slowly and sometimes not containing the correct response. The browser seemed to be straining under the effort, finally it became clear to me that the problem was in fact Firebug its self!

        On OS X (10.6.6) + Firefox 3.6.15 + Firebug 1.6.2, I tried uploading a ~446MB video file through the browser with a similar upload progress set up to what you’ve written here. The Firebug window doesn’t even have to be open and the browser almost freezes as the upload happens, JavaScript performance takes a massive nose dive and amongst other weirdness, the upload progress bar does not work properly because the XHR requests are not coming in and being processed properly.

        Disabled Firebug in the Firefox Add-ons menu and suddenly I can upload the same file, all the same code on the client and server, and the progress bar works fine. So I guess the moral of the story is beware Firebug when uploading large files.

      • Gautam Rege says:

        Woh! This is indeed very interesting. During development, we needed faster iterations, so we tested with only a 40MB file :)

        On production, we have tested large files but of course, we never started firebug!

        Thanks for the update – it’s going to be a major time saver for a lot of people reading this post!

  18. Hey Gautham,
    Great work, we found your tutorial particularly useful.
    Thanks!

    Only if upload module would have integrated directly with mogilefs module, that would be something.

  19. Kirk says:

    Hi Josh, thanks for writing such a great post on this topic. I am having a bit of trouble with my upload-progress-module tho… it seems I am getting a 405 Error returned from my ajax call to the upload progress.

    Do you have any idea why this might be?

    Instead of explaining everything in detail again, I have a stackoverflow question open if you would be so kind to take a look: http://stackoverflow.com/questions/12154776/why-wont-my-upload-limit-rate-for-nginx-work/12307111

    here is a pastie of my nginx config: http://pastie.org/4681229

    Either way, thanks for a great blog post

    Kirk

  20. Pingback: Compile Nginx on Mac OSX - nonocast

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s