Uploading files ‘efficiently’ has also been a pain point in most apps. I finally got to experiment with:
- nginx upload module (http://www.grid.net.ru/nginx/upload.en.html)
- nginx upload progress module http://wiki.nginx.org/NginxHttpUploadProgressModule) and
- upload multiple files via paperclip.
There are resources available for handing these individually but not together. I used references from here and here. It is advisable to read these 2 links before proceeding. The nuances of the directives are explained there and I don’t want to repeat it.
Nginx installation via passenger
When you are using the passenger-install-nginx-module, make sure you use the customize build with at least the following custom arguments:
--add-module=<path>/nginx_upload_module-2.2.0/ --add-module=<path>/masterzen-nginx-upload-progress-module-ac62a29/
FYI, I downloaded v2.2.0 for nginx upload module and v.0.7 for nginx upload progress module. Nginx upload module ensures that the file is uploaded by nginx and stored in a temporary location and the nginx upload progress module tracks this download and provides progress reports for it. Its important to remember that both can be used independently but its more fun to use them together!
Here is my nginx.conf:
http {
...
upload_progress proxied 1m;
server {
listen 80;
server_name zooppa;
root /Users/gautam/work/showmetv/public;
# Match this location for the upload module
location /videos/fast_upload {
proxy_pass http://127.0.0.1;
upload_pass @fast_upload_endpoint;
upload_store /Users/gautam/work/showmetv/tmp/upload_store 1;
# set permissions on the uploaded files
upload_store_access user:rw group:rw all:r;
# Set specified fields in request body
# this puts the original filename, new path+filename and content type in the requests params
upload_set_form_field fast_asset[][original_name] "$upload_file_name";
upload_set_form_field fast_asset[][content_type] "$upload_content_type";
upload_set_form_field fast_asset[][filepath] "$upload_tmp_path";
upload_pass_form_field "^X-Progress-ID$|^authenticity_token$";
upload_cleanup 400 404 499 500-505;
track_uploads proxied 30s;
}
location ^~ /progress {
# report uploads tracked in the 'proxied' zone
upload_progress_json_output;
report_uploads proxied;
}
location @fast_upload_endpoint {
passenger_enabled on;
rails_env development;
}
location / {
rails_env development;
passenger_enabled on;
}
}
Some important aspects of the above code which require some explanation:
upload_set_form_field upload[fast_asset][][original_name] "$upload_file_name";
Here the parameter being passed to the controller is
Parameters: { "action" => create,
"fast_asset"=>[{"content_type"=>"application/octet-stream",
"original_name"=>"1.flv",
"filepath"=>"/Users/gautam/work/showmetv/tmp/upload_store/5/0025383985"},
{"content_type"=>"image/jpeg", "original_name"=>"1_5575099.jpg",
"filepath"=>"/Users/gautam/work/showmetv/tmp/upload_store/6/0025383986"}],
"controller"=>"videos", "X-Progress-ID"=>"de03d98d4408e2f0d9b5eb34dd82322c" }
Notice that the upload[fast_asset] is an Array of Hashes for each file uploaded. Be sure to set X-Progress-ID in the list of fields to be passed through and ensure that track_uploads is in the same location which is uploading the files. Here is the HTML I used:
...
<head>
<%= javascript_include_tag 'plugins/jquery.uploadProgress' %>
<%= stylesheet_link_tag 'upload' %>
</head>
...
<script type="text/javascript">
$(function() {
$('form').uploadProgress({
/* scripts locations for safari */
jqueryPath: "/javascripts/jquery.js",
uploadProgressPath: "/javascripts/plugins/jquery.uploadProgress.js",
/* function called each time bar is updated */
uploading: function(upload) {$('#percents').html(upload.percents+'%');},
/* selector or element that will be updated */
progressBar: "#progressbar",
/* progress reports url */
progressUrl: "/progress",
/* how often will bar be updated */
interval: 2000
});
});
</script>
...
<%= form_tag video_fastupload_path, :multipart => true, :method => :post, :id => 'upload' %>
<dl>
<dt for="firstname">Video</dt>
<dd>
<fieldset>
<%= file_field_tag :video_file %>
</fieldset>
</dd>
<dt for="video_image">Video Image</dt>
<dd>
<fieldset>
<%= file_field_tag :video_image %>
</fieldset></dd>
</dd>
</form>
<div id="uploading" style="float:left;">
<div id="progress">
<div id="progressbar"> </div>
<div id="percents"></div>
</div>
</div>
Note that the file_field name is irrelevant. A POST request with file fields which matches the location for nginx upload module would replace the file fields with its own — in the case above with upload[fast_asset] array. The above code ensures that the file is uploaded and stored in the temporary location via the nginx upload module and it is tracked by the nginx upload progress module. I have used the jquery.uploadProgress.js from here – its simple and sweet.
Its also important to know how to actually transfer the uploaded files using paperclip. Its simple enough:
class Video < ActiveRecord::Base
# I stored files in CloudFiles but you could use S3 buckets.
has_attached_file :video,
:storage => :cloud_files,
:cloudfiles_credentials => "#{RAILS_ROOT}/config/rackspace_cloudfiles.yml",
:container => "#{ApplicationConfig['cloudfiles']['videos']['container']}"
has_attached_file :image,
:styles => { :small_thumb => [ "50x50", :jpg ],
:large_thumb => [ "100x100", :jpg ],
:medium_thumb => [ "170x126", :jpg ],
:detail_preview => [ "608x336", :jpg ]
}
def fast_asset=(files)
if files && files.respond_to?('[]')
files.each do |item|
self.tmp_upload_dir = "#{item['filepath']}_1"
tmp_file_path = "#{self.tmp_upload_dir}/#{item['original_name']}"
FileUtils.mkdir_p(self.tmp_upload_dir)
FileUtils.mv(item['filepath'], tmp_file_path)
if file.index(item) == 0
self.video = File.open(tmp_file_path)
else
self.image = File.open(tmp_file_path)
end
end
end
end
This is simplistic code to just save the uploaded files as the paperclip assets. Now another issue presents itself. We have uploaded the file to a temporary location and now want to transfer it to S3 or CloudFiles. However, the tracking continues to be in force till the request completes. After uploading to the nginx temporary store, we should update the paperclip assets asynchronously via delayed job. If this is not done, the HTTP upload request will complete after the files are uploaded to S3 or Cloudfiles, so the progress bar will show 100% (completed) and still wait till we upload to the remote store!
Using a delayed job to create this resource is a perfect solution – the user sees the progress bar complete and once the files are uploaded, the request is complete from the users perspective — the rest is server side processing.
The controller code is simpler:
class VideosController < ActionController::Base
def create
...
Delayed::Job.enqueue CreateVideo.new(params)
...
end
end
And the delayed_job worker:
class CreateVideo < Struct.new(:params)
def perform
...
video = Video.new(params[:video])
video.fast_asset = params[:fast_asset]
...
end
end
The performance — 10 fold! I found files uploading faster and keeping the Rails webserver on lesser resources.Here are a few images of uploading in progress. This is a 90MB file on a LAN – uploaded in 15 seconds. Earlier, with no nginx upload module and just direct upload to a paperclip asset, this took about 1.5 minutes.
Josh Software 
What does your routes file look like?
The routes file just requires a route for ‘/videos/fast_upload’! Eg.
map.video_fastupload ‘/videos/fast_upload’, :controller => ‘videos’, :action => ‘create’
The create method will get ALL the params, including the injected fast_asset.
Does the progress bar work for you in chrome? It doesn’t seem to work for me.
Also, what is your suggestion for updating (re-uploading) a file?
Jai,
Progress bar works fine in Chrome. Here are some things you may have not configured:
1. Check if the request is indeed reaching the server — the /progress?X-Progress-ID=. Check nginx access.log to see if you are seeing the request come in.
2. You can also use curl if do have the X-Progress-ID with you.
Does the upload succeed at all? If it does, it could even be the positioning of block in the nginx.conf. Do paste relevant information or code-snippet. It would help.
@Jai Regarding re-upload of a file, it depends on your Model not the upload progress bar. If you wanted to know about ‘Cancellation’ of a file upload in progress, its different.
I found that you can simply navigate away from the page (or reload the page) — nginx detects broken connection and aborts the upload process.
Hope this helps.
I used this for my routes:
match ‘/images/create/fast_upload’ => ‘images#create’, :via => [:post]
And in my controller :
@image = @resource.images.build(params[:image])
That gives me an error:
NoMethodError (undefined method `images’ for nil:NilClass):
app/controllers/images_controller.rb:58:in `create’
It throws this error at the very end of the upload. Also there is no difference in time in the upload with this configuration, so the upload module must not be taking the file.
Thanks..
@chief I don’t think the error you are getting is related to the upload progress bar — it seems that you have not initialized @resource.
Check your code – or paste the snippet of how @resource was created.
Started POST “/images” for …. at Sat Dec 11 07:26:49 +0000 2010
Processing by ImagesController#create as HTML
Parameters: {“commit”=>”Create Image”, “authenticity_token”=>”ConM/EvSKBRDSjPX8Z3AmvyYFJ7SkeMkOedEl5rz0R4=”, “utf8″=>”\342\234\223”, “image”=>{“photo”=>#<$
Completed in 86ms
@chief
It seems that your nginx rules may not be functioning at all – and the request is being processed by your webserver.
I am pretty sure you are not seeing a progress bar, as you dont have the X-Progress-ID in your parameters.
Check the nginx.conf file to see if the “location /images/create/fast_upload ” is correct.OR paste the contents of your nginx.conf here.
Cheers!
At this point I am just working on the upload module and not the progress module.
nginx.conf
http://pastie.org/1368622
My route is:
match ‘/images/create/fast_upload’ => ‘images#create’, :via => [:post]
controller:
@image = Image.new(params[:image])
Model
http://pastie.org/1368633
This uploads through the ruby server and not the nginx module.
Just a curious check — I hope the form is being posted to “/images/create/fast_upload” and not to images_path.
Would also recommend you add proxy_pass http://127.0.0.1; to the location.
true, :method => :post do %>
Description
“30×15” %>
Started POST “/images/fast_upload” for 68.122.69.57 at Wed Dec 15 06:59:47 +0000 2010
Processing by ImagesController#create as HTML
Parameters: {“upload”=>{“fast_asset”=>{“original_name”=>”cessna.jpg”, “content_type”=>”image/jpeg”, “filepath”=> /path/shared/uploads_tmp/9/0032387699”}}, “Location”=>””, “authenticity_token”=>”DFOrANYgXhk0rNsT3ou6AFLCsDnEbwHiYtjPnf00+B8=”, “Description”=>””}
[paperclip] Saving attachments.
Rails creates a new record, but all the parameters I input are nil.
new.html.erb
true, :method => :post do %>
.
.
.
“30×15” %>
new.html.erb
http://pastie.org/1378565
@chief — you file is getting uploaded by nginx upload module — otherwise the fast_asset fields would not have been there.
Add the fields you want to be ‘passed through’ in your upload_pass_form_field and you should get the params as you entered in your form.
I saw the pastie and if you keep a standard prefix (or suffix) to your field names you could simply add that as part of the regex for upload_pass_form_field
How do the nonfile parameters get passed? Looking at the image I also see a title which is probably a string and a description, probably text data type, on the form. Are the data in these fields uploaded by nginx also?
Karl,
Nginx does not add any more fields. These extra fields (from the images) are passed via upload_pass_form_field configuration.
In the nginx.conf above, you can configure the regex for upload_pass_form_field and allow custom fields to be passed-thru.
In the webportal above, we have this config:
upload_pass_form_field “^X-Progress-ID$|^contest_id$|^ad|^video|^authenticity_token$|^format$”;
So all form fields starting with ‘ad’ or ‘video’ like ‘ad[title]’, ‘video[tags]’ etc. gets passed.
@Gautam: This is a great write up, thanks for putting it together.
Just wondering if you’ve noticed any issues with Firefox 3.6? On my local development machine, the progress bar works pretty well in both Safari and Chrome, updating quite smoothly every two seconds.
But with Firefox, I’m lucky to see one update of the progress bar before it completes. I see this behaviour with Firefox 3.6 when using the same files I just tested uploading in Safari and Chrome and saw the progress bar working fine.
Hi Gaelian,
I have tested with FF 3.6 and its working fine. Can you check with Firebug -> Net and see what is the response of the /progress calls?
Also, (of course) if there are any JS errors 🙂 Or paste a link to a pastie — I can check it out.
Hi Gautam,
Thanks for the reply.
Funny you should mention Firebug, I spent some time trying to debug this issue using Firebug to view the XHR responses. They appeared to be happening but very slowly and sometimes not containing the correct response. The browser seemed to be straining under the effort, finally it became clear to me that the problem was in fact Firebug its self!
On OS X (10.6.6) + Firefox 3.6.15 + Firebug 1.6.2, I tried uploading a ~446MB video file through the browser with a similar upload progress set up to what you’ve written here. The Firebug window doesn’t even have to be open and the browser almost freezes as the upload happens, JavaScript performance takes a massive nose dive and amongst other weirdness, the upload progress bar does not work properly because the XHR requests are not coming in and being processed properly.
Disabled Firebug in the Firefox Add-ons menu and suddenly I can upload the same file, all the same code on the client and server, and the progress bar works fine. So I guess the moral of the story is beware Firebug when uploading large files.
Woh! This is indeed very interesting. During development, we needed faster iterations, so we tested with only a 40MB file 🙂
On production, we have tested large files but of course, we never started firebug!
Thanks for the update – it’s going to be a major time saver for a lot of people reading this post!
Hey Gautham,
Great work, we found your tutorial particularly useful.
Thanks!
Only if upload module would have integrated directly with mogilefs module, that would be something.
MogileFS looks interesting – never too late to give it a shot! 🙂
Hi Josh, thanks for writing such a great post on this topic. I am having a bit of trouble with my upload-progress-module tho… it seems I am getting a 405 Error returned from my ajax call to the upload progress.
Do you have any idea why this might be?
Instead of explaining everything in detail again, I have a stackoverflow question open if you would be so kind to take a look: http://stackoverflow.com/questions/12154776/why-wont-my-upload-limit-rate-for-nginx-work/12307111
here is a pastie of my nginx config: http://pastie.org/4681229
Either way, thanks for a great blog post
Kirk
@Kirk,
The nginx config seems correct. Can you past the error log – it would be in the nginx error log. This does seem to be a setup issue. I’ll try this config out and let you know.
I have since figured it out. Thanks for your response and great tutorial. It works perfectly now!
Kirk
what was the error
pretty much an ID-10-T error. Haha.. Everything was good in config but I had a typo in my rails code
🙂 cool. Glad this helped
Hi I have nginx 1.10 with nginx upload and upload progress bar module can you help me how to use with php