Imagine that you have a mobile application that needs to store images in Google Cloud Storage. Instead of sending these from your mobile device to your API for storage, it’s more efficient to have your mobile application save these files to cloud storage directly. But how might you do that?
The answer: Signed URLs – a URL with time-limited authentication embedded within it.
Google Cloud Storage supports signed URLs, and this guide explains how one might build an application to make use of that feature.
What’s a Signed URL Look Like?
In short, a signed URL may look something like this:
Basically it’s a URL pointing to the resource on Google Cloud Storage with URL encoded variables detailing the encryption algorithm used for the authentication key, expiration time, and so on. Google maintains libraries in a number of different languages that help you generate these URLs, and has instructions on how to generate a URL in your language of choice manually if you don’t have a library available to do it for you.
Getting Started: Service Accounts
To have an application make use of Google Cloud services (including storage), you need to have what’s called a “service account”. This represents the authenticated entity performing various actions on behalf of your program.
Creating a service account is pretty easy – just head to console.cloud.google.com, click the hamburger icon on the left, then hover over “IAM & Admin”, then click “Service Accounts”.
(Screenshot: Finding the Service Accounts Option)
From here, you’ll create a new service account and assign it, for the purposes of this guide and for simplicity’s sake, assign it the “Cloud Storage -> Storage Admin” permission group. This will grant it all the permissions it needs to act on behalf of your application to list buckets, list files, create and delete things, etc.
(Screenshot: Cloud Storage, Storage Admin Group)
Next, in step 3 you can just click “continue” (no need to worry about this part). After that, click the name of the account you just created in the list given, and we’ll need to create a key for it. This is an authentication key file that must be kept out of source control and treated as authentication credentials (sort of like an SSH key). Treat it with care as once you create it, you’ll never be able to download it a second time; instead you’ll have to delete it and create a new key all over again (it is probably not a bad idea to do this from time to time anyway to rotate old keys, just in case).
To generate a key, after clicking the account in the list, you’ll see a screen with information about that account, and if you scroll down you’ll see the option for “add key”.
(Screenshot: What you see after clicking “Add Key”)
Instead of importing a key, we’ll have Google create one for us. Select “JSON” as the key type and click “Create”. After that you’ll get a JSON file downloaded via your browser. Save this file somewhere safe, and not in source control! This is the file that tells your program all the authentication credentials it will need to make use of that service account we just created.
Finally, for the sake of simplicity, we’re going to create a new bucket in the dashboard instead of through code. While it’s possible to do this via code, you only need to do it once and it’s easier to just create it while we’re here. From the hamburger menu, choose “Storage -> Browser” and create a new bucket. Call it whatever you like; in this article we’ll just call it “example-bucket”.
That’s all the configuration needed on Google’s end; from here on, we’ll be developing a simple Ruby script that will create signed URLs, and using cURL on the command line to interact with those URLs.
A Simple Ruby Program to Create Signed URLs
Google maintains a library in Ruby (and has libraries for other languages) to help you create signed URLs much more easily than it would be without it. To see a few examples for the libraries available, check out Google’s documentation here. We’ll make use of these libraries in the Ruby program we’re going to write in this article.
First, we need to set an environment variable that the Ruby gem is going to automatically use to pick up the authentication credentials needed to operate on your account. This is how we tell the library what service account to use – the one we created above.
Start by exporting, in your shell, an environment variable:
export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/my-key.json"
The value of this variable needs to be the path to wherever you saved the JSON key you received when you created a key in the steps above.
Next, install the ‘google-cloud-storage’ gem either through Rubygems or Bundler, then create a blank file, called sign.rb, wherever you like (or in the same directory as your Gemfile). Here’s the contents of sign.rb:
#!/usr/bin/env ruby
# frozen_string_literal: true
require 'google/cloud/storage'
bucket_name = ‘example-bucket’
file_name = 'my_file.jpg' # This is what it will be named in the bucket
exp = 20 * 60 # 20 minutes
# Instantiating a new Google::Cloud::Storage object will automatically read your
# environment variables for GOOGLE_APPLICATION_CREDENTIALS, find the file, and
# load it up to connect with Google Cloud Storage as your service account.
storage = Google::Cloud::Storage.new
# Here we create a URL to *save* or upload a file using HTTP PUT. If you wanted
# to view a file in a non-public bucket, you'd use GET and leave out the
# headers part here.
url = storage.signed_url bucket_name, file_name, method: 'PUT', expires: exp,
version: :v4, headers: { 'Content-Type' => 'image/jpg' }
puts "Upload your file to:"
puts url
puts "To view your file after uploading:"
url = storage.signed_url bucket_name, file_name, method: 'GET', expires: exp,
version: :v4
puts url
(This code is available as a GitHub Gist here.)
If you run this program, you’ll see output similar to the following:
(Screenshot: Example output from sign.rb.)
So let’s break down the code above. We start by requiring the Google Cloud Storage library, then instantiating a new object from it. This is where the library automatically looks for the environment variable we set earlier and loads authentication credentials. This is why we need that environment variable set before we get to this stage.
Next, we create two URLs – one with an HTTP PUT, and one with HTTP GET. The first is for uploading the file for storage, and the second is for viewing it when we’re done.
Uploading the File
Now, let’s actually upload a file using cURL from the command line:
curl -X PUT -H 'Content-Type: image/jpg' --upload-file /path/to/your/file.jpg https://storage.googleapis.com/example-bucket/my_file.jpg\?X-Goog-Algorithm\=GOOG4-RSA-SHA256\&X-Goog-Credential\=dev-storage%40whatever.iam.gserviceaccount.com%2F20200904%2Fauto%2Fstorage%2Fgoog4_request\&X-Goog-Date\=20200904T104730Z\&X-Goog-Expires\=1200\&X-Goog-SignedHeaders\=content-type%3Bhost\&X-Goog-Signature\=4c24eb3b517802534ce321c70e5a4c0424e01012681997eb096b305d71c76939dcb8e195de9bf5e311c74231fcbd9c4034225ab0a08fce3476f91cbe6f7473e4f102a4042da4f7caf90d474d535ceaa29b23477e1ece2a50675538fb6572f09f47da2c35dbdee154239f1ea600efcf4f36edc8ed16fd0f11ebc2eebf80b3ebca9e8f28204608122f33838bf3acffdf8bf4772bf519b73a5c6b6e8eba4b6c596ee791b1a023e60c0a00120b46de712a7f7b0821b35f13dd6da304b49b2f4b59968081d615f3944aa8edce17094b5415e7e2bbfb2811d1e4ba209364452399747eac18b27cdb987fe541505187d5d9a63070eb02c1ff9db80a265afb9eec52193a
You won’t get anything back when the upload is finished – the terminal will just give you a prompt as normal. But rest assured, the image has been uploaded.
Viewing the File
Now you copy and paste the second URL into your browser to see the uploaded image.
(Screenshot: Showing the uploaded image)
And there you have it! A simple and useful example of how to use Google Cloud Storage Signed URLs.
Summary:
Here, we explained the detailed process of using signed URLs in google cloud storage. Also, the code for this article is available as a public GitHub Gist.
The code for this article is available as a public GitHub Gist.
Josh Software 