Imagine a situation where you need to log every attempt at accessing your application regardless of how many application servers are deployed and when. Text logs aren’t enough and can be difficult to aggregate across multiple disparate filesystems. Not to mention, retrieval of any given text-based record (or series thereof) after several years can be a significant undertaking. So you decide to log that information in your application database. But how do you scale the data storage for logs independently of data storage for your application?
We encountered this situation recently while working on a medical API to be consumed by a custom built mobile application. Targeted for deployment in the United States, the law known as HIPAA – the Health Insurance Portability and Accountability Act – will require us to log all attempts to access PHI, or protected health information.
To independently scale those logs from the application database, we took the interesting approach of using Rails 6’s multiple databases feature to set up a separate database for the application logs, allowing us to control, monitor and scale access request storage independently from the healthcare data that the application is responsible for storing and serving.
This article is a guide in how to set up your application to do the same thing. We’ll explore the application configuration needed, database migrations, and application code to make use of this feature.
Disclaimer: This is NOT a guide on HIPAA compliance and it should absolutely NOT be assumed that logging access requests is all you have to do to be compliant with the law. Don’t use this as a guide on compliance for HIPAA or any other mandate. Always get real guidance from an expert who knows the law well.
Configuration
The first step in utilizing Rails 6’s multiple database support is to modify your config/database.yml to designate a primary database, and then add a second named database configuration. That second database will have a parameter telling Active Record where to find database migrations, and one telling it the name of your database. This guide assumes both databases are on the same host for simplicity, but once you’ve seen the configuration, you should easily be able to extrapolate for a multi-host setup.
default: &default
adapter: postgresql
host: localhost
development:
primary:
<<: *default
database: yourapp_dev
access_logs:
<<: *default
database: yourapp_access_logs_development
migrations_paths: db/access_logs_migrate
test:
primary:
<<: *default
database: yourapp_test
access_logs:
<<: *default
database: yourapp_access_logs_test
migrations_paths: db/access_logs_migrate
production:
primary:
<<: *default
database: yourapp_prod
access_logs:
<<: *default
database: yourapp_access_logs_production
migrations_paths: db/access_logs_migrate
Note that the parameter to specify the location of database migrations is called “migration paths”, plural, even though we only have one here. That particular idiosyncrasy gave me no small amount of frustration while trying to figure out why my Rails app wasn’t happy with my database.yml.
Migrations
Now create a new directory under db, and call it access_logs_migrate. Create a new migration file there with the following in it:
class CreateAccessLogs < ActiveRecord::Migration[6.0]
def change
# First set up UUID capabilities for PostgreSQL
enable_extension('pgcrypto')
enable_extension('uuid-ossp')
create_table :access_logs, id: :uuid do |t|
t.string :ip
t.string :request_uri
t.string :request_method
t.bigint :profile_id
t.bigint :current_user_id
t.timestamps
end
end
end
As you can see, there’s nothing special about this migration; we designated which database will have this table created in it by placing it under the directory we configured in database.yml.
Here we’re creating a table with UUIDs for a primary key (because you could conceivably run into the top limits of bigint otherwise and let me tell you, that’s not fun) and logging IP address, the requested resource and method, and in this case I’m also logging, if available, a profile ID and if the current user is logged in, their ID as well. Those last two params are optional and will likely vary based on your application.
From now on, when you run migrations, ActiveRecord will run migrations in both the normal location and now in your new directory that you just created. Calling rake db:migrate is all you need to do from here to get that table created.
Application Code
The general idea here is that we’re going to use a standard Rails model to log access requests in a before_action in application_controller.rb. We decided to put it here since the vast majority of requests need to be logged, and we can use skip_before_action to save some space on the few access requests that don’t need to be logged, like requests for options that a mobile application would use to generate a select list, for example.
Before you create a model, let’s create a class that it can inherit from to access the new database:
# app/models/access_log_base.rb
class AccessLogBase < ApplicationRecord
self.abstract_class = true
connects_to database: { writing: :access_logs, reading: :access_logs }
end
Now you can create a model that inherits from this class:
# app/models/access_log.rb
class AccessLog < AccessLogBase
validates :ip, presence: true
validates :request_uri, presence: true
validates :request_method, presence: true
def subject
Profile.find(profile_id)
end
end
In our use case, we created a subject method that allows us to locate the “subject” of a given access log – in other words, who the potential target was for any given request. This may be useful later on if we’re ever the subject of an audit or have to conduct an investigation for some reason.
Finally, in app/controllers/application_controller.rb:
# app/controllers/application_controller.rb
class ApplicationController < ActionController::API
before_action :log_access_request
private
def log_access_request
AccessLog.create(
ip: request.ip,
request_uri: request.url,
request_method: request.method,
profile_id: params[:profile_id],
current_user_id: current_user&.id
)
end
end
Since every controller in our application inherits from ApplicationController, our access requests will now automatically be logged on every request. But you may not always want that; in those cases, just skip_before_action :log_access_request in your controller of choice. In our case, it was less common that we’d not want it logged than it was that we would, which is why we made it automatic.
Wait, before action? Why not after for faster responses?
You may be wondering if making the log an after_action might speed up response times. While theoretically possible, we opted not to do this because in our case, we have to log every request by law. We can’t risk there being a situation where the information is served, but no record of that was recorded because to do so is a violation of HIPAA. In other words, if we can’t record the access log, we can’t serve it either, and having this as a before_action prevents that possibility because if recording the log fails, the rest of the request will fail too, meaning that PHI will never be accessed or otherwise served. This is a choice we’ve made to protect our client, and while it’s possible there could be performance implications in some circumstances, those issues are manageable as long as database storage availability and query response times are monitored and addressed accordingly.
Using multiple databases in your application used to involve quite a bit of extra work, or extra libraries, but now it’s quite simple and easy to use. While your use case will likely differ from ours, being able to independently scale your databases by splitting model data across multiple locations can have benefits that, thanks to Active Record, are now more easily accessible than ever.
Josh Software 