How to isolate your Rails blobs in subfolders

#rails #activestorage #heroku

Ruby on Rails has a very nice abstraction for storing user-provided assets and images to any cloud (AWS S3, Google, Azure, local files etc) called Active Storage. If you are spinning up new deployments of your application with each branch/pull request, you will want the uploaded assets to go nicely into their own subfolders. Much easier to clean them up later. Unfortunately this is not possible out of the box.

Fortunately, with Ruby, we can fix anything.

Firstly, we need a solution that does not alienate all existing blobs. We only want to put new blobs into subfolders. Existing blobs can stay exactly where they are.

I've deployed two demonstration apps that show the internal key to demonstrate that the images are stored in different subfolders:

The new files in each app are now being stored in subfolders, whilst legacy files are kept in their original file in the root of the bucket:

$ aws s3 ls s3://mybucket/
                           PRE subfolder-demo-1111/
                           PRE subfolder-demo-2222/
2021-06-11 12:08:16    1579488 g7643rle6b6bgn38v16734062wqw
2021-06-11 13:50:33    5034683 ki79m2n0p5ib3hg8l6bf1z7yrclt

In the sample solution below, I'm going to prefix all uploaded blobs with the name of the deployment. On Heroku this is available at runtime from the environment variable $HEROKU_APP_NAME after you turn on dyno metadata.

Each user-provided file is stored in your blobstore and registered in your database with an ActiveStorage::Blob. The ActiveStorage::Blob#key value guarantees that two files uploaded with the same name will be stored with unique file names.

If we change #key to have different prefix for each deployment then our blobs will be stored in isolated subfolders.

To feel good about myself as a professional, let's write a test first.

# spec/lib/activestorage_blob_spec.rb
require "rails_helper"

RSpec.describe ActiveStorage::Blob do
  let(:blob) { ActiveStorage::Blob.create_and_upload! io: StringIO.new("This is a test file"), filename: "test.txt" }
  let(:key) { blob.key }

  it "keys have no special prefix by default" do
    expect(ENV).to receive(:[]).with("HEROKU_APP_NAME").and_return(nil)
    # default key looks like "nk2gxeujmuoldqr6o8ng6gov9g5c"
    expect(key).to match(%r{\w{28}})
  end

  it "keys have no special prefix by default" do
    expect(ENV).to receive(:[]).with("HEROKU_APP_NAME").and_return("rcrdcp-pr-123").twice
    # expect like "rcrdcp-pr-123/nk2gxeujmuoldqr6o8ng6gov9g5c"
    expect(key).to match(%r{rcrdcp-pr-123/\w{28}})
  end
end

If the application is not running on Heroku, then keep using the default #key — a 28-character long random string.

If the app is running on Heroku, then put the app name at the front of the string, e.g. myapp-pr-123/nk2gxeujmuoldqr6o8ng6gov9g5c.

Currently, the #key value is created indirectly by has_secure_token :key before the creation of an ActiveStorage::Blob.

Our solution will be to re-create this key value with an additional before_create call.

# config/initializers/active_storage.rb

Rails.configuration.to_prepare do
  ActiveStorage::Blob.class_eval do
    before_create :generate_key_with_prefix

    def generate_key_with_prefix
      self.key = if prefix
        File.join prefix, self.class.generate_unique_secure_token
      else
        self.class.generate_unique_secure_token
      end
    end

    def prefix
      ENV["HEROKU_APP_NAME"]
    end
  end
end

You can use any environment variable that differentiates your deployments.

On Heroku, to access the $HEROKU_APP_NAME variable you need to turn on dyno metadata and deploy the app again.

heroku labs:enable runtime-dyno-metadata

Thanks to Arian Faurtosh for the starting point for this solution https://github.com/rails/rails/issues/32790#issuecomment-844095704. I found I needed to take a different route due to the behaviour of has_secure_token which might a recent change.