Louis Sommer

Posted on Jan 16, 2023 • Edited on Jan 18, 2023

Building the Ultimate Search for Rails - Episode 1

#rails #stimulusreflex #search #facets

During summer 2021, I got lucky enough to cross Twitter paths with Peter Szinek, who introduced me to his team and got me hired at RCRDSHP, Obie Fernandez’s latest Web 3.0 project involving Music NFTs. Being myself a pro musician and music producer, I was thrilled to finally be able to mix my two passions-turned-into-a-living. Surrounded by awesome developers, I learned and built tons of cool stuff, among which a reactive, super performant server-side-rendered search experience using StimulusReflex, ElasticSearch and close to no JavaScript. The feature is still live and pretty much unchanged.

The purpose of this series will be to first reimplement this super friendly UX with basic filters and sort options, along with StimulusReflex. Then, we’ll see how ElasticSearch can allow more complex filters and search scenarios, while improving performance. In the last episode, we’ll try and replace StimulusReflex with the new Custom Turbo Stream Actions and compare implementation/behaviour. If time allows, I might add a bonus episode to show how to deploy all this in production. Let’s dig in.

What are we building?

Remember how back in the day, people used to collect art printed on actual paper ? A bit like NFTs, only physical. Weird, right? Well, let’s picture an app that would allow users to buy and sell limited edition art prints. The prints would include photographs, movie posters, and illustrations of various formats. It should look like that:

Here’s what the DB looks like:

Please note the tags column is of string array type. One might argue that the Listing table, in our case, could easily be skipped. But for the sake of keeping a real-world complexity scenario, let’s say that we’d like to keep the actual Prints separate from their listings (and since the app allows users to sell their prints, we might reasonably think that a print could be listed several times).

OK. Show me the gear

First things first: on the frontend, we’ll use StimulusReflex (a.k.a SR) to build a super reactive and friendly search experience with very little code, and little to no JavaScript. For those unfamiliar:

StimulusReflex is a library that extends the capabilities of both Rails and Stimulus by intercepting user interactions and passing them to Rails over real-time websockets. The current page is quickly re-rendered and morphed to reflect the new application state.

Sounds a bit like Hotwire on paper, though you’ll see how their philosophy greatly differs in the last episode of this series. We’ll also use a sprinkle of CableReady, a close cousin of SR.

On the backend, we'll need a few tools. Apart from the classics (ActiveRecord scopes and the pg_search gem), you’ll see how the (yet officially unreleased but production-tested) all_futures gem, built by SR authors, will act as an ideal ephemeral object to temporarily store our filter params and host our search logic. Finally, we’ll use pagy for pagination duties.

Philtre d'amour

(Please indulge this shitty French pun, the expression philtre d'amour meaning love potion but also sounds like beloved filter)

Let’s start by creating some simple data. We’ll add a few artworks of different kind : photographs, illustrations, and posters. Each will have several tags from a given list and an author. For now, let’s just generate one print per artwork, and a listing for each. Prints can be one of 3 available formats , while listings will be of varying price.
Now let’s list our different features:

Search by name or author
Filter by minimum price
Filter by maximum price
Filter by category
Filter by print format
Filter by tags
Order by price
Order by date listed

Before I started building my feature, my former colleague and friend @marcoroth pointed me to leastbad’s Beast Mode, from which I took heavy inspiration to get going. That’s how I discovered his gem all_futures, which provides us with an ActiveRecord-like object that will persist to Redis. Let’s see how things look like.

# app/controllers/listings_controller.rb
class ListingsController < ApplicationController
  def index
    @filter ||= ListingFilter.create
    @listings = @filter.results
  end
end

# app/models/listing_filter.rb
class ListingFilter < AllFutures::Base
  # Filters
  attribute :query, :string
  attribute :min_price, :integer, default: 1
  attribute :max_price, :integer, default: 1000
  attribute :category, :string, array: true, default: []
  attribute :tags, :string, array: true, default: []
  attribute :format, :string, array: true, default: []
  # Sorting
  attribute :order, :string, default: "created_at"
  attribute :direction, :string, default: "desc"

  def results
    # TODO: Build a query out of these attributes
  end
end

Notice how params are absent from the controller, and nothing gets passed to our ListingFilter object? And how come @filter could potentially be already defined? You’ll see why in a bit, so let’s first look at building the query.

In his approach, @leastbad simply created an ActiveRecord scope for each filter, then very cleverly and neatly, chained them to build his final filtered query, much like this:

# In app/models/listing_filter.rb
def results
  Listing.for_sale
    .price_between(min_price, max_price)
    .from_categories(category)
    .with_tags(tags)
    .with_formats(format)
    .search(query)
    .order(order => direction)
end

You might wonder: “But what if filters are empty and arguments blank? The chain’s gonna break!”. Well, have a look at the scopes declaration:

# app/models/listing.rb
class Listing < ApplicationRecord
  belongs_to :print

  scope :for_sale,        ->{ where(sold_at: nil) }
  scope :price_between,   ->(min, max) { where(price: min..max) }
  scope :with_formats,    ->(format_options) { joins(:print).where(prints: {format: format_options}) if format_options.present? }
  scope :from_categories, ->(cat_options) { joins(:artwork).where(artworks: {category: cat_options}) if cat_options.present? }
  scope :with_tags,       ->(options) { joins(:artwork).where("artworks.tags && ?", "{#{options.join(",")}}") if options.present? }
  scope :search           ->(query) { # TODO } 
end

In ListingFilter, the crucial bit is to make sure that every attribute has a default value. The magic then occurs in the if statement at the end of the scopes expecting an argument: if the lambda returns nil, then it will essentially be ignored, and the collection returned as is. Such a nice trick. Time for some specs to ensure that things actually work:

RSpec.describe ListingFilter, type: :model do
  let!(:photo) { Artwork.create(name: "Dogs", author: "Elliott Erwitt", year: 1962, tags: %w[Animals B&W USA], category: "photography") }
  let!(:poster) { Artwork.create(name: "Fargo", author: "Matt Taylor", year: 2021, tags: %w[Cinema USA], category: "poster") }
  let!(:photo_print) { photo.prints.create(format: "30x40", serial_number: 1) }
  let!(:photo_print_2) { photo.prints.create(format: "18x24", serial_number: 200) }
  let!(:poster_print) { poster.prints.create(format: "40x50", serial_number: 99) }
  let!(:photo_listing) { photo_print.listings.create(price: 800) }
  let!(:photo_listing_2) { photo_print_2.listings.create(price: 400) }
  let!(:poster_listing) { poster_print.listings.create(price: 200) }
  let!(:sold_listing) { poster_print.listings.create(price: 300, sold_at: 2.days.ago) }

  describe "#results" do

    it "doesn't return a sold listing" do
      expect(ListingFilter.create.results).not_to include(sold_listing)
    end

    context "Filter options" do
      it "Filters by price" do
        filter = ListingFilter.create(min_price: 100, max_price: 300)
        expect(filter.results).to match_array([poster_listing])
        filter.update(max_price: 1000)
        expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
      end

      it "Filters by format" do
        filter = ListingFilter.create(format: ["40x50"])
        expect(filter.results).to match_array([poster_listing])
        filter.update(format: ["40x50", "30x40"])
        expect(filter.results).to match_array([photo_listing, poster_listing])
      end

      it "Filters by category" do
        filter = ListingFilter.create(category: ["photography"])
        expect(filter.results).to match_array([photo_listing_2, photo_listing])
        filter.update(category: ["photography", "poster"])
        expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
      end

      it "Filters by tags" do
        filter = ListingFilter.create(tags: ["Cinema"])
        expect(filter.results).to match_array([poster_listing])
        filter.update(tags: ["Cinema", "Animals"])
        expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
      end

      it "Filters by multiple attributes" do
        filter = ListingFilter.create(tags: ["Cinema"], max_price: 300, category: ["poster"])
        expect(filter.results).to match_array([poster_listing])
      end
    end
  end
end

All green. I hope you’ll appreciate how easy it is to test this. Let’s quickly add pg_search to our Gemfile, then take care of the search scope:

# app/models/listing.rb
class Listing < ApplicationRecord
  include PgSearch::Model

  belongs_to :print
  has_one :artwork, through: :print

  scope :search, ->(query) { basic_search(query) if query.present? }
  # Skipping the other scopes...

  pg_search_scope :basic_search,
    associated_against: {
      artwork: [:name, :author]
    },
    using: {
      tsearch: {prefix: true}
    }
  #...
end

Since our listings don’t carry much information, we’ll have to jump a few tables to look where we need, namely the name and author columns of our Artwork model. Unfortunately, pg_search doesn’t support associated queries further than 1 table away, thus the has_one... through relationship we needed to add. Let’s add some tests for the search:

context "Search" do
  it "renders all listings if no query is passed" do
    filter = ListingFilter.create(query: "")
    expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
  end

  it "can search by artwork name or author" do
    filter = ListingFilter.create(query: "Erwitt")
    expect(filter.results).to match_array([photo_listing, photo_listing_2])
    filter.update(query: "Fargo")
    expect(filter.results).to match_array([poster_listing])
  end

  it "can both search and sort" do
    filter = ListingFilter.create(query: "Erwitt", order_by: "price", direction: "asc")
    expect(filter.results.to_a).to eq([photo_listing_2, photo_listing])
    filter.update(query: "Erwitt", order_by: "price", direction: "desc")
    expect(filter.results.to_a).to eq([photo_listing, photo_listing_2])
  end
end

We run the tests and of course, everything is gr… Oh no. Looks like the last test is acting up:

Apparently, a known problem of pg_search is that it doesn’t play well with eager loading, nor combinations of join and where queries. The recommended workaround (and my usual plan B when ActiveRecord queries start to get ugly) is to use a subquery:

# In app/models/listing_filter.rb
def results
  filtered_listings_ids = Listing.for_sale
    .price_between(min_price, max_price)
    .from_categories(category)
    .with_tags(tags)
    .with_formats(format)
    .pluck(:id)

  Listing.where(id: filtered_listings_ids)
    .search(query)
    .order(order_by => direction)
    .limit(200)
end

Let’s also add some last specs for the sort options and run all this.

context "Sort options" do
  specify "Recent listings first (default behaviour)" do
    filter = ListingFilter.create
    expect(filter.results.to_a).to eq([poster_listing, photo_listing_2, photo_listing])
  end

  specify "Most expensive first" do
    filter = ListingFilter.create(order_by: "price", direction: "desc")
    expect(filter.results.to_a).to eq([photo_listing, photo_listing_2, poster_listing])
  end

  specify "Least expensive first" do
    filter = ListingFilter.create(order_by: "price", direction: "asc")
    expect(filter.results.to_a).to eq([poster_listing, photo_listing_2, photo_listing])
  end
end

Everything’s green… Except for the search and filter option. The error’s gone, but the test still fails; the ordering doesn’t seem to work, despite all the sorting tests being green. After another lookup on pg_search known issues, it appears that order statements following the search scope don’t work. Workarounds include using reorder instead, or moving the order clause up the chain. I opted for the first option, which make all tests pass. Let's move on.

Stairway to Heaven

Now that we know that our backend is working as it should, let’s wire up our stuff. I’m gonna skip on Stimulus Reflex setup and configuration and dive right in. You can easily follow the official setup or, if you use import-maps, follow @julianrubisch’s article on the topic. I also know that leastbad has been working on an automatic installer that detects your configuration and sets everything up for you if you care to try it before the next version of SR gets released.

Once you’re done with that, let’s begin with the sort first. Let’s recap our sorting options and store them somewhere:

class ListingFilter < AllFutures::Base
  SORTING_OPTIONS = [
    {column: "created_at", direction: "desc", text: "Recently added"},
    {column: "price", direction: "asc", text: "Price: Low to High"},
    {column: "price", direction: "desc", text: "Price: High to Low"}
  ]
  #...
  attribute :order_by, :string, default: "created_at"
  attribute :direction, :string, default: "desc"
  #...

  # Memoizing the value to avoid re-computing at every call
  def selected_sorting_option
    @_selected_option ||= SORTING_OPTIONS.find {|option| order_by == option[:column] && direction == option[:direction] }
  end
end

Then in our “Sort by” dropdown, we’ll have something like :

<div class="dropdown">
  <button>
    Sort by:<span><%= @filter.selected_sorting_option[:text] %></span>
  </button>
  <!-- Skipping lots of HTML -->
  <% ListingFilter::SORTING_OPTIONS.each do |option| %>
    <% if option == @filter.selected_sorting_option %>
      <span class="font-semi-bold ..."><%= option[:text] %></span>
    <% else %>
      <a data-reflex="click->Listing#sort"
        data-column="<%= option[:column] %>"
        data-direction="<%= option[:direction] %>"
        data-filter-id="<%= @filter.id %>"
        href="#"
      >
        <%= option[:text] %>
      </a>
    <% end %>
  <% end %>
</div>

Even if you're unfamiliar with StimulusReflex, it should still remind you of the way we invoke regular stimulus controllers. Only here, when our link gets clicked, it should trigger the sort action (a ruby method) from the Listing reflex (a ruby class). Let’s code it:

# app/reflexes/listing_reflex.rb
class ListingReflex < ApplicationReflex
  def sort
    @filter = ListingFilter.find(element.dataset.filter_id)
    @filter.order_by = element.dataset.column
    @filter.direction = element.dataset.direction
    @filter.save
  end
end

And sure enough, it works! So what's going on here? Well, clicking the link invokes our reflex, which gets executed right before our current controller action runs again. It allows us to execute any kind of server-side logic, as well as play with the DOM in various ways, but with ruby code. Then, the DOM gets morphed over the wire.

What we did in our specific case: since our filter object is being persisted in Redis, it has a public id, which we stored as a data-attribute, and later retrieved from our reflex action. Then, we fetched the object from memory and updated it with new attributes. This is why @filter will be already defined by the time we get to that point. By default, not specifying anything more in our action will cause SR to just re-render the whole page before running the controller action. We could be more specific here, and just choose to morph a few elements to save precious milliseconds. But for demo purposes we’ll leave it as is.

Let’s add a filter next. We’ll start with the first one, by minimum price.

<div class="text-sm text-gray-600 flex justify-between">
  <label for="min-price">Minimum Price:</label>
  <span><output id="minPrice">50</output> $</span>
</div>
<input type="range"
  data-reflex="change->Listing#min_price"
  data-filter-id="<%= @filter.id %>"
  name="min-price"
  min="50"
  max="1000"
  value="<%= @filter.min_price %>"
  class="accent-indigo-600"
  oninput="document.getElementById('minPrice').value = this.value"
>

I got lazy and didn’t want to code an extra stimulus controller just to show the price value. But apart from that, we just need to add the new #min_price action:

# app/reflexes/listing_reflex.rb
class ListingReflex < ApplicationReflex
  def sort
    @filter = ListingFilter.find(element.dataset.filter_id)
    @filter.order_by = element.dataset.column
    @filter.direction = element.dataset.direction
    @filter.save
  end

  def min_price
    @filter = ListingFilter.find(element.dataset.filter_id)
    @filter.min_price = element.dataset.value
    @filter.save    
  end
end

And here in action:

I think by now you get the picture. Let’s just do the search and one of the checkbox filters.

In the view:

<!-- Search -->
<input type="search" value="<%= @filter.query %>" data-filter-id="<%= @filter.id %>" data-reflex="change->Listing#search">

<!-- Format Filter -->
<% Print::FORMATS.each_with_index do |format, index| %>
  <div class="flex items-center">
    <input data-reflex="change->Listing#format" <%= "checked" if @filter.format.include? format %> data-filter_id="<%= @filter.id %>" value="<%= format %>" type="checkbox">
  </div>
<% end %>

Our Reflex actions are starting to be pretty similar to each other, which calls for a refactor. You can’t do any better than leastbad’s approach, especially if you start having more complicated logic going on (like custom morphs or pagination):

# app/reflexes/listing_reflex.rb
class ListingReflex < ApplicationReflex
  def sort
    update_listing_filter do |filter|
      filter.order_by = element.dataset.column
      filter.direction = element.dataset.direction
    end
  end

  def min_price
    update_listing_filter do |filter|
      filter.min_price = element.value.to_i
    end
  end

  def max_price
    update_listing_filter do |filter|
      filter.max_price = element.value.to_i
    end
  end

  def format
    update_listing_filter do |filter|
      filter.format = element.value
    end
  end

  def search
    update_listing_filter do |filter|
      filter.query = element.value
    end
  end

  private

  def update_listing_filter
    @filter = ListingFilter.find(element.dataset.filter_id)
    yield @filter
    @filter.save
    # Add custom morphs here or any logic before the controller action is run
  end
end

And so on with the other filters. We can now combine search, filters and sort options with no page refresh.

Ambrosia on the cake

Let’s enhance the UX a bit. Right now there’s no pagination. Straight after adding pagy, clicking any page link will navigate, causing the params to reset. Let’s fix this by overriding pagy’s default template and wire links to our Reflex instead:

 <!-- views/listings/_pagy_nav.html.erb -->
<% link = pagy_link_proc(pagy) -%>
<%#                            -%><nav class="pagy_nav pagination space-x-4" role="navigation">
<% if pagy.prev                -%>  <span class="page prev"><a class="text-indigo-400" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= pagy.prev || 1 %>">Previous</a></span>
<% else                        -%>  <span class="page prev text-gray-300">Previous</span>
<% end                         -%>
<% pagy.series.each do |item|  -%>
<%   if    item.is_a?(Integer) -%>  <span class="page"><a class="text-indigo-400 hover:text-indigo-600" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= item %>"><%== item %></a></span>
<%   elsif item.is_a?(String)  -%>  <span class="page page-current font-bold"><%= item %></span>
<%   elsif item == :gap        -%>  <span class="page text-gray-400"><%== pagy_t('pagy.nav.gap') %></span>
<%   end                       -%>
<% end                         -%>
<% if pagy.next                -%>  <span class="page next"><a class="text-indigo-400 hover:text-indigo-600" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= pagy.next || pagy.last %>">Next</a></span>
<% else                        -%>  <span class="page next disabled">Next</span>
<% end                         -%>
<%#                            -%></nav>

# Add the paginate method to ListingReflex
def paginate
  update_listing_filter do |filter|
    filter.page = element.dataset.page
  end
end

# And update the controller
def index
  @filter ||= ListingFilter.create
  @pagy, @listings = pagy(@filter.results, items: 12, page: @filter.page, size: [1,1,1,1])
  # Can sometimes happen over navigation when collection gets changed in real time
rescue Pagy::OverflowError
  @pagy, @listings = pagy(@filter.results, items: 12, page: 1, size: [1,1,1,1])
end

Another issue is that at the moment, updating filters and our sorting options don’t update the URL params; refreshing the page clears everything, and we’re not able to save or share the result of our search to someone. Let’s take care of that as well. What we want is for our URL to always reflect the current state of filters on one hand, then be able to load our filter params from the URL on the other hand.

First step is made easy by the mighty cable_ready library, namely its push_state operation. Not only is it almost magical, but it is ready to use in any Reflex. Have a look at all you can do with it. Here is what our main action needs to do what we want:

# reflexes/listing_reflex.rb
def update_listing_filter
  @filter = ListingFilter.find(element.dataset.filter_id)
  yield @filter
  @filter.save
  # Updating URL with serialized attributes from our filter
  cable_ready.push_state(url: "#{request.path}?#{@filter.attributes.to_query}")
end

Now if you change any filter, type any query, change page or switch sorting option, the URL will update itself, including every filter attribute. Our last step is to load these attributes from the params on initial page load:

class ListingsController < ApplicationController
  include Pagy::Backend

  def index
    @filter ||= ListingFilter.create(filter_params)
    @pagy, @listings = pagy(@filter.results, items: 12, page: @filter.page, size: [1,1,1,1])
  rescue Pagy::OverflowError
    @pagy, @listings = pagy(@filter.results, items: 12, page: 1, size: [1,1,1,1])
  end

  private

  # Don't forget to update this list when adding filter options
  def filter_params
    params.permit(
      :query,
      :min_price,
      :max_price,
      :page, 
      :order_by, 
      :direction,
      category: [],
      tags: [],
      format: []
    )
  end
end

It’s starting to get pretty nifty. One last issue UX-wise : since we can no longer refresh to clear it all, we lack a Clear All button. Just add a link, then wire it to a Reflex action such as:

def clear
  ListingFilter.find(element.dataset.filter_id).destroy
  @filter = ListingFilter.create
  cable_ready.push_state(url: request.path)
end

And here you are, as close as ever to eternal bliss in search paradise. You can have a look at the live app here. UPDATE: Unfortunately fly.io's Upstash Redis doesn't play well with ActionCable, so connection gets lost after a while, preventing SR to work properly. While I tackle this issue, feel free to clone the repo. Not so heavenly after all.

Behold the afterlife

Let’s recap what we learned. Thanks to StimulusReflex , we learned how to build a super reactive search and filter interface with clean and extendable code, great performance, and almost no JavaScript. We saw how cable_ready could provide some sprinkle of magic behaviour on top of StimulusReflex. We were able to cleanly and temporarily persist, then update our search data thanks to all_futures. We also learned how to chain conditional scopes in a safe manner.

Unfortunately, good things rarely last forever. In our next episode, we’ll see how new requirements and a bigger set of records will party poop our not-so-eternal dream. You'll get to see how ElasticSearch can save the day and allow us to build the ultimate search engine.

Thanks for reading folks, and see you on the other side!