During summer 2021, I got lucky enough to cross Twitter paths with Peter Szinek, who introduced me to his team and got me hired at RCRDSHP, Obie Fernandez’s latest Web 3.0 project involving Music NFTs. Being myself a pro musician and music producer, I was thrilled to finally be able to mix my two passions-turned-into-a-living. Surrounded by awesome developers, I learned and built tons of cool stuff, among which a reactive, super performant server-side-rendered search experience using StimulusReflex, ElasticSearch and close to no JavaScript. The feature is still live and pretty much unchanged.
The purpose of this series will be to first reimplement this super friendly UX with basic filters and sort options, along with StimulusReflex. Then, we’ll see how ElasticSearch can allow more complex filters and search scenarios, while improving performance. In the last episode, we’ll try and replace StimulusReflex with the new Custom Turbo Stream Actions and compare implementation/behaviour. If time allows, I might add a bonus episode to show how to deploy all this in production. Let’s dig in.
What are we building?
Remember how back in the day, people used to collect art printed on actual paper ? A bit like NFTs, only physical. Weird, right? Well, let’s picture an app that would allow users to buy and sell limited edition art prints. The prints would include photographs, movie posters, and illustrations of various formats. It should look like that:
Here’s what the DB looks like:
Please note the tags
column is of string array
type. One might argue that the Listing
table, in our case, could easily be skipped. But for the sake of keeping a real-world complexity scenario, let’s say that we’d like to keep the actual Prints
separate from their listings (and since the app allows users to sell their prints, we might reasonably think that a print could be listed several times).
OK. Show me the gear
First things first: on the frontend, we’ll use StimulusReflex (a.k.a SR) to build a super reactive and friendly search experience with very little code, and little to no JavaScript. For those unfamiliar:
StimulusReflex is a library that extends the capabilities of both Rails and Stimulus by intercepting user interactions and passing them to Rails over real-time websockets. The current page is quickly re-rendered and morphed to reflect the new application state.
Sounds a bit like Hotwire on paper, though you’ll see how their philosophy greatly differs in the last episode of this series. We’ll also use a sprinkle of CableReady, a close cousin of SR.
On the backend, we'll need a few tools. Apart from the classics (ActiveRecord
scopes and the pg_search gem), you’ll see how the (yet officially unreleased but production-tested) all_futures gem, built by SR authors, will act as an ideal ephemeral object to temporarily store our filter params and host our search logic. Finally, we’ll use pagy for pagination duties.
Philtre d'amour
(Please indulge this shitty French pun, the expression philtre d'amour meaning love potion but also sounds like beloved filter)
Let’s start by creating some simple data. We’ll add a few artworks of different kind : photographs
, illustrations
, and posters
. Each will have several tags
from a given list and an author
. For now, let’s just generate one print per artwork, and a listing for each. Prints can be one of 3 available formats
, while listings will be of varying price
.
Now let’s list our different features:
- Search by name or author
- Filter by minimum price
- Filter by maximum price
- Filter by category
- Filter by print format
- Filter by tags
- Order by price
- Order by date listed
Before I started building my feature, my former colleague and friend @marcoroth pointed me to leastbad’s Beast Mode, from which I took heavy inspiration to get going. That’s how I discovered his gem all_futures, which provides us with an ActiveRecord-like object that will persist to Redis. Let’s see how things look like.
# app/controllers/listings_controller.rb
class ListingsController < ApplicationController
def index
@filter ||= ListingFilter.create
@listings = @filter.results
end
end
# app/models/listing_filter.rb
class ListingFilter < AllFutures::Base
# Filters
attribute :query, :string
attribute :min_price, :integer, default: 1
attribute :max_price, :integer, default: 1000
attribute :category, :string, array: true, default: []
attribute :tags, :string, array: true, default: []
attribute :format, :string, array: true, default: []
# Sorting
attribute :order, :string, default: "created_at"
attribute :direction, :string, default: "desc"
def results
# TODO: Build a query out of these attributes
end
end
Notice how params are absent from the controller, and nothing gets passed to our ListingFilter
object? And how come @filter
could potentially be already defined? You’ll see why in a bit, so let’s first look at building the query.
In his approach, @leastbad simply created an ActiveRecord
scope for each filter, then very cleverly and neatly, chained them to build his final filtered query, much like this:
# In app/models/listing_filter.rb
def results
Listing.for_sale
.price_between(min_price, max_price)
.from_categories(category)
.with_tags(tags)
.with_formats(format)
.search(query)
.order(order => direction)
end
You might wonder: “But what if filters are empty and arguments blank? The chain’s gonna break!”. Well, have a look at the scopes declaration:
# app/models/listing.rb
class Listing < ApplicationRecord
belongs_to :print
scope :for_sale, ->{ where(sold_at: nil) }
scope :price_between, ->(min, max) { where(price: min..max) }
scope :with_formats, ->(format_options) { joins(:print).where(prints: {format: format_options}) if format_options.present? }
scope :from_categories, ->(cat_options) { joins(:artwork).where(artworks: {category: cat_options}) if cat_options.present? }
scope :with_tags, ->(options) { joins(:artwork).where("artworks.tags && ?", "{#{options.join(",")}}") if options.present? }
scope :search ->(query) { # TODO }
end
In ListingFilter
, the crucial bit is to make sure that every attribute has a default value. The magic then occurs in the if
statement at the end of the scopes expecting an argument: if the lambda returns nil
, then it will essentially be ignored, and the collection returned as is. Such a nice trick. Time for some specs to ensure that things actually work:
RSpec.describe ListingFilter, type: :model do
let!(:photo) { Artwork.create(name: "Dogs", author: "Elliott Erwitt", year: 1962, tags: %w[Animals B&W USA], category: "photography") }
let!(:poster) { Artwork.create(name: "Fargo", author: "Matt Taylor", year: 2021, tags: %w[Cinema USA], category: "poster") }
let!(:photo_print) { photo.prints.create(format: "30x40", serial_number: 1) }
let!(:photo_print_2) { photo.prints.create(format: "18x24", serial_number: 200) }
let!(:poster_print) { poster.prints.create(format: "40x50", serial_number: 99) }
let!(:photo_listing) { photo_print.listings.create(price: 800) }
let!(:photo_listing_2) { photo_print_2.listings.create(price: 400) }
let!(:poster_listing) { poster_print.listings.create(price: 200) }
let!(:sold_listing) { poster_print.listings.create(price: 300, sold_at: 2.days.ago) }
describe "#results" do
it "doesn't return a sold listing" do
expect(ListingFilter.create.results).not_to include(sold_listing)
end
context "Filter options" do
it "Filters by price" do
filter = ListingFilter.create(min_price: 100, max_price: 300)
expect(filter.results).to match_array([poster_listing])
filter.update(max_price: 1000)
expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
end
it "Filters by format" do
filter = ListingFilter.create(format: ["40x50"])
expect(filter.results).to match_array([poster_listing])
filter.update(format: ["40x50", "30x40"])
expect(filter.results).to match_array([photo_listing, poster_listing])
end
it "Filters by category" do
filter = ListingFilter.create(category: ["photography"])
expect(filter.results).to match_array([photo_listing_2, photo_listing])
filter.update(category: ["photography", "poster"])
expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
end
it "Filters by tags" do
filter = ListingFilter.create(tags: ["Cinema"])
expect(filter.results).to match_array([poster_listing])
filter.update(tags: ["Cinema", "Animals"])
expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
end
it "Filters by multiple attributes" do
filter = ListingFilter.create(tags: ["Cinema"], max_price: 300, category: ["poster"])
expect(filter.results).to match_array([poster_listing])
end
end
end
end
All green. I hope you’ll appreciate how easy it is to test this. Let’s quickly add pg_search
to our Gemfile, then take care of the search scope:
# app/models/listing.rb
class Listing < ApplicationRecord
include PgSearch::Model
belongs_to :print
has_one :artwork, through: :print
scope :search, ->(query) { basic_search(query) if query.present? }
# Skipping the other scopes...
pg_search_scope :basic_search,
associated_against: {
artwork: [:name, :author]
},
using: {
tsearch: {prefix: true}
}
#...
end
Since our listings don’t carry much information, we’ll have to jump a few tables to look where we need, namely the name
and author
columns of our Artwork
model. Unfortunately, pg_search
doesn’t support associated queries further than 1 table away, thus the has_one... through
relationship we needed to add. Let’s add some tests for the search:
context "Search" do
it "renders all listings if no query is passed" do
filter = ListingFilter.create(query: "")
expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
end
it "can search by artwork name or author" do
filter = ListingFilter.create(query: "Erwitt")
expect(filter.results).to match_array([photo_listing, photo_listing_2])
filter.update(query: "Fargo")
expect(filter.results).to match_array([poster_listing])
end
it "can both search and sort" do
filter = ListingFilter.create(query: "Erwitt", order_by: "price", direction: "asc")
expect(filter.results.to_a).to eq([photo_listing_2, photo_listing])
filter.update(query: "Erwitt", order_by: "price", direction: "desc")
expect(filter.results.to_a).to eq([photo_listing, photo_listing_2])
end
end
We run the tests and of course, everything is gr… Oh no. Looks like the last test is acting up:
Apparently, a known problem of pg_search is that it doesn’t play well with eager loading, nor combinations of join
and where
queries. The recommended workaround (and my usual plan B when ActiveRecord queries start to get ugly) is to use a subquery:
# In app/models/listing_filter.rb
def results
filtered_listings_ids = Listing.for_sale
.price_between(min_price, max_price)
.from_categories(category)
.with_tags(tags)
.with_formats(format)
.pluck(:id)
Listing.where(id: filtered_listings_ids)
.search(query)
.order(order_by => direction)
.limit(200)
end
Let’s also add some last specs for the sort options and run all this.
context "Sort options" do
specify "Recent listings first (default behaviour)" do
filter = ListingFilter.create
expect(filter.results.to_a).to eq([poster_listing, photo_listing_2, photo_listing])
end
specify "Most expensive first" do
filter = ListingFilter.create(order_by: "price", direction: "desc")
expect(filter.results.to_a).to eq([photo_listing, photo_listing_2, poster_listing])
end
specify "Least expensive first" do
filter = ListingFilter.create(order_by: "price", direction: "asc")
expect(filter.results.to_a).to eq([poster_listing, photo_listing_2, photo_listing])
end
end
Everything’s green… Except for the search and filter
option. The error’s gone, but the test still fails; the ordering doesn’t seem to work, despite all the sorting tests being green. After another lookup on pg_search
known issues, it appears that order
statements following the search scope don’t work. Workarounds include using reorder
instead, or moving the order
clause up the chain. I opted for the first option, which make all tests pass. Let's move on.
Stairway to Heaven
Now that we know that our backend is working as it should, let’s wire up our stuff. I’m gonna skip on Stimulus Reflex setup and configuration and dive right in. You can easily follow the official setup or, if you use import-maps
, follow @julianrubisch’s article on the topic. I also know that leastbad has been working on an automatic installer that detects your configuration and sets everything up for you if you care to try it before the next version of SR gets released.
Once you’re done with that, let’s begin with the sort first. Let’s recap our sorting options and store them somewhere:
class ListingFilter < AllFutures::Base
SORTING_OPTIONS = [
{column: "created_at", direction: "desc", text: "Recently added"},
{column: "price", direction: "asc", text: "Price: Low to High"},
{column: "price", direction: "desc", text: "Price: High to Low"}
]
#...
attribute :order_by, :string, default: "created_at"
attribute :direction, :string, default: "desc"
#...
# Memoizing the value to avoid re-computing at every call
def selected_sorting_option
@_selected_option ||= SORTING_OPTIONS.find {|option| order_by == option[:column] && direction == option[:direction] }
end
end
Then in our “Sort by” dropdown, we’ll have something like :
<div class="dropdown">
<button>
Sort by:<span><%= @filter.selected_sorting_option[:text] %></span>
</button>
<!-- Skipping lots of HTML -->
<% ListingFilter::SORTING_OPTIONS.each do |option| %>
<% if option == @filter.selected_sorting_option %>
<span class="font-semi-bold ..."><%= option[:text] %></span>
<% else %>
<a data-reflex="click->Listing#sort"
data-column="<%= option[:column] %>"
data-direction="<%= option[:direction] %>"
data-filter-id="<%= @filter.id %>"
href="#"
>
<%= option[:text] %>
</a>
<% end %>
<% end %>
</div>
Even if you're unfamiliar with StimulusReflex, it should still remind you of the way we invoke regular stimulus controllers. Only here, when our link gets clicked, it should trigger the sort
action (a ruby method) from the Listing
reflex (a ruby class). Let’s code it:
# app/reflexes/listing_reflex.rb
class ListingReflex < ApplicationReflex
def sort
@filter = ListingFilter.find(element.dataset.filter_id)
@filter.order_by = element.dataset.column
@filter.direction = element.dataset.direction
@filter.save
end
end
And sure enough, it works! So what's going on here? Well, clicking the link invokes our reflex, which gets executed right before our current controller action runs again. It allows us to execute any kind of server-side logic, as well as play with the DOM in various ways, but with ruby code. Then, the DOM gets morphed over the wire.
What we did in our specific case: since our filter object is being persisted in Redis, it has a public id, which we stored as a data-attribute, and later retrieved from our reflex action. Then, we fetched the object from memory and updated it with new attributes. This is why @filter
will be already defined by the time we get to that point. By default, not specifying anything more in our action will cause SR to just re-render the whole page before running the controller action. We could be more specific here, and just choose to morph
a few elements to save precious milliseconds. But for demo purposes we’ll leave it as is.
Let’s add a filter next. We’ll start with the first one, by minimum price.
<div class="text-sm text-gray-600 flex justify-between">
<label for="min-price">Minimum Price:</label>
<span><output id="minPrice">50</output> $</span>
</div>
<input type="range"
data-reflex="change->Listing#min_price"
data-filter-id="<%= @filter.id %>"
name="min-price"
min="50"
max="1000"
value="<%= @filter.min_price %>"
class="accent-indigo-600"
oninput="document.getElementById('minPrice').value = this.value"
>
I got lazy and didn’t want to code an extra stimulus controller just to show the price value. But apart from that, we just need to add the new #min_price
action:
# app/reflexes/listing_reflex.rb
class ListingReflex < ApplicationReflex
def sort
@filter = ListingFilter.find(element.dataset.filter_id)
@filter.order_by = element.dataset.column
@filter.direction = element.dataset.direction
@filter.save
end
def min_price
@filter = ListingFilter.find(element.dataset.filter_id)
@filter.min_price = element.dataset.value
@filter.save
end
end
I think by now you get the picture. Let’s just do the search and one of the checkbox filters.
In the view:
<!-- Search -->
<input type="search" value="<%= @filter.query %>" data-filter-id="<%= @filter.id %>" data-reflex="change->Listing#search">
<!-- Format Filter -->
<% Print::FORMATS.each_with_index do |format, index| %>
<div class="flex items-center">
<input data-reflex="change->Listing#format" <%= "checked" if @filter.format.include? format %> data-filter_id="<%= @filter.id %>" value="<%= format %>" type="checkbox">
</div>
<% end %>
Our Reflex actions are starting to be pretty similar to each other, which calls for a refactor. You can’t do any better than leastbad’s approach, especially if you start having more complicated logic going on (like custom morphs or pagination):
# app/reflexes/listing_reflex.rb
class ListingReflex < ApplicationReflex
def sort
update_listing_filter do |filter|
filter.order_by = element.dataset.column
filter.direction = element.dataset.direction
end
end
def min_price
update_listing_filter do |filter|
filter.min_price = element.value.to_i
end
end
def max_price
update_listing_filter do |filter|
filter.max_price = element.value.to_i
end
end
def format
update_listing_filter do |filter|
filter.format = element.value
end
end
def search
update_listing_filter do |filter|
filter.query = element.value
end
end
private
def update_listing_filter
@filter = ListingFilter.find(element.dataset.filter_id)
yield @filter
@filter.save
# Add custom morphs here or any logic before the controller action is run
end
end
And so on with the other filters. We can now combine search, filters and sort options with no page refresh.
Ambrosia on the cake
Let’s enhance the UX a bit. Right now there’s no pagination. Straight after adding pagy
, clicking any page link will navigate, causing the params to reset. Let’s fix this by overriding pagy’s default template and wire links to our Reflex instead:
<!-- views/listings/_pagy_nav.html.erb -->
<% link = pagy_link_proc(pagy) -%>
<%# -%><nav class="pagy_nav pagination space-x-4" role="navigation">
<% if pagy.prev -%> <span class="page prev"><a class="text-indigo-400" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= pagy.prev || 1 %>">Previous</a></span>
<% else -%> <span class="page prev text-gray-300">Previous</span>
<% end -%>
<% pagy.series.each do |item| -%>
<% if item.is_a?(Integer) -%> <span class="page"><a class="text-indigo-400 hover:text-indigo-600" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= item %>"><%== item %></a></span>
<% elsif item.is_a?(String) -%> <span class="page page-current font-bold"><%= item %></span>
<% elsif item == :gap -%> <span class="page text-gray-400"><%== pagy_t('pagy.nav.gap') %></span>
<% end -%>
<% end -%>
<% if pagy.next -%> <span class="page next"><a class="text-indigo-400 hover:text-indigo-600" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= pagy.next || pagy.last %>">Next</a></span>
<% else -%> <span class="page next disabled">Next</span>
<% end -%>
<%# -%></nav>
# Add the paginate method to ListingReflex
def paginate
update_listing_filter do |filter|
filter.page = element.dataset.page
end
end
# And update the controller
def index
@filter ||= ListingFilter.create
@pagy, @listings = pagy(@filter.results, items: 12, page: @filter.page, size: [1,1,1,1])
# Can sometimes happen over navigation when collection gets changed in real time
rescue Pagy::OverflowError
@pagy, @listings = pagy(@filter.results, items: 12, page: 1, size: [1,1,1,1])
end
Another issue is that at the moment, updating filters and our sorting options don’t update the URL params; refreshing the page clears everything, and we’re not able to save or share the result of our search to someone. Let’s take care of that as well. What we want is for our URL to always reflect the current state of filters on one hand, then be able to load our filter params from the URL on the other hand.
First step is made easy by the mighty cable_ready
library, namely its push_state
operation. Not only is it almost magical, but it is ready to use in any Reflex. Have a look at all you can do with it. Here is what our main action needs to do what we want:
# reflexes/listing_reflex.rb
def update_listing_filter
@filter = ListingFilter.find(element.dataset.filter_id)
yield @filter
@filter.save
# Updating URL with serialized attributes from our filter
cable_ready.push_state(url: "#{request.path}?#{@filter.attributes.to_query}")
end
Now if you change any filter, type any query, change page or switch sorting option, the URL will update itself, including every filter attribute. Our last step is to load these attributes from the params on initial page load:
class ListingsController < ApplicationController
include Pagy::Backend
def index
@filter ||= ListingFilter.create(filter_params)
@pagy, @listings = pagy(@filter.results, items: 12, page: @filter.page, size: [1,1,1,1])
rescue Pagy::OverflowError
@pagy, @listings = pagy(@filter.results, items: 12, page: 1, size: [1,1,1,1])
end
private
# Don't forget to update this list when adding filter options
def filter_params
params.permit(
:query,
:min_price,
:max_price,
:page,
:order_by,
:direction,
category: [],
tags: [],
format: []
)
end
end
It’s starting to get pretty nifty. One last issue UX-wise : since we can no longer refresh to clear it all, we lack a Clear All button. Just add a link, then wire it to a Reflex action such as:
def clear
ListingFilter.find(element.dataset.filter_id).destroy
@filter = ListingFilter.create
cable_ready.push_state(url: request.path)
end
And here you are, as close as ever to eternal bliss in search paradise. You can have a look at the live app here. UPDATE: Unfortunately fly.io's Upstash Redis
doesn't play well with ActionCable, so connection gets lost after a while, preventing SR to work properly. While I tackle this issue, feel free to clone the repo. Not so heavenly after all.
Behold the afterlife
Let’s recap what we learned. Thanks to StimulusReflex
, we learned how to build a super reactive search and filter interface with clean and extendable code, great performance, and almost no JavaScript. We saw how cable_ready
could provide some sprinkle of magic behaviour on top of StimulusReflex
. We were able to cleanly and temporarily persist, then update our search data thanks to all_futures
. We also learned how to chain conditional scopes in a safe manner.
Unfortunately, good things rarely last forever. In our next episode, we’ll see how new requirements and a bigger set of records will party poop our not-so-eternal dream. You'll get to see how ElasticSearch
can save the day and allow us to build the ultimate search engine.
Thanks for reading folks, and see you on the other side!
Resources
- The repo on Github
- The live app
- StimulusReflex documentation
- CableReady documentation
- Beast mode, by @leastbad
- all_futures
- pg_search
- pagy
- Movie posters courtesy of Plakat
- Illustrations courtesy of Amen Artwork
- My mostly analog photobook
Top comments (9)
Great write up Louis. Looking forward to the next episode.
Thanks David ! Coming very soon :) Probably this weekend 💪
any updates i found the article great
Sorry, just came back from an intense 3-month music tour, episode 2 is nearly done :)
Thanks. Hope you are well.
Did you solve the issue with fly.io?
i kiss your heart, thats what i need rn <3
Awesome !
Very well written and super exhaustive.
Kudos Louis !
thank you, it's very educative for me.