Official Trailblazer 2.1 Release Notes

(Not that there's an unofficial one anywhere, but yeah...)

This was posted on 27 Feb 2021 on the TRB website.

Note: We have fully documented the migration path to 2.1 while upgrading several complex apps. It worked.™

After around 3 years of silence, Trailblazer is back with its 2.1 release. Lots of improvements on the core gems come hand-in-hand with a quite flat migration path, so don't ya worry! We're also proud to tell you all about our new upcoming projects in this post, as well as communicating a looot of background about what's been going on in those past few years.

Here's a quick summary for those who want to get back to coding.

• API CHANGE The call API has slightly changed from two arguments to just one.
• AUTOLOADING We removed the trailblazer-loader gem just like Apple removed the headphone jack from the iPhone 6. This brings you faster startup and consistency with Rails autoloading.
• DOCUMENTATION We have a new website.
• TRACING We heard your cries. You can now see what path an operation took and use exception tracing for a beautiful dev experience.

Good to see you, again!

It almost feels unreal finishing up this release post. It's been so long! The work put into Trailblazer 2.1 has been tremendous, it could easily have been TRB 3.0, or even TRB III, since Roman version numbering turns out to be quite a fancy thing to do. However, as much as the internals have been improved, as little has changed on the public APIs of Trailblazer, so we decided to go with a minor release.

With the introduction of another new callback after_save_commit in Rails 6 it becomes obvious that we do have a problem in software engineering: how do we structure business code? We do have a few dozen database abstractions in every language, well written and designed, but, yeah, how do I structure the actual code of my domain, of my application's needs? What's the best way to put together application functions, put them in order, prevent other functions from being executed when it's not the time, and so on?

Trailblazer is an architectural style coming as a framework to implement our patterns, trying to answer that one question: "How do I implement business processes?". It gives you strong conventions along with patterns to structure your code flow better, handle errors, and integrate those components into bigger flows - or processes.

Having said that, I let you decide whether or not you join the ride! There is nothing wrong with building your own "service layer", and many companies have left the Traiblazer track in the past years due to problems they had and that we think we now fixed.

Speaking of problems, the craziness of the past years working on Trailblazer 2.1 can be visualized with the following screenshot.

Very often, in order to fix one function A in a commercial app, I had to rewire a track in the endpoint gem, which would only work if I add another DSL option in the dsl gem, which couldn't work straight away because the normalizer in the context gem needed to be extended, which was only possible if you could finally extract the start event via the activity gem whose test suite was currently broken because I needed to document feature x first, which couldn't be documented because before I had to allow dynamic params in the macro gem and... I think you got my point.

Luckily, I got help from a band of great people, I took it easy on the conference front, and for some other reasons I still don't understand all this is now finished and here's TRB 2.1.

About the Design

A big change from TRB 1.1 and even 2.0 is that we apply functional principles in most places. We barely keep state in instance variables. Rather, data is passed around from operation to operation, from step to step. We use OOP and inheritance solely for compile-time configuration. You define classes, steps, tracks and flows, inherit those, customize them using Ruby's built-in mechanics, but this all happens at compile-time. At runtime, no structures are changed anymore, your code is executed dynamically but only the ctx (formerly options) and its objects are mutated. This massively improves the code quality and with it, the runtime stability, which Ruby is famous for *cough.

I extracted a lot of smaller libraries from the original trailblazer gem. Most gems use the activity gem itself, which is the core of an operation and provides the runtime object that executes your steps in a certain order. It's so simple that I sometimes wonder why it took years to develop it! But all those little interfaces, structures and principles that you see as "common sense", they took a while to figure out.

A lot of work went into the dsl-linear gem that provides a DSL to create activities (or operations) using your old friend #step. It's not even a DSL since it doesn't create code at run-time, it doesn't even create code: it creates runtime objects, activities, that are executed when your operation is run. You could create those objects manually without the DSL, or even write your own DSL, which we're hoping for!

So, whenever you hear the medieval argument "Trailblazer is just a nasty DSL!", forgive your opponent, you now know better. The entire framework is based on small, clean Ruby structures that can be executed programmatically.

Also, the more I use Trailblazer in projects or even in Trailblazer itself, I feel how needed those new abstractions are. Yes, you can write everything with your own code, you don't need abstractions for flow control and automatic error handling, which makes me wonder why you're not programming in assembler since Ruby is also an "unnecessary abstraction" on top of a processor. We need abstractions, unless you want to program like we did 30 years ago.

Ok, enough of this - let's jump into the things that actually help you.

Loader / Startup Speed

Startup speed and loading of operations has been a never-ending problem.

To make it short: we returned to the Rails Way™, lowering our heads in shame, and adhere to the Rails file and class naming structure for operations. What used to be named Memo::Create is now Memo::Operation::Create, which reflects its file location concepts/memo/operation/create.rb (still the same!) and works with Rails' autoloading.

The big benefit is: we could remove the trailblazer-loader gem. The alternative loader was a naive attempt by me to combine "our" naming style with the Rails file structure. After endless fire-fighting against broken development reloading, inconsistencies, slow startup times in develpment mode, more incompatibilities with Rails' autoloader and some clarifying discussions with the great Xavier Noria himself (author of Rails loading and zeitwerk), we ditched our loading mechanism.

If you want to take advantage of the fast server startup times and (almost) seamless hot reloading of classes, you need to rename your operations.

This change is not super dramatic at all, given that in the future, you won't be calling operations yourself all too much anymore. With the upcoming workflow and endpoint gems, those abstractions will invoke the business logic for you, so don't worry too much.

New website

In case you haven't noticed: you are reading this release blog post on a newly designed website! Not only does it come with the most beautiful, romantic and inspiring hero backdrop ever (featuring a James-Cameron-worthy parallax effect, too!), it also attempts to provide all the old documentation while bringing you the hottest off-the-press docs for our new gems.

We try to keep the "information architecture" - a word I wouldn't have learned without the inspiring Alex Coles - as simple as possible: so far, we got a handful of pages accessible through the top navigation, and then the documentation behind the DOCS link. Here, the right sidebar helps you to navigate within the chapter.

After all, we're a nerdy developer's library and don't need no million hipster landing pages! One is more than enough and took months of collaboration between us, the design and web team in Argentina and our awesome comic artist Josh Bauman who also drew the epic Trailblazer book illustrations in 2016.

The documentation has been designed to make coherent information as quickly accessible as possible. We even added a search feature - something our beloved user base has asked for many times. Just hit the / key (or Shift+7) in the docs.

API CHANGES

The new 2.1 version comes with a few necessary but reasonable changes in method signatures. As painful as that might sound to your Rails-spoiled ears, we preferred to fix design mistakes now before dragging them on forever.

New call API

In versions before 2.1, the automatic merging of the params part and the additional options was confusing many new users and an unnecessary step.

# old style
result = Memo::Create.(params, "current_user" => current_user)

The first argument (params) was merged into the second argument using the key "params". You now pass one hash to call and hence do the merging yourself.

# new style
result = Memo::Operation::Create.(params: params, current_user: current_use )

Your steps use the existing API, and everything here is as it used to be before.

class Memo::Operation::Create < Trailblazer::Operation
  step :create_model

  def create_model(options, params:, **)
    # ..
  end
end

The new call API is much more consistent and takes away another thing we kept explaining to new users - an indicator for a flawed API. Believe it or not, but not a single time has this topic come up again on our support forum.

<%= h4 "Symbol keys for ctx" %>

The ctx (or options) object now behaves like a HashWithIndifferentAccess, a pattern used for Rails' params object. In other words, you can now access ctx[:current_user] instead of ctx["current_user"] and conveniently use the keyword argument version of it, too.

Another handy addition to ctx is aliasing. Before 2.1, you had to refer to, say, the main contract via ctx["contract.default"]. While I designed that name "back in the days" with multiple contracts per operation in mind, it turned out to be quite clumsy since you couldn't access it as a keyword argument.

You're able to set aliases now for the ctx object, allowing magical things to happen, amongst those an aliasing from "contract.default" to :contract.

class Memo::Operation::Create < Trailblazer::Operation
  step :inspect_contract

  def inspect_contract(ctx, contract:, **)
    puts contract #=> same as ctx["contract.default"]
  end

The ctx object is now documented in its own section.

Developer Experience

Debugging deeply nested operations before 2.1 sucked. Even if there was no nesting applied, an exception thrown from a random step was hard to find, the native Ruby stacktrace didn't really help, the TRB and Rails framework noise made it a painful experience to spot the source of an exception.

This was one of the major arguments in many companies against Trailblazer, besides it "being too complex" while it seemed much easier to program pure Ruby as we did in the 90s. And we liked it that way!

To tell you the truth, the new tracing feature was the original reason why I decided to write 2.1 and make you sit and wait in agony for years. Nevertheless, tracing is simply blowing my mind. I can't count how many hours and angering rushs of adrenaline I've saved since the introduction of the wtf? method and its helpful higher-level stack trace.

Not only does tracing show you the path the execution took in a successful invokation, it also shows you where an exception happenend. And not "in Ruby", it's "in Trailblazer", in the actual step where things broke.

With the new developer gem, you can trace executions, find exceptions, observe certain variables in the ctx throughout the flow, render operations and activities to visually understand what's going on, and much more.

As this new gem turns out to be a massive time saver for everyone, yes, even for us, we're focusing on extending and adding functions such as visualizing nested activities.

It might sound a bit overly sentimental or even cheesy, but I personally think that this is lifting the dev experience to a new level that will be hard to come by in pure Ruby. Yes, Trailblazer is adding new abstractions and concepts and they are different to the 90s-Ruby, but now, at the latest, it becomes obvious how this improves the developing process. We're no longer talking in two-dimensional method stack traces or byebug hoops, the language and conception is changing to the actual higher level code flow, to activities sitting in activities structured into smaller step units.

Trailblazer::Developer.wtf?(Memo::Operation::Create, [ctx])

Once you've used the wtf? method, you will feel it!

Unlimited Wiring

Over the past years, harnessing the flow control and error handling mechanics we added in 2.0, one thing became obvious pretty quickly: users want more wiring possibilities. The linear railway and "fast track" concepts have been a helpful tool for structuring code flow, but people need to be able to model arbitrary flows.

Needless to say that 2.1 provides you just that. We added the concepts of outputs to steps that can be added or "re-wired" using the Wiring API.

You may add additional outputs, rewire, go back, error-out in additional termini, go nuts, and all that using a super simple DSL.

class Execute < Trailblazer::Activity::Railway
  step :model, Output(:failure) => End(:not_found)
  step :validate
  step :save,  Output(:retry) => Id(:model)
end

This works for both Activity subclasses and operations (which are just special-flavored Activity::FastTracks).

While you're still wondering whether we've completely lost it, we already use those new wiring mechanics to build everything from simple edge cases in code flows to long-running, complex business workflows.

A major improvement here is the ability to maintain more than two explicit termini. In 2.0, you had the success and the failure termini (or "ends" as we used to call them). Now, additional ends such as not_found can be leveraged to communicate a non-binary outcome of your activity or operation.

Using a terminus to indicate a certain outcome - in turn - allows for much stronger interfaces across nested activities and less guessing! For example, in the new endpoint gem, the not_found terminus is then wired to a special "404 track" that handles the case of "model not found". The beautiful thing here is: there is no guessing by inspecting ctx[:model] or the like - the not_found end has only one meaning!

Check the endpoint gem if you want to see that in action.

Core team

In the past 1 ½ years something weird happened: a real core team formed around the Trailblazer gems. I say "real" because in the past 15 years of OSS, I've had people come and go, being of great help but never staying and taking over long-term responsibilities - which I found to be the pivotal element of a core team.

Eventually, those kids convinced me to start the Trailblazer organization on Github and move over all "apotonick gems". Over the course of time, I saw myself giving away that aforementioned responsibility with a smile on my face, adding owners and collaborators to gems, yes, even giving away entire gems, letting people work on documentation and just trusting someone and their skills.

I have no words to describe how good that feels!

For me, a dream has come true. I work with crazy geniuses who share many of my opinions (not all, and that's good). I learned to "let go" and simply trust others to maintain certain gem suites. Messages like "don't worry, I'll do it" combined with a pull requests minutes later - things I literally dreamed of a few years ago, are now part of my daily routine.

Here are those kids, in the order of appearance: <3

• Celso Fernandes who's been part of TRB since the 12ths commit or something. He's currently focusing on the business side, consulting, and is the magical sysop of our infrastructure.
• Emanuele Magliozzi worked for many years as a TRB consultant and is now the master of reform and its 2.x line and interrupting our important daily work with Aussie bantering.
• Abdelkader Boudih who's still unaware that I secretly call him "The Machine" since I'm not sure he's an AI bot trained to fulfill basically any request an OSS author might have.
• Pedro Visintin, besides keeping me busy talking about punk rock guitars, is the person behind the PRO editor and our sharply-dressed agile marketing director.
• Kamil Milewski has been contributing to TRB for years both intellectually and with many code additions and even more deletions.
• Krzysztof Piotrowski has been tirelessly helping with writing example applications and turned out to be the refactoring expert on board.
• Adam Piotrowski is resisting my deceiving calls to work on the code side and rather manages our processes - so we gladly let him do just this!
• Yogesh Khater is now working full-time for Trailblazer, working on all 48 repositories at the same time (plus our private ones!), releasing gems when I sleep, leaving me jobless and peacefully napping in the sun.

In addition to the organically formed core team, I started Trailblazer GmbH 4 years ago with my relocation from Australia back to Europe. One of our consulting clients is the central police department of a German state that has kept me busy for more than three years now. And yes, at TRB GmbH, we do pay people to work on OSS!

License and PRO

Around 2 years ago I decided to end the experiment of "TRB PRO" as I felt I didn't provide enough value to paying users. In the end, we had around 150 companies and individuals signed up, which was epic and a great funding source for more development.

We're now relaunching PRO, but instead of a paid chat and (never existing) paid documentation, your team gets access to paid gems, our visual editor for workflows, and a commercial license.

The editor (that mostly plays along with the workflow gem) is already in use for a bunch of commercial projects that we consult, but will be launched in its stable version somewhere this year.

The core gems of Trailblazer have been re-licensed to LGPLv3 to raise awareness for this. For 99% of all our users, this has no legal implication at all - go use TRB for free if you don't feel like paying money for OSS.

With all this "monetization" happening around Trailblazer, we will also make sure that all free and paid parts of the project grow adult and maintan an LTS - or long-term support - status. Those are good news to all you users out there having been scared to use gems of this project, not knowing whether or not they're being maintained, breaking code in the future or making your developers addicted to and then cutting off the supply chain. Trailblazer 2.1 onwards is LTS, and the last 1 ½ years of collaboration have proven that.

Upcoming gems and plans

We use Trello boards now to organize our work. Yeah, you can teach old dogs new tricks! Not only does it help to structure myself, it also manifests how many ideas and improvements we got lined up for the near future.

• COMPILE-TIME PERFORMANCE While we're pretty good on the runtime performance, compiling all those activities takes more time than it could. We already started improving the compilation process and will ship updates in the next months. Optimization in this case is nothing crazy, just something I neglected while designing the framework.
• TRAILBLAZER-TEST The official stable release is only weeks away bringing you a bunch of new assertions that drastically reduce coding effort for tests! Of course, Minitest and RSpec will both be supported.
• TRAILBLAZER-STORY will follow as it turned out to be inevitable for setting up application state for tests. Instead of fumbling around with factories and traits in your tests, you "tell a story" about what to create in which order, easily customizable, and all written using activities. Currently, I'm working on designing the interfaces and it's real fun!
• TRAILBLAZER-WORKFLOW is another dream 'o mine come true. It allows creating long-term processes (or state machines) based on BPMN diagrams that can be modeled using our editor. I'm pretty confident that you, coming from AASM or your own state machine engine, might find this approach highly interesting. We're using this gem in several commercial apps already, so the first release should be out this summer (I didn't say which hemisphere).
• TRAILBLAZER-ENDPOINT I've been promising for many years and it turned out I couldn't have fully designed it without the tools we do have now. Endpoint is the missing link between your routing (Rails, Hanami, ...) and the "operation" to be called. It provides standard behavior for all cases 404, 401, 403, etc and lets you hook in your own logic like Devise or Tyrant authentication, again, using TRB activity mechanics.
• TYRANT is an authentication framework fully written in TRB activities, interacting with Rails, workflows and your database. It's highly customizable using the TRB mechanics and definitely less rough to work with than Devise. Looking forward to the release here!

<%= img "/2.1/blog-ep-draft.png" %>

Documentation

Writing documentation for the new website has been fun. Yes, fun! Mainly because we have a helpful little framework behind the page rendering that pulls code snippets from real tests out of the actual gems!

Many users have asked whether there will be an updated version of the Trailblazer book. While it's really cute and humbling to still have around $10 per month income from the now free book, keep in mind it was published end of June 2016, that's pretty much four years ago!

What this means is: I better refrain from writing a new book and we rather focus on more and better docs. Writing examples and technical docs as a team has been working out pretty well recently and we will add more Trailblazer tutorials in the next half year. A comprehensive workflow tutorial demonstrating long-running processes and all things Trailblazer is also in the works! The legendary cfp-app will become a Rails-to-TRB refactoring tutorial.

We decided against paid documentation, so all will be freely available on our shiny new website.

Staying up-to-date

All new publications, changes, ideas, conferences and parties will be posted regularly on our Facebook page. Yepp, that's true. We found it much easier to maintain and follow than most other social media platforms.

And if you have anything to talk about with us, use the new Zulip chat. We're looking forward meeting you! See you on the trail!

Nick Sutterer, @apotonick