Cover Photo by Marie P on Unsplash
Disclaimer: I work at Keyrock but views expressed here are my own. However, if you like the views, do go checkout the company as it is full of awesome people and I have learned much whilst working there, including Rust.
Where we were
In 2019, whilst working at Keyrock, the dev team (around 4-6 of us during that period) had reached the limits of what our JavaScript prototype trading services could do when it came to handling the complex world of high frequency trading (HFT). We had to process multiple feeds of information concurrently and have data available, within certain time constraints, inside a single service, with reliability. Of course, I am aware JS was probably not the best choice but we had a team-wide capability and have you ever tried to prototype in JS? It's so fast... Anyway, we had rapidly prototyped our first generation of services in JavaScript and whilst JS is suited to many use cases, (first strong opinion of the article) HFT is not one of them. Not that this stopped us trying 😇.
In choosing a new technology for the second generation of services we went through a research phase, trying out Go-lang, Kotlin, Elixir, and also another language I was tinkering with... Rust. We knew we didn't want to go the C++ route for reasons I will expand on in a different article. I'd done some Rust tests with basic sorting algorithms and couldn't quite believe the speed and kept going on about it at our office in Brussels. Another senior dev challenged the technology (vs what we had in production) and said he didn't want to keep hearing me go on about it unless it was significantly faster than what we had; he did some deeper experiments himself and soon went on to become our most ardent Rust emissary. Rust had the speed we needed, approachable concurrency primitives, acceptable syntax, and great tooling. We dove into this new world and today most of our production code is Rust.
However, there was something else about this language, even at the early stages, that I couldn't quite put my finger on. While sometimes complex to deal with, at the time I knew our decision was right even though I couldn't expand much on why, beyond the speed, lack of garbage collector, and concurrency support. Was that risky? I didn't think so - we did (just) enough research and I personally had Apple's philosophy ringing in my head - that we should use the opportunity to both eliminate a technology we had maxed out, and replace it with a new technology entering it's dawn.
Three years later I can express explicitly what I think I sensed intuitively - And I feel compelled to write about it! If you are new to Rust, I know where you are going... it is a world of initial confusion and frustration, followed eventually by a feeling that this just makes sense.
Today we see rust everywhere. Blockchains, web services, libraries supporting Python and JavaScript modules, even inside the holy of holies - the Linux kernel. This three-part article is about why that is the case.
Why is this happening?
Is it a cult? Is it mass hysteria? Or it just the latest fashion in hipster coding languages that's running out of control? Well, maybe there's an element of this - but there are also many concrete reasons this revolution is happening. And make no mistake - it is a revolution.
From the 2021 Stack Overflow Developer Survey
We can begin with the factors that are easy to understand and accept, and then I'll get into the less obvious but immensely powerful features of the language.
Let's start with the entire list and then break them down:
Part 1
- Syntax
- Separation of Data and Code
- Type safety
- Declarative Programming and Functional Composition features
- Evolved package management
- Concurrency ready std lib and great async support
- Clear, helpful compiler errors
- Zero cost abstractions (Optimisation)
- Energy efficiency
- WASM
- Tauri - High performance, secure, cross-platform desktop computing
- Macros!
- Cryptocurrency and Blockchains
- Everybody else's tooling
- Speed
- No garbage collection
- Memory safety features
Syntax
At university I had some experience with Pascal and Smalltalk (it was a long time ago), before that I was coding Motorola 68000 series assembler on a Commodore Amiga and before that (my first exposure to low-level coding) I was playing with machine code on the MOS Technology 6510 in the Commodore 64.
Why am I sharing this? Because the coding syntax in that world was not "C" style. During the following years I also used C, C++, Java, JavaScript, Perl, C#, and later R, which are known as C-type or C-family of languages. (Python is a little different but it's not far off). The other languages above are not and this often leads to frustration in learning the language and difficulty in finding people that can "convert" to those languages.
The C-type languages above are actually a very small selection of the number of languages using C-family syntax (around 73). Why so many? Well, after the success of C & C++, it was theorised that it would be much easier to dive into a familiar syntax if learning a new language. For example Sun Microsystems worked on a project to clean up and and make C++ better by being able to run it everywhere - called Oak. Later they needed to change the name and they called it Java.
The fact that Rust retains this C-type "DNA" is by design. Systems programming is primarily the domain of C & C++ developers, with a smaller group of Objective-C (a superset of C) and Swift developers in the Apple universe. Rust honours this legacy - and targets those developers - by fusing a very familiar syntax together with symbols and features that facilitate new paradigms in a way that is only mildly distracting at the beginning of the Rust journey, and makes a lot of sense when you get used to it.
I've had a tech-challenge submissions from C++ devs with only 1-2 months of Rust experience and they "get it", often much more quickly than devs with other backgrounds. These submissions are not idiomatic Rust at all but they show the power of a familiar syntax.
Separation of Code and Data
I'm a big fan of Functional Programming (FP) - and Elixir and Erlang in particular. The original sub-heading here was: A reasonable step away from the era of misunderstood Object Oriented philosophy and architecture, towards separation of code and data, using Traits and Functions.
That's a long section title. I rewrote it several times. It was really difficult to find a title that represents this factor that wouldn't trigger the masses of converts of the confusing, and usually misunderstood, interpretation of Object Oriented Programming (OO, or OOP), who have spent years teaching and developing methods and patterns for structuring code, based on an approach that has virtually zero evidence but instead was the result of millions spent in marketing, over around 3-4 decades, plus the directed efforts of non-technical managers to split coding into "units of work".
There are various histories that document this (my favourite is by the eternally affable Richard Feldman, author of the Elm programming language). A very short summary:
A long time ago programs were made of data and procedures and were often a highly coupled mess of data sharing. Then a guy call Alan (Kay, 1966) said programs should pass messages instead of share data (an idea from Simula, 1962). He and others made a language of objects with associated functions that passed messages between objects (Smalltalk '72). But it seems (to keep the story short) he was somewhat mis-heard, and later languages implemented these ideas with their own odd assumptions.
“C++ is a significant barrier to learning and correctly applying object-oriented methodologies and technologies.”
— Bruce Webster, Pitfalls of object-oriented development
(New York: M&T Books, 1995), 139 (quoted in Hsu, Mahoney - see below)
Alan also tried to explain some of the confusion many years later:
“I made up the term ‘object-oriented’, and I can tell you I didn’t have C++ in mind.”
— Alan Kay, OOPSLA ‘97
“OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things.”
— Alan Kay, 1998, https://wiki.c2.com/?AlanKayOnMessaging
"I'm sorry that I long ago coined the term "objects" for this topic because it gets many people to focus on the lesser idea."
— Alan Kay, 2014, http://lists.squeakfoundation.org/pipermail/squeak-dev/1998-October/017019.html
Unfortunately though, by that time the world had mostly put Functional Programming, and landmark computer science theories such as Carl Hewitt's Actor model theory (which were partly inspired by Smalltalk '72) aside, and progressed the "OO super-soldier" ideas of Objective-C and C++ and mutated them into the uncontrollable "Hulk" of Java. (a language which once had so much kudos that another mostly unrelated language was named after it, and mutated to look like it).
"Object-oriented programming, for instance, was seen by one adherent as a way to impose the traditional workplace relationships of manufacturing. Taylorism was implicit in all the proposed solutions to the crisis."
— Hsu, Cornell University, 2020
By the time the corporate sponsors and their agents had finished with computer magazine articles and adverts, university course sponsorships, and mass marketing of their consultancy services behind the languages, the idea that OO was the de facto way to code had become ubiquitous. That word will back later in this article.
There are exceptions: Managing a handler for a side-effect should (for me) pretty much always be architecture similar to an "object + methods" because it just works wonderfully (e.g. DB driver code, wrapped in a Class or Struct + Methods in Rust) because there is an exclusive association between the code and datatype, in that case. But for flexible, sane, maintainable architecture for concurrent, distributed systems, (meaning mass interconnectivity of functionality) there are less troublesome, more flexible, elegant, and testable alternatives (for examples, please go read Carl Hewitt and argue with him and others like the creators of Erlang, and not me, please. Or better, first watch the excellent Brian Will and Richard Feldman videos linked below).
"I think the lack of reusability comes in object-oriented languages, not functional languages. Because the problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.
If you have referentially transparent code, if you have pure functions — all the data comes in its input arguments and everything goes out and leave no state behind — it’s incredibly reusable."
— Joe Armstrong on software reusability.
Thankfully, the pain of the "taught" way of OO coding is finally leading the world to rediscovery of FP benefits (even Java has an FP/OO hybrid successor - Kotlin) and alternative practices and architectural approaches. Mozilla and the Rust core-team thankfully chose an alternative architectural approach when making Rust: Traits. This was a stroke of genius in my opinion as it has resulted in (very amusingly) people migrating towards FP architectural practices without even knowing they are doing this, often whilst complaining about FP.
As a simple intro, Rust devs can create data structures called Structs and associated functions and/or methods (with an explicit self reference) with those Structs. This however has limitations when creating abstractions and eventually leads to the discovery of Rust Traits. Traits are akin to interfaces in some OO languages. A trait usually consists of function signatures that need to be expressed by a struct claiming to express that trait. In this way they are like a set of interrelated behaviours for some unknown type. Not quite a pure function but a constrained function, with expectations of the type that will be transformed by the function.
So what? Well... it means we can:
- Create a set of pure functions that process data, or we can...
- define a set of methods as behaviours for some generic type with some expected traits, or...
- we can create a known Struct with a set of traits and expressed functions for those traits, specifically designed for that Struct - a little like an Object + methods - but with the trait we retain the principle of the struct - or "object" - being open for extension - one of the first five principles of object oriented design, all of which can still be expressed in Rust without permanently coupling data to code.
This lack of opinion in the features of the language means this is enough flexibility (for example) to build applications with the hailed "functional core, imperative (OO) shell" architecture, and yet provide the option to lean on the safety of Rust (see Part 3) when coding this way. In this way Rust makes for an incredible language that allows for a wide scope of architectural styles. It says "don't be afraid! If you love the pain of early architectural assumptions that OO enforces via the myth of encapsulation, you can still feel that hurt with Rust!". ;)
More Info:
Sun preps 500m Java brand push
Object-Oriented Programming is Bad, Brian Will, Youtube, 2016
Why isn't functional programming the norm? Feldman, Youtube 2019
Connections between the Software Crisis and Object-Oriented Programming, Hansen Hsu
No Silver Bullet, Brooks, 1987
Functional Core, Imperative Shell, destroyallsoftware.com, 2012 by @garybernhardt
Type safety
Static Types
In some of the few studies carried there are observed benefits to statically typed languages. If you haven't heard of Quorum, check them out - they are bringing evidence to software development - an industry where our practices were prescribed by managers and shaped from a desire to deskill and control developers (see also, Hsu, above) rather than via gathering evidence of best practice.
For developer convenience in IDEs static typed languages win all day long, as they do when it comes to automated error checking (e.g. functional signatures) in code. Typing is so important to some developers that JavaScript has an entire superset language created around it that enforces types (cheers Microsoft - 👍 you did a good. I don't get to say that often).
Dynamic Types
Look I'm not selling the point - I love me some JavaScript when I'm playing around or I'm guiding friends/relatives to learn code. But if I want to create a financial system/trading bot etc, in that context, explicitness rules. I don't want to second guess type inference anywhere. But I really like dynamic types; enter Rust Generics and Traits...
Rust Generics
The generics system in Rust allows for the ability to specify the Traits (see section above) that the the "type" expresses. In simplistic terms this means we can use dynamic types (a generic type) but restrict those types to those that express a specific set of Traits. It often feels like the best of both worlds. It can create some complexity in areas but in general it works extremely well for modelling complex abstractions.
More Info:
Human factors evidence in language design, Quorum
Software as a Labour Process, Ensmenger & Aspray, 2002
TypeDevil: Dynamic Type Inconsistency Analysis for JavaScript, Pradel, Shuh & Sen
The Plot to Deskill Software Engineering, Glass, Communications of the ACM, 2015
Declarative Programming and Functional Composition features
There are common problems with attempting to encapsulate the world as objects and behaviours all the time (see Separation of Code and Data, above). Once those behaviours start mixing, the variations require more and more complex objects with multiple parents and specific variations of methods, often leading to a lot of duplication of code and code that is difficult to maintain when changes need to be applied (like hybrid objects that facilitate/abstract communication between completely unrelated objects) - The way a method expresses behaviour (the knowledge of it) is spread throughout the 'children' of the objects (it's not very DRY code and it's not very flexible). There's an entire industry of rules and patterns to help deal with this mess.
Functional Programming uses a paradigm that helps to solve these types of scenarios in a rather elegant way: Functional Composition. Generic, "pure" functions replace methods-attached-to-data and instead "process" data that conforms to defined characteristics. This way every inherited variation of the data structure can be processed by a set of functions that describe behaviours, and those behaviours can then be composed to create more complex behaviours. But each behaviour is only expressed once (#極度乾燥). You have probably used these approaches already if you have played with Iterators or Streams.
How does Rust facilitate this? With Generics, Traits, the Fn() and related types, closures, and iterators. Most importantly, these features are not pushed into your face with enforced interior immutability like other FP languages. Rust instead provides a buffet of functional features that one may or may not choose to use - but it's a heavily stacked buffet, leading to two outcomes: Experienced FP devs generally won't go hungry, and people new to FP can learn, experiment, and introduce architectural changes gradually.
A small example that will make sense to FP devs but may irritate or confuse people new to the FP paradigm: By default all variables are immutable and functions all implicitly return their "final value", or the result of their final expression, as long as there is no semi-colon ending the expression. If there is no value (if there is a semi-colon that ends the last expression), the function will still return an empty tuple. Taken together this means Rust functions (by default, without usage of the mut keyword) have no internally mutable variables. They process some input, and output a new value/s based on that input. If we then remove the usage of mut on the params input to the function, we have an FP style data-processing function that cannot mutate it's input or it's internal variables, and thereby results in an idempotent function (always produces the same effect as the first time it's run).
How can these be used with traditional loops and if/else branching? Well that would be clumsy - so Rust also contains high level abstractions like map, filter, fold, and other reducers that allow the building of iterators that apply functions; hello declarative programming! Structured correctly, this raises the possibility of creating highly optimised branchless code - but branchless code that is still readable!
If this sounds alien to your "normal" coding world, it's likely you are coming from an imperative/object-oriented programming paradigm - and that's fine. Even without knowing the definition of these mysterious words: We can add mut to function params, and mut in front of variables, and all the usual if/else/for/while tools are present.
In this way Rust uniquely exists as a bridging or transition tool from one paradigm to the other.
More info:
https://www.freecodecamp.org/news/imperative-vs-declarative-programming-difference/
https://www.fpcomplete.com/blog/2018/10/is-rust-functional/
https://github.com/hemanth/functional-programming-jargon
https://hbfs.wordpress.com/2008/08/05/branchless-equivalents-of-simple-functions/
That's it for Part 1. In Part 2 I'm going to focus on:
- Evolved package management
- Concurrency ready std-lib and great async support
- Clear, helpful compiler errors
- Zero cost abstractions (Optimisation)
- Energy efficiency
- WASM
- Tauri - High performance, secure, cross-platform desktop computing
Top comments (0)