Rust has macros similar in purpose to C++’s macros/templates in that it’s code that runs at compile time that itself writes code (meta-programming). However, the similarities pretty much end there. There are two types:
- Declarative macros (aka “macros by example” or
macro_rules!
macros) - Procedural macros. Which come in 3 varieties:
- Function-like: e.g.
custom!()
- Derive mode: e.g.
#[derive(CustomMode)]
(applied tostruct
andenum
) - Attribute: e.g.
#[CustomAttribute]
(applied to almost anything)
- Function-like: e.g.
This is a quick primer on derive mode procedural macros.
Use Case: FFI to NNG
Quick summary of the problem we’re trying to solve.
Problem
NNG has a straight-forward C API. Instead of the function overloading you would find in C++ (or many other languages), there’s a collection of functions for each type. The nng_socket
type has get and set option functions:
int nng_getopt_bool(nng_socket s, const char *opt, bool *bvalp);
int nng_getopt_int(nng_socket s, const char *opt, int *ivalp);
// ...
int nng_setopt(nng_socket s, const char *opt, const void *val, size_t valsz);
int nng_setopt_bool(nng_socket s, const char *opt, int bval);
// ...
Similar functions exist for the nng_dialer
type:
int nng_dialer_getopt_bool(nng_dialer d, const char *opt, bool *bvalp);
int nng_dialer_getopt_int(nng_dialer d, const char *opt, int *ivalp);
// ...
int nng_dialer_setopt(nng_dialer d, const char *opt, const void *val,
size_t valsz);
int nng_dialer_setopt_bool(nng_dialer d, const char *opt, bool bval);
// ...
And likewise for nng_listener
and nng_pipe
.
Wrapping the bindgen generated functions in our runng-sys crate is easy:
impl GetOpts for NngSocket {
fn getopt_bool(&self, option: NngOption) -> NngResult<bool> {
unsafe {
let mut value: bool = Default::default();
NngFail::succeed(nng_getopt_bool(self.socket, option.as_cptr(), &mut value), value)
}
}
fn getopt_int(&self, option: NngOption) -> NngResult<i32> {
unsafe {
let mut value: i32 = Default::default();
NngFail::succeed(nng_getopt_int(self.socket, option.as_cptr(), &mut value), value)
}
}
// ...
}
Here we’re already using helpers to deal with the boiler-plate:
-
NngFail::succeed()
turns the C return value into one for Rust (int -> Result<T>
) -
.as_cptr()
turns anenum
(with a&'static [u8]
) into a Cconst char *
naming the option (NngOption -> *const i8
)
We started to replicate this mass of code for nng_listener
, nng_dialer
, et al. But let’s be honest, copy-pasting code is one of the most dubious code smells.
Solution
Using Rust’s derive mode macros, we’d like to write something like:
#[derive(NngGetOpts,NngSetOpts)]
#[prefix = "nng_"]
#[nng_member = "socket"]
pub struct NngSocket {
socket: nng_socket,
}
Which would substitute the prefix
and nng_member
values into #PREFIXgetopt_bool(self.#NNG_MEMBER, ..
to form nng_getopt_bool(self.socket, ..
. Effectively source code string substitution with an end result similar to:
impl GetOpts for NngSocket {
fn getopt_bool(&self, option: NngOption) -> NngResult<bool> {
nng_getopt_bool(self.socket, ..)
}
// ...
}
impl SetOpts for NngSocket {
fn setopt_bool(&mut self, option: NngOption, value: bool) -> NngReturn {
nng_setopt_bool(self.socket, ..)
}
// ...
}
Derive Mode Procedural Macros
Start a new library module with cargo new --lib runng_derive
and in Cargo.toml
:
[package]
name = "runng_derive"
# ...
[lib]
# 1
proc-macro = true
[dependencies]
# 2
syn = "0.15"
quote = "0.6"
Notes:
-
proc-macro = true
is needed otherwise get:error: the `#[proc_macro_derive]` attribute is only usable with crates of the `proc-macro` crate type
In lib.rs
:
#![recursion_limit="128"] // 1
extern crate proc_macro;
extern crate syn;
#[macro_use] // 2
extern crate quote;
use proc_macro::TokenStream;
use syn::{
Lit,
Meta,
MetaNameValue,
};
// 3
#[proc_macro_derive(NngGetOpts, attributes(prefix, nng_member))]
pub fn derive_nng_get_opts(tokens: TokenStream) -> TokenStream {
derive_nng_opts(tokens, gen_get_impl)
}
Notes:
-
recursion_limit
was recommended by the compiler after failing with:error: recursion limit reached while expanding the macro 'stringify'
Bring
quote
crate’s macros into scope (NB: not required in Rust 2018)#[proc_macro_derive]
defines our#[derive(..)]
macro. Within that,attributes(..)
defines our “derive mode helper attributes” (i.e.#[prefix]
and#[nng_member]
).
The input to this function is the stream of tokens of whatever code the attribute is attached to. The output is the new code we want to generate.
First extract our #[prefix ..]
and #[nng_member ..]
attributes:
fn derive_nng_opts<F>(tokens: TokenStream, gen_impl: F) -> TokenStream
where F: Fn(&syn::Ident, String, String) -> TokenStream
{
// Parse TokenStream into AST
let ast: syn::DeriveInput = syn::parse(tokens).unwrap();
let mut prefix: Option<String> = None;
let mut member: Option<String> = None;
// Iterate over the struct's #[...] attributes
for option in ast.attrs.into_iter() {
let option = option.parse_meta().unwrap();
match option {
// Match '#[ident = lit]' attributes. Match guard makes it '#[prefix = lit]'
Meta::NameValue(MetaNameValue{ref ident, ref lit, ..}) if ident == "prefix" => {
if let Lit::Str(lit) = lit {
prefix = Some(lit.value());
}
},
// ...
}
}
gen_impl(&ast.ident, prefix.unwrap(), member.unwrap())
}
Generate the code:
fn gen_get_impl(name: &syn::Ident, prefix: String, member: String) -> TokenStream {
// Create function identifier like `nng_getopt_bool`
let getopt_bool = prefix.clone() + "getopt_bool";
let getopt_bool = syn::Ident::new(&getopt_bool, syn::export::Span::call_site());
// ...
let member = syn::Ident::new(&member, syn::export::Span::call_site());
// Generate the `impl`
let gen = quote! {
impl GetOpts for #name {
fn getopt_bool(&self, option: NngOption) -> NngResult<bool> {
unsafe {
let mut value: bool = Default::default();
NngFail::succeed( #getopt_bool (self.#member, option.as_cptr(), &mut value), value)
}
}
}
// ...
};
gen.into()
}
The quote
macro does the work of taking the source code we provide, substituting in values prefixed with #
, and then generating a TokenStream
to return from our macro.
Debugging
cargo-expand (by the author of the syn
crate) prints out the result of macro expansion and is handy for debugging:
# Nightly toolchain must be installed (it doesn't need to be default)
rustup toolchain install nightly
# Install `cargo expand`
cargo install cargo-expand
# rustfmt for formatting output
rustup component add rustfmt-preview
# Pygments to colorize output
sudo pip install Pygments
# Output macro expansion
cargo expand
Example
One thing to watch out for is the type of the #XYZ
variables. Without the call to syn::Ident::new()
:
let getopt_bool = prefix.clone() + "getopt_bool";
//...
fn getopt_bool(&self, option: NngOption) -> NngResult<bool> {
unsafe {
let mut value: bool = Default::default();
NngFail::succeed( #getopt_bool ( /* ... */ )
}
}
cargo check
:
error[E0618]: expected function, found `&'static str`
cargo expand
:
// ...
fn getopt_bool(&self, option: NngOption) -> NngResult<bool> {
unsafe {
let mut value: bool = Default::default();
NngFail::succeed("nng_getopt_bool"(self.socket,
option.as_cptr(),
&mut value), value)
}
}
// ...
It generated "nng_getopt_bool"(...)
when we need nng_getopt_bool(...)
. Give it a String
and it will give you one right back!
cargo expand
dumps the output for the entire module which might be more than you want. It can be limited to a single test with cargo expand --test name_of_the_test
and there’s some other useful options.
More Helpful Helper
Explicitly naming the field with #[nng_member = "socket"]
is brittle. What we really want is to extract the name of the field based on the placement of the attribute:
#[derive(NngGetOpts,NngSetOpts)]
#[prefix = "nng_"]
pub struct NngSocket {
#[nng_member]
socket: nng_socket,
}
It’s pretty easy to walk through the hierarchy of enums
and struct
s representing the source code:
fn get_nng_member(ast: &syn::DeriveInput) -> Option<syn::Ident> {
match ast.data {
syn::Data::Struct(ref data_struct) => {
match data_struct.fields {
// Structure with named fields (as opposed to tuple-like struct or unit struct)
// E.g. struct Point { x: f64, y: f64 }
syn::Fields::Named(ref fields_named) => {
// Iterate over the fields: `x`, `y`, ..
for field in fields_named.named.iter() {
// Get attributes `#[..]` on each field
for attr in field.attrs.iter() {
// Parse the attribute
let meta = attr.parse_meta().unwrap();
match meta {
// Matches attribute with single word like `#[nng_member]` (as opposed to `#[derive(NngGetOps)]` or `#[nng_member = "socket"]`)
Meta::Word(ref ident) if ident == "nng_member" =>
// Return name of field with `#[nng_member]` attribute
return field.ident.clone(),
_ => (),
}
}
}
},
_ => (),
}
},
_ => panic!("Must be a struct"),
}
None
}
This is a pretty basic implementation; there’s no error handling, doesn’t support #[nng_member]
on a nested field, etc. But, since this macro is just used internally to our crate for educational purposes it’s probably ok for now.
All this is pretty similar to a common use of discriminated unions in F# (and probably every other language with functional tendencies). Supporting more features and robust handling will probably turn into an exercise in recursive match
ing.
Resources
For further information on macros, and the reading order I would recommend:
- Rust Programming Language- basic introduction
- The Reference- basic introduction
-
“Creating an enum iterator using Macros 1.1”- example of
derive
macro - “Debugging Rust’s new Custom Derive System”- overview of development/debugging
syn
examples-
syn-
syn
crate documentation. This is needed to writederive_nng_opts()
when working with the AST. - (Optional) Little Book of Rust Macros- covers
macro_rules!
The last item, the LBRM, was actually one of the first things I read but probably the worst place to start. If nothing else, read the “Macros in the AST” section. It’s a good overview of macros in Rust and was helpful in coercing my mental model (string substitution) to reality (intermediary form used by compiler).
); // End of Story
We’ve barely scratched the surface here. For one thing my slavish use of syn::Ident::new(.., syn::export::Span::call_site())
warrants investigation- it’s merely the first thing that worked. The syn
crate has a lot of stuff.
I’d like to revisit the Little Book of Rust Macros and take a shot at macro_rules!
.
Top comments (4)
Thank you, this was very useful.
As a side note: is there a way I could contact you? Promise nothing fishy, just wanted to discuss life in Cyprus, as I see you're living there and I'm considering moving there. If you don't want to share your email here, there is an email on my github linked on my profile here that you could write to. Much appreciated!
Hmmm looks like there is an errant back-tick somewhere in the markdown.
Yeah, just fixed it....
Great, can’t wait to dive into this post. Bookmarked for later.