In this article we’ll be talking about Serde, how you can use it in your Rust application as well as some more advanced tips and tricks.
What is serde?
The serde
Rust crate is used to efficiently serialize and deserialize data in many formats. It does this by providing two traits you can use, aptly named Deserialize
and Serialize
. Being one of the most well-known crates in the ecosystem, it currently supports (de)serialization to over 20 types.
To get started, you’ll want to install the crate into your Rust application:
cargo add serde
Using serde
Deserializing and Serializing data
The simple way to serialize and deserialize data is by adding the serde derive
feature. This adds a macro that you can use to implement Deserialize
and Serialize
automatically - you can do this with the --features
flag (-F
for short):
cargo add serde -F derive
Then we can add a macro to any struct or enum that we want to implement Deserialize
or Serialize
for:
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize)]
struct MyStruct {
message: String,
// ... the rest of your fields
}
This allows us to use any crate with serde
support to convert between said formats. As an example, let’s use serde-json
to convert to and from JSON format:
use serde_json::json;
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize)]
struct MyStruct {
message: String,
}
fn to_and_from_json() {
let json = json!({"message": "Hello world!"});
let my_struct: MyStruct = serde_json::from_str(&json).unwrap();
assert_eq!(my_struct, MyStruct { message: "Hello world!".to_string());
assert!(serde_json::to_string(my_struct).is_ok());
}
If you’re interested in using serde-json
for your Rust application, we have an article talking about JSON parsing libraries which you can check out here.
We can also deserialize and serialize to/from many sources including from a file stream I/O, a JSON byte array and more!
Implementing Deserialize and Serialize manually
In order to better understand how serde
works under the hood, we can also implement Deserialize
and Serialize
manually. This is quite complicated, but for now we will stick with a simple implementation. Here is a simple implementation for serializing an i32
primitive type:
use serde::{Serializer, Serialize};
impl Serialize for i32 {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
serializer.serialize_i32(*self)
}
}
To be able to convert the type, serde
internally requires us to use a type that implements Serializer
. To implement Serialize
for a type that isn’t directly a primitive, we can extend this by serializing into a primitive, then converting into whatever type we want from the primitive. If we want custom serialization for structs, we can also do the same with use of the SerializeStruct
trait:
use serde::ser::{Serialize, Serializer, SerializeStruct};
struct Color {
r: u8,
g: u8,
b: u8,
}
impl Serialize for Color {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
// 3 is the number of fields in the struct.
let mut state = serializer.serialize_struct("Color", 3)?;
state.serialize_field("r", &self.r)?;
state.serialize_field("g", &self.g)?;
state.serialize_field("b", &self.b)?;
state.end()
}
}
Note that to serialize a field, the field type also needs to implement Serialize
. If you have a custom type that doesn't implement Serialize
, you will either need to implement Serialize
or use a Serialize
derive macro (if the struct/enum type holds types that all implement Serialize
).
The Deserialize
trait is a little bit different and is a fair bit more complicated to implement. To be able to deserialize to a type, the type itself needs to implement Sized
which means that there are a number of types which can’t use this trait (for example &str
) because they are unsized types. To deserialize a type, you also need to use a type that implements the Visitor
trait.
The Visitor
trait uses the Visitor design pattern in Rust. This means that it encapsulates an algorithm that operates over a collection of same-sized objects. It allows you to write multiple different algorithms for operating over the data, without needing to change any original functionality. You can find out more about this here.
Below is an example for a MessageVisitor
type that attempts to deserialize multiple types to String:
use std::fmt;
use serde::de::{self, Visitor};
struct MessageVisitor;
impl<'de> Visitor<'de> for MessageVisitor {
type Value = String;
fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
formatter.write_str("A message that can either be deserialized from an i32 or String")
}
fn visit_string<E>(self, value: String) -> Result<Self::Value, E>
where
E: de::Error,
{
Ok(value)
}
fn visit_str<E>(self, value: &str) -> Result<Self::Value, E>
where
E: de::Error,
{
Ok(value.to_owned())
}
fn visit_i32<E>(self, value: i32) -> Result<Self::Value, E>
where
E: de::Error,
{
Ok(value.to_string())
}
}
As you can see, the implementation is quite large! However, it also allows us to make the implementation much simpler. By implementing the Visitor
trait, we can pass the type that implements it to our Deserialize
method and then deserialize JSON into our struct:
use serde::{Deserialize, Deserializer};
impl<'de> Deserialize<'de> for MyStruct {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
// note: don't use unwrap in production!
let message = deserializer.deserialize_string(MessageVisitor).unwrap();
Ok(Self { message })
}
}
There is also documentation on deserializing structs which you can find here. However, generally speaking it is recommended that you use the derive
feature macros as the manual implementation (as seen on the page itself) is quite large. The implementation involves mostly using a visitor that can visit a map or sequence then iterate through the elements to deserialize it.
Using serde attributes
When it comes to serde, the crate also has a number of useful attribute macros that we can use on our types to allow things like field renaming when deserializing a field or serializing to a struct. One of the best examples of this would be when you’re interacting with an API written in a language that may have a key that is a reserved keyword in Rust. You can add a #[serde(rename)]
attribute macro like so:
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize)]
pub struct MyStruct {
#[serde(rename = "type")]
kind: String
}
This allows you to get around the issue!
You can also rename all of your fields to another casing by using the rename_all
attribute:
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct MyStruct {
my_message: String
}
Now when you serialize this struct, my_message
should automatically turn into myMessage
! Perfect for working with APIs written in other languages or with different conventions.
If you’d prefer to not wrap fields in Option
, you can also implement default values by using #[serde(default)]
. This simply allows fields to be filled in with default values instead of automatically erroring out. You can also use #[serde(default = "path")]
to be able to point to functions for providing the automatic default. For example, this struct and function:
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize)]
pub struct MyStruct {
#[serde(path = "my_function")]
my_message: String,
}
fn my_function() -> String {
"Hello world!".to_string()
}
serde
also offers other useful attributes, like being able to deny unknown fields using #[serde(deny_unknown_fields)]
on top of the struct. This allows you to make sure that the struct is exactly as-is when serializing and deserializing.
Deserializing and Serializing enums
Let’s examine this enum type:
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize)]
enum MyEnum {
Data { id: String, data: Value },
SomeOtherData { id: i32, name: String }
}
Note that when converting to and from this enum, it can take two options:
- A String field named
id
and a JSON value with the keydata
(this can be a map, a value or anything that theJson
value can hold) - An
i32
field namedid
and aString
field namedname
You can then match the enum variant for further processing.
When the first enum variant is written in JSON, you can see that it should correspond with this:
{
"Data": {
"id": "your_id_here",
"data": { .. }
}
}
This type of data is “externally tagged” - meaning the data is characterized by the identifier being on the outside of the JSON object. We can add inline tagging so that the identifier is on the inside of the crate - let’s have a look at what this would look like:
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize)]
#[serde(tag = "type")]
enum MyEnum {
Data { id: String, data: Value },
SomeOtherData { id: i32, name: String }
}
Now the JSON representation looks like this:
{
"type": "Data",
"id": "your_id_here",
"data": { .. }
}
Interested in reading more? The serde
documentation has a page on tagging which you can find here.
Crates that work well with Serde
serde_with
serde_with
is a crate that provides custom de/serialization helpers to use with serde
's with
annotation. Normally, you can define a module for the (de)serializer to use that follows a custom module for custom (de)serialization:
#[derive(Deserialize, Serialize)]
pub struct MyStruct {
#[serde(with = "my_module")]
my_message: String
}
When using serde_with
, it works by replacing the with
annotation with a new one called serde_as
. With this new attribute macro you can do quite a few things:
- De/serializing a type using
Display
andFromStr
traits. - Support arrays larger than 32 elements.
- Skip serializing empty Option types.
- Deserialize a comma separated list into a
Vec<String>
.
To use serde_with
, you need to add it to your Cargo.toml either manually or by using the following command:
cargo add serde_with
Then you need to add serde_as
to the type you want to use it for, like so:
use serde_with::{serde_as, DisplayFromStr};
#[serde_as]
#[derive(Deserialize, Serialize)]
struct MyStruct {
// Serialize with Display, deserialize with FromStr
#[serde_as(as = "DisplayFromStr")]
my_number: u8,
}
This struct lets you convert to/from a string but have the type itself in your Rust struct be u8
! Pretty useful, right?
This crate also comes with a guide that you can use to fully capitalise on serde_with
. Overall, a strong companion crate for serde
.
serde_bytes
serde_bytes
is a crate that allows for optimised handling of &[u8]
and Vec<u8>
types - while serde
is capable of dealing these types by itself, some formats can be de/serialized more efficiently. It’s quite simple to use - you just add it to your Cargo.toml and then add it via the #[serde(with = "serde_bytes")]
annotation like so:
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize)]
struct MyStruct {
#[serde(with = "serde_bytes")]
byte_buf: Vec<u8>,
}
Overall, an easy to use and simple crate that improves performance without much knowledge required.
Finishing up
I hope you enjoyed reading about Serde! It's a pretty powerful Rust crate and forms the backbone of most Rust applications.
Read more:
Top comments (0)