Javascript Composable Decoders with Validation

I'm a suspicious person. If I don't find my slippers by my bed, I suspect. If I see a cat wearing a coat and looking at my direction, I suspect. The same way, when I receive data from an external source, I suspect.

Doesn't matter what the Swagger or the specs say, if you are receiving data from an API or any other external source, it is always good to know that it fulfills a format and any business restrictions. This is where TypeScript types stop helping you. Even if there are nice libraries that help cover this gap, like io-ts, you might still miss a good way to trace what failures happened and where.

Here I'll describe how I implemented a Javascript decoder that accumulates the errors of the received data while keeping all needed data after the decoding process. You can see the full snippet here.

The Problem

To illustrate the example, I'll be receiving a list of videogames data, like this:

const videogames = [
  {id: 1, name: 'Doom', genre: 'FPS', rating: 7},
  {id: 2, name: 'NieR: Automata', genre: 'Action RPG', rating: 100},
  {id: 3, name: 'Dead Cells', genre: 'Rogue-like', rating: 8},
]

We will have some restrictions as well, namely:

• The dataset must be an array of videogames.
• A videogame must have an id, a name, a genre and a rating.
• name can't be empty
• genre must be one of our recognised genres (FPS, RPG, Simulator, Strategy, and Platforms).
• rating must be between 1 and 10.

If you are a keen observer, you'll see the example dataset already breaks some of those restrictions. Excellent.

What we want is to be able to parse this kind of datasets, know all the errors that happened so we can inform the user or developer, and keep or throw away invalid data at our convenience.

The Tool to Solve It

In order to do so, I'll use my library Validation. Validation is a monad. A monad is a software pattern for a type/class that has certain functions and certain restrictions. Being a monad means two things:

• It is a wrapper for something (in our case, values) and can be constructed using Validation.of(value) (this is called Applicative).
• It implements "Mappable" or has a map function (this is called Functor), and "Chainable", or a chain function (and this, Chain).

Mapping a monad means applying a function to its values without modifying the container, as if it was an array: [1, 2].map(x => x + 1) === [2, 3]

Chaining a monad means applying a function to its values and changing the container for the one returned by the function. It is also called flatMap because, if you map a function that returns another container and don't flatten the result, you end up with a container inside a container:
[1, 2].map(x => [x, 0]) === [[1, 0], [2, 0]], but
[1, 2].flatMap(x => [x, 0]) === [1, 0, 2, 0]

Validation<E, V> is a monad that can be of two types: Valid and Invalid. E and V here are generic types for the two values that a Validation wraps: Its errors and its value.

A Valid type only stores some data of type V, and affirms it is "valid" so far. It can be constructed with Validation.of, but also using Validation.valid

An Invalid type stores two values: Some invalid data of type V, and a list of errors of type E. It can be constructed using Validation.invalid.

Having all that we can validate a specific rating like this:

const ratingDecoder = rating => (
  isBetween(1, 10)(rating)
    ? Validation.valid(rating)
    : Validation.invalid(rating, `Rating must be between 1 and 10, but received ${rating}`)
)

Here we're returning a Valid(rating) in case rating fulfills the restriction, and Invalid(rating, errorMessage) when rating does not fulfill it.

The unicorn stuff we have here that other validation libraries don't offer is that we're keeping the rating value, even if we know it is invalid, because we may need this information later.

The Plan

Ok, so what's the plan? How are we going to use this to decode all the data?

First, we want to do it recursively, and secondly, with composable decoders that also describe our data shape. For example, our videogameDecoder will be something like this:

const videogameDecoder = videogame => doSomeStuff({ // We'll complete this later
  id: idDecoder,
  name: nameDecoder,
  genre: genreDecoder,
  rating: ratingDecoder,
}, videogame)

This way, videogameDecoder is serving two purposes:

• It is a decoder function that returns a validated videogame.
• It is a declaration of the videogame type shape, like PropTypes. This is also a good documentation when we don't have TypeScript.

We will do this with all levels, all shapes and types. In our case, this is our four attributes (id, name, genre, and rating), our videogame type, and our videogameArray type. Let's begin:

Decoding For Fun

We'll begin with the nameDecoder function. Assuming we have a function isFilled(str) that tells us if a name is non-empty, we can do something similar to the ratingDecoder before:

const nameDecoder = name => (
  isFilled(name)
    ? valid(name)
    : invalid(name, 'name can not be empty')
)

So we'll have to do this with all the attributes. Isn't it a little boilerplate-y? Luckily, Validation comes with several helpers, one of which is fromPredicateOr(errorFn, predicate). It can be used to create a function that will receive our value and return a Validation of it. Let's look at how can we use it:

const nameDecoder = fromPredicateOr(() => 'name can not be empty', isFilled)

Much better! Now, when we call nameDecoder, it will check isFilled and return a Valid or Invalid depending on its truthiness.

Moreover, if another type that is not a videogame needs to have a name that can not be empty, we can reuse this decoder!

We now have all the attributes decoded:

const idDecoder = valid
const nameDecoder = fromPredicateOr(() => 'name can not be empty', isFilled)
const genreDecoder = fromPredicateOr(() => 'genre must be in validGenres', flip(includes)(validGenres))
const ratingDecoder = fromPredicateOr(() => 'rating must be between 1 and 10', isBetween(1, 10))

What about idDecoder? It doesn't have any restriction, so it will be valid always, but we still need to provide a Validation out of it, so we will use the valid constructor directly.

The Videogame Type

The function videogameDecoder that we want will receive a videogame object, validate each one of its attributes, and then group (or reduce) all validations into one single validation:

// Videogame
{
  id: 3,
  name: 'Dead Cells',
  genre: 'Rogue-like',
  rating: 8
}

// Videogame with validated attributes
{
  id: valid(3),
  name: valid('Dead Cells'),
  genre: invalid('Rogue-like', ['genre is not in validGenres']),
  rating: valid(8),
}

// Validated videogame without invalid properties
invalid({
  id: 3,
  name: 'Dead Cells',
  rating: 8,
}, ['genre is not in validGenres'])

Note that, in the last step, we are choosing not to keep the invalid values. It doesn't need to be like that, we can choose to keep them, but we won't need them any longer in this example.

In order to do the first step, we could use the evolve method from Ramda, but we won't because it doesn't apply any function to missing attributes, and we want to detect a missing attribute to say it is invalid.

We could also iterate over the object properties:

const validations = {id: idDecoder, name: nameDecoder, /* ... */}

Object.keys(validations).reduce(
  (acc, k) => ({
    ...acc,
    [k]: property(k, videogame).chain(val => validations[k](val)),
  }),
  {}
)

Note how, in the fourth line, we are using the Validation.property method that returns a Valid if it finds that attribute, and an Invalid otherwise. Then, we chain it to a function that will return a Valid if the validation for that attribute passes, or an Invalid otherwise.

How does that work?

When we chain Validations, it remembers the errors that we had detected previously, and adds them to any new errors. It will behave like this:

valid(1).chain(n => valid(2)) === valid(2)
valid(1).chain(n => invalid(2, ['error'])) === invalid(2, ['error'])
invalid(1, ['error1']).chain(n => invalid(2, ['error2']) === invalid(2, ['error1', 'error2'])

This way, information about the errors is preserved.

Instead of doing it this way, we will use another Validation helper: validateProperties. It does exactly what we wanted:

const videogameWithValidatedProperties = validateProperties({
  id: idDecoder,
  name: nameDecoder,
  /* ... */
}, videogame)

On to the second and last step, we need to iterate over the object properties and add only the properties that are valid. We can check this using Validation.isValid() method, and access the value inside with Validation.value:

const allProperties = obj => (
  Object.keys(obj).reduce((validatedObj, k) => (
    validatedObj.chain(validObj => obj[k].isValid()
      ? Validation.of({...validObj, [k]: obj[k].value})
      : obj[k].map(() => validObj)
    )),
    valid({})
  )
)

However, this is a complex function, and a common enough one to have its own helper as well, Validation.allProperties, so we'll use that.

At the end, we will have our videogameDecoder pretty terse thanks to the helpers:

const videogameDecoder = videogame => {
    const videogameWithValidatedProperties = Validation.validateProperties({
        id: idDecoder,
        name: nameDecoder,
        genre: genreDecoder,
        rating: ratingDecoder,
    }, videogame)
    return Validation.allProperties(videogameWithValidatedProperties)
}

We can improve that if we refactor it using point-free style with the help of the pipe function from Ramda:

const videogameDecoder = pipe(
    Validation.validateProperties({
        id: idDecoder,
        name: nameDecoder,
        genre: genreDecoder,
        rating: ratingDecoder,
    }),
    Validation.allProperties,
)

Validation Arrays

Just as Validation has some helpers to deal with objects, it has other ones to deal with arrays.

As it turns out, these operations are well defined in the functional programming world, because FP loves lists. Enter the Monoid.

A monoid is, just like a monad, another programming pattern (although they don't have much more in common, even if the names look similar). A type is a monoid if it is "Concatenable" and has an "empty" function that returns an empty element.

Therefore, a monoid will always have two functions:

• empty returns the empty element. With arrays, that would be [].
• concat concatenates the values of two monoids and returns another monoid. With arrays, that would be Array.concat.

This means JS arrays are a monoid, and if they had an .empty() method that returned [], they would even be Static Land compliant. But they don't.

Validation, as it turns out, is conveniently a Static Land compliant monoid when the wrapped value is an array (when it is not, it is casted to an array when concatenating). This means we have the full power of the monoids in the palm of our hands.

The List Type

Now for the last function: videogameArrayDecoder. It receives an array of videogames, and returns a Validation of the array.

We can do that in two steps just like before: Validate each of the videogames, then accumulate (reduce) the Validations into a single Validation.

// Array of videogames
[vg1, vg2, vg3]

// Array of videogame Validations
[valid(vg1), invalid(vg2, err2), invalid(vg3, err3)]

// Validation of array of videogames
invalid([vg1], [...err2, ...err3])

Note that, just like before, in the last step we will be dropping the invalid videogames off the list because we want to.

To validate each of the videogames, we can do that with a conventional Array.map like this:

const validatedVideogames = videogames.map(videogameDecoder)

Eezy-peezy. For the second step, we want to reduce the array of validations to a validation of arrays. As we know, Validation acts as a monoid when the values are arrays, so let's map them to one-element arrays:

const toArrayValidation = Validation.map(x => [x])
const videogameArrayValidations = validatedVideogames.map(toArrayValidation)

Now we are ready to concat them, because they contain arrays. Validation.concat method concatenates the valid values, and drops the invalid values, just like we want. This means we can reduce the list like follows:

const videogamesValidation = videogameArrayValidations
    .reduce(Validation.concat, Validation.empty())

This looks awesome because it is the very definition of generating a list with a monoid. It is so awesome it has its own function in the library:

const videogamesValidation = Validation.sequence(videogameArrayValidations)

If we instead wanted to keep the invalid values, we would have to do it another way:

const losslessSequence = l => l.reduce((valList, val) => (
  valList.chain(list => val.map(x => [...list, ...x]))
), Validation.empty())

const videogamesValidation = losslessSequence(videogameArrayValidations)

By using map inside chain, what we are doing is concatenating all values inside the new validation in each iteration, and then chaining it to the original one to keep the errors, because the chain function preserves all the errors.

So how will the decoder look like?

const videogameArrayDecoder = videogames => {
    const validatedVideogames = videogames.map(videogameDecoder)
    return Validation.sequence(validatedVideogames)
}

If we refactor it using point-free style and Ramda, we get this:

const videogameArrayDecoder = pipe(map(videogameDecoder), Validation.sequence)

The Result

Finally, this is the complete code of our whole decoder:

const {Validation, valid, invalid} = require("@rexform/validation")
const {isNil, isEmpty, complement, either, includes, flip, both, lte, gte, pipe, map} = require('ramda')

const videogames = [
    {id: 1, name: 'Doom', genre: 'FPS', rating: 7},
    {id: 2, name: 'NieR: Automata', genre: 'Action RPG', rating: 100},
    {id: 3, name: 'Dead Cells', genre: 'Rogue-like', rating: 8},
]

const validGenres = ['FPS', 'Platforms', 'RPG', 'Strategy', 'Simulator']

const isFilled = complement(either(isNil, isEmpty))
const isBetween = (a, b) => both(flip(lte)(b), flip(gte)(a))

const nameDecoder = Validation.fromPredicateOr(() => 'name can not be empty', isFilled)
const genreDecoder = Validation.fromPredicateOr(() => 'genre must be in validGenres', flip(includes)(validGenres))
const ratingDecoder = Validation.fromPredicateOr(() => 'rating must be between 1 and 10', isBetween(1, 10))

const videogameDecoder = pipe(
    Validation.validateProperties({
        id: valid,
        name: nameDecoder,
        genre: genreDecoder,
        rating: ratingDecoder,
    }),
    Validation.allProperties,
)

const videogameArrayDecoder = pipe(map(videogameDecoder), Validation.sequence)

videogameArrayDecoder(videogames)

And this is the result:

Invalid(
  [{id: 1, name: 'Doom', genre: 'FPS', rating: 7}],
  [
    "genre must be in validGenres",
    "rating must be between 1 and 10",
    "genre must be in validGenres",
  ]
)

The only missing problem is that, when we see the errors, we don't know what videogame produced them. We can fix that if we come back to our videogameDecoder and add the videogame id in the error message (or, instead of the id, the whole videogame object stringified, if we want).

We can use the function mapError to add the id to the error message. The function mapError works like map, but for the wrapped error array instead of the wrapped value. It will only modify each of the errors without changing the Validation:

const videogameDecoder = pipe(
    Validation.validateProperties({
        id: valid,
        name: nameDecoder,
        genre: genreDecoder,
        rating: ratingDecoder,
    }),
    Validation.allProperties,
    videogame => videogame.mapError(e => `In ID=${videogame.value.id}: ${e}`),
)

That's it, now the result will have much more meaning:

Invalid(
  [{id: 1, name: 'Doom', genre: 'FPS', rating: 7}],
  [
    "In ID=2: genre must be in validGenres",
    "In ID=2: rating must be between 1 and 10",
    "In ID=3: genre must be in validGenres",
  ]
)

We finished our new videogame decoder, hurray! 😄

Thanks

If you reached this point, thank you! This is my first article, and I welcome any suggestions or feedback. I hope you learned something out of it, but if you didn't, maybe you can teach me something!

Also, if you liked it, give Validation a try 😉