Getting Funky With Filters

Some time back, I was playing around with filters. Again, the question had been asked, "can I filter an array of objects by more than one property? The answer, of course, is yes. There are two main ways of doing this, using Array.prototype.filter(), and we'll start by talking about both.

Let's say, for the sake of argument, that we had a sample data JSON object of 1000 users. If you want to make one to play around with, I found a GREAT data-fakery site that created a 1000-user JSON object for me, over at Mockaroo . It's pretty customizeable, and for this, it worked WONDERS. Here's a sample of one of the objects:

{
    "id": 617,
    "first_name": "Orly",
    "last_name": "Ilieve",
    "email": "oilieveh4@bloomberg.com",
    "job_title": "Structural Analysis Engineer",
    "age": 75,
    "gender": "Female",
    "language": "Fijian",
    "favorite_color": "Crimson",
    "last_login": "7/19/2019",
    "online": true
  },

So we have a users array, containing 1000 of those. Suppose we wanted to filter by language=="Arabic" and also online==true. We could do this easily, using Array.prototype.filter(), in one of two ways:

First, we could filter once, then reduce the filtered array by filtering again.

const arabicSpeakers = users.filter(user => user.language=="Arabic");
const arabicOnline = arabicSpeakers.filter(user=> user.online===true);

Or in a single pass:

const arabicOnline = users.filter(user=>user.language=="Arabic").filter(user=>user.online);

And that works great. I did shorthand the second filter, as the filter function is checking for a true/false value - and if the user is online, we simply return that true.

The downside is, if there are a LOT of records returned by the first filter, then we're repeatedly touching all those records... twice. filter() is not a fast alternative to a simple for(...) loop, in fact it has considerable overhead - particularly if the dataset is massive.

So the second option: we could check all the object properties at once, simply filter for more than one property. This requires that we understand a little about javascript's logic operators, as we'll use || or && (logical OR and logical AND, in order), but let's see how that would look:

const arabicOnline = users.filter(user=>user.language=="Arabic" && user.online )

That is considerably shorter, and loses the overhead of touching multiple records every time. That does the trick! But...

Here we go.

What if we wanted to change that filter some? Suppose we wanted to get all users currently online who spoke Arabic, or who were women who like the color Crimson? Well, that gets a little more complicated, but if we parse it out, we get something like:

if user.online AND (
   user.language=="Arabic" OR (
     user.gender=="Female" AND 
     user.favorite_color=="Crimson"
   )
)

We will use the parentheses in our function, but the use of conjunctions will change:

const filteredList = users.filter(user => user.online && (user.language=="Arabic" || (user.gender=="Female" && user.favorite_color="Crimson" ) );

And that can get tedious. But there are TWO points I'm making here. First, we can do really complex filters on arrays of objects, and they work just fine. And second, there's got to be a better, cleaner way. And that's what this post is about!

Teeny tiny bites

So, the point of all this is... functional programming. I was so excited last week, got all crazy excited, because I was building these insanely massively nested filter queries, and it was WORKING. Blew my mind, and I was so excited about sharing it. Until I actually shared it.

The folks I chose to share with are professionals, both peers and those I consider to be mentors of mine. And, while they thought it was neat, they were just not as excited as I was. And that bothered me, for quite a while, until I realized - they weren't excited because it's a fundamental thing.

There are things that are so simple that, when we realize them, we wonder that they weren't common knowledge before. But they likely were. The issue isn't the knowledge, then, but that I may not be part of the "tribe" where that knowledge is common.

So, if you're one of those who already know this, great! This may just be a refresher for you. And that's okay. But for the rest of the class, this can be eye-opening.

So let's jump back a bit. We have a number of filter functions, right? If we look back at that last compound filter, there are four basic conditions we check for:

• Is user.online===true?
• Is user.language==='Arabic'?
• Is user.gender==='Female'?
• Is user.favorite_color==='Crimson'

Those are the basic conditions. Each of those can be run as unique, independent functions, each can be used as a filter, and each is testable in isolation. And (and here's the power of functional programming) each is composable.

What does this mean? Let's start by writing out each of those four as a function in themselves:

const isOnline = user => user.online;
const isArabic = user => user.language === 'Arabic';
const isFemale = user => user.gender==='Female';
const likesCrimson = user => user.favorite_color==='Crimson';

And that works fine. Each one is testable against an array of objects, each returns a true or false based on the user's property matching as we want, each does what it says on the box. Incidentally, in terms of conventions, functions that return a true or false are called predicate functions (as in "decisions are predicated on this thing").

But it's ugly, and inelegant. I don't like it. Nope. Nuh-uh.

Why not? Because each line does the exact same thing: given a property, find a given matching value on some object. They all do the same thing. So we're repeating code unnecessarily. What can we do? We can step back one step further. We can abstract that out, by writing a generic function, which we'll call filterByProp(). I use the full name like that, because I also have a sortByProp(), a findByProp() and a reduceByProp(), all of which use the same basic idea: given an array of objects, work with a given property. Here's how this one might look:

const filterByProp = (prop) 
  => (val) 
    => (obj) => obj[prop]===val;

That's it. That's the whole shebang. We start by calling filterByProp() with the property name we want to filter by, and we get a function back. That function is waiting for a value to which we compare that property. So we call the function we just got back, passing a value, and we get another function back, that's waiting for the object itself. This third function? That's the one that our filter function can consume.

And here's one way to look at using it:

const byOnlineStatus = filterByProp("online");
// byOnlineStatus is now a function, waiting to be
//  given a value to match that property against.

const isOnline = byOnlineStatus(true);
// and isOnline is now the same function as we wrote
//  above: isOnline = (user) => user.online===true;

What we've done here is functional currying. We've started with a function, and passed in a value, and got back a function awaiting a second value. When we pass in that second value, we get a function awaiting the third, the final object against which we'll check. A shorthand version of the same isOnline() might look like this:

const isOnline = filterByProp("online")(true);
const isArabic = filterByProp("language")("Arabic");
const isFemale = filterByProp("gender")("Female");
const likesCrimson = filterByProp("favorite_color")("Crimson");

Again, they each work in isolation, they can each be tested, both in isolation and in integration, they are tidy, and they are elegant.

When I speak of elegance here, I mean that, within our filterByProp() function, I have zero knowledge of what's going on inside our object. I don't hard-code into the function itself what those properties or values might be, I simply create a function that says "Hey, I've got some object - tell me if it has this property-value pair". It's object-agnostic.

So with those pieces, we could now do:

const filteredList = users.filter(
  user => isOnline(user) &&
         (isArabic(user) ||
           ( isFemale(user) && likesCrimson(user) 
         )
       );

Much more succinct, much more readable, our filter has become... well wait. Not so much. In order to compose our functions, we need to call them all inside our outer function. The way we've done this, you'll note that each of our inner filter functions is being called independently. Not so pretty, not so well documenting.

What to do, what to do...

Here's my thinking: what if we had a mechanism that would let us compose those functions into larger pieces, each of which can simply be plugged in as a filter function?

To do this, we'll need functions that combine our filters, and we want to combine them in two different ways: we want to replace the && with a function we'll call and(), and replace the || with a function we'll call or(). Each of these should take multiple functions as parameters, and return a function that checks if all the conditions of those functions are met (in the case of and), or if some of them are met (in the case of or). So let's dive in:

// So we pass in any number of parameters, and we turn
//  that into an array of funcs. We want every func in
//  that array to return true for a given object.

const and = (...funcs) => obj => funcs.every(func => func(obj) ) 

// now to use this, we can combine our functions, taking
//  (isFemale(user) && likesCrimson(user) ) and turning it to:
const isFemaleAndLikesCrimson = and(isFemale, likesCrimson);

// The or function is exactly the same, except for the function
//   we use on the array of funcs:
const or = (...funcs) => obj => funcs.some(func => func(obj) );
// Here, we are saying "if one or more of these pass, I'm good!"

// with this one, we can combine the next level out: We've gone from
//  (isArabic(user) || (isFemale(user) && likesCrimson(user) ) )
// to
//  (isArabic(user) || isFemaleAndLikesCrimson)
// so next we simply:
const isArabicOr_IsFemaleAndLikesCrimson = or(isArabic, isFemaleAndLikesCrimson);

// and, for the final piece of our complex filter function:
const isOnlineAnd_IsArabicOr_IsFemaleAndLikesCrimson = and(isOnline, isArabicOr_IsFemaleAndLikesCrimson);

Note that I used the underscores simply to denote groupings, but they're not necessary. I worked from the innermost combination out, composing larger and larger functions that, as a final result, return the exact same data type as the composed functions. Why does that matter? They become interchangeable. Our ridiculously-long-named function can now be used as a filter function's callback:

const filteredList = users.filter(isOnlineAnd_isArabicOr_isFemaleAndLikesCrimson);

So that function we pass into the filter function is now complete and self-documenting. We can see at a glance what we're filtering for (though I would really prefer a way to delineate the separation between ors and ands - if anyone has suggestions, I'm open to them!), we can read what's happening, we can test each smaller part in isolation, and each larger part as we grow, and we have a great start for a functional toolkit.

I suppose, if we wanted to keep it nice and legible, we could back it up a small step:

const filteredList = users.filter( 
  and(isOnline, 
    or( isArabic, 
      and(isFemale, likesCrimson)
    )
  )
);

That is a more readable version. The outermost and(...) is the function actually being called by the filter(...), and that one is simply composing its two parameters into an array on the fly, and so on, down the line.

What's the takeaway?

Good question, that. First, that we can compose small, simple functions into very complex ones that still do simple things. Second, that those small parts, while unaware of things outside themselves, can be very powerful with those things of which they are aware. And third, that we have the beginnings of a functional library:

// This is a great tool to add to a functional library, and one you can use over and over for other situations.
const filterByProp = (prop) => (value) => (obj) => obj[prop]===value;

// the two functions, and() and or(), are powerful composers.
const and = (...funcs) => (obj) => funcs.every(func => func(obj) );
const or = (...funcs) => (obj) => funcs.some(func => func(obj) );

There are others, we'll continue to grow this collection, but this is a good place to start.