Use the well known RxJS operators to manipulate arrays or iterables in a blazing fast way using the new tiny library rxjs-transducer
Most JavaScript developers by now have learned to use Array’s builtin methods like filter
, map
, reduce
, some
, and every
to manipulate arrays of data in a functional programming style. This style of programming has the advantage of being easier to reason about than imperative loop style programming. It does have a number of drawbacks, however:
The operations only work for arrays, not iterables.
There is a fairly limited set of operations. Notable omissions are operations like
take
,skip
,first
,last
,single
.Bad performance: When chaining multiple operations, they will each create an intermediate array and thus iterate the array as many times as there are operators. E.g:
Will create 3 intermediate arrays and then iterate the last array to reduce it to a string, a total of 4 array iterations. Not exactly efficient. Of course this is not an issue when the source contains 10 or 100 elements. But if there are millions of elements, it could be a problem.
Using RxJS operators instead
When thinking of RxJS, you usually think asynchronous stream processing, but RxJS will in fact process streams synchronously when possible. This means that we can use RxJS to create a synchronous stream of values from an array or other iterable using the from function:
This snippet will only iterate the array once, transforming and filtering values as it goes along. It is however a bit clunky to have to use the from, pipe, and subscribe keywords, so I have written a small transducer library that reduces the snippet above to:
The cool thing about this transducer is that it supports iterables such as infinite sequences so you can do stuff like:
Furthermore it is written in TypeScript, so it will give you full TypeScript support in your IDE:
Performance
So, how does it perform you say?
Let us make a test using console.time where we map, filter and reduce an array of 10,000,000 random numbers:
310ms vs 47ms! So in this case the rxjs-transducer is more than 6 times as fast as standard array chaining. Your mileage may vary, but in almost all cases it will be quite a lot faster than array chaining.
How to get it
npm install rxjs-transducer
(< 1KB GZipped)
Check out the code on GitHub:
rasmusvhansen / rxjs-transducer
A transducer implementation using the excellent operators from RxJs
rxjs-transducer
A transducer implementation using the excellent and well known operators from RxJS The benefits are:
- Performance: Doing a array.map().filter().reduce() causes the array to be iterated 3 times. Using rxjs-transducers, the array is only iterated once. Doing a
filter().map().Math.max()
on an array with 1,000,000 items is roughly three times as fast with the transducer as with normal array operations. - Ability to work with lazy and infinite collections (generators)
- Access to a huge library of well tested operators from RxJS such as
map
,filter
,reduce
,skip
,take
,takeWhile
, and many others - Full TypeScript support
Installation
npm install rxjs-transducer --save
Usage
TypeScript / ES6
import { transducer } from 'rxjs-transducer';
import { map, filter, reduce, skip, take } from 'rxjs/operators';
const source = ['a', 'ab', 'abc', 'abcd', 'abcde'];
// Works as standard array
…Playground
I have created a StackBlitz playground for you to try it out in your browser:
Let me know what you think.
Top comments (9)
Arguably,
take
is present in the form ofslice
, while the lack offirst
with an array is a non-issue. The chaining performance was already improved here: dev.to/danielescoz/improving-javas.... Using RxJS for that task is an interesting approach, though.Nice one, is this the best implementation of Transducers in TS(or even all of JS) right now, or only popular because of it's part of RxJS?
In any case, two questions:
Hi, thanks for replying. I only just created the library a few days ago, so I wouldn't say it is popular (yet) :-)
But if you have a project where you are already using RxJS and are already familiar with the operators, I think it has a low entry barrier.
new Map(transducer(xxx)(yyy))
which has a performance penalty of first allocating the array.Thanks for the feedback!
My apologies.
I assumed it was a not-even-recently-added part of the ancient RxJS behemoth.
Guess that's what I get for not clicking the I have written a transducer function link :)
I see this post was more of a camouflaged release announcement than a tutorial for an established package. 😉
I think that the line that sent me on this path the most was
Oh. I will make it more clear that this is a new library, and make it less camouflaged :-)
The first two code examples are mixed up.
It shows the transducer pipe first, then the native ES5 Array methods.
Yes, I noticed that too, but when checking the Markdown, the links to the gists are correct. Seems like a glitch in dev.to?
I created an issue here: github.com/thepracticaldev/dev.to/...
😱
This is great!