In this post we will understand what ES2015 generators are in Javascript/Typescript. Generators rely heavily on iterators, so if you don't know or would like to refresh your memory, take a look at my last post.

Introduction

As we know, iterators allows us to have total control of iterating through some structure, we get to decide if and when we get the next element of our iteration sequence, while hiding from our iterator's consumers implementation details of how we get these elements. However, everything has a cost, iterators can be quite tricky to implement since we have to keep track of the states that will control the flow of execution so that we can, for example, mark the iterator as complete.

Generators allows us to easily create iterators, making possible to implement some really cool stuff like stopping execution of functions to resume them later (sounds familiar to async/await?), pass values to the generator between these pauses and more.

The basics

Generators can be quite complicated and somewhat different from what we are used to, so pay close attention to the details. A generator declaration is very similar to a function declaration:

function* fooGen() {
    console.log("Hello from fooGen");
}
function foo() {
    console.log("Hello from foo")
}

You define a generator by using function* fooGen (you can actually do function * fooGen or function *fooGen). This is the only difference between our generator declaration and the declaration of our foo function but they actually behave very differently. Consider the following:

foo(); // Hello from foo
fooGen(); //

Our invocation of foo is as expected, however the invocation of fooGen didn't log anything. That seems odd, but this is the first big difference between functions and generators. Functions are eager, meaning whenever invoked, they will immediately begin execution while generators are lazy, meaning they will only execute our code whenever you explicitly tell them to execute. You may argue "but I ordered it to execute", however calling the generator doesn't execute its code, it only does some internal initialization.

So how do I tell a generator to execute our code? First let's see what fooGen() returns us. If we look at the type of fooGen, we will see the following: function fooGen(): Generator<never, void, unknown>, so let's look at what this Generator type is:

interface Generator<T = unknown, TReturn = any, TNext = unknown> extends Iterator<T, TReturn, TNext> {
    // NOTE: 'next' is defined using a tuple to ensure we report the correct assignability errors in all places.
    next(...args: [] | [TNext]): IteratorResult<T, TReturn>;
    return(value: TReturn): IteratorResult<T, TReturn>;
    throw(e: any): IteratorResult<T, TReturn>;
    [Symbol.iterator](): Generator<T, TReturn, TNext>;
}

Wait, this interface has a next, return and throw methods isn't this an iterator? The answer is yes, but also notice that it is an iterable. So this interface is actually somewhat similar to the IterableIterator interface. If you want to know why they aren't the same, take a look at this question.

To order the generator to execute our code, we only need to call next:

foo(); // Hello from foo
const it = fooGen();
it.next() // Hello from fooGen

Let's return some value from our generator:

function* fooGen() {
    console.log("Hello from fGen");
    return "Bye from fGen";
}

const it = fooGen();
const result = it.next(); // Hello from fGen
console.log(result); // { value: 'Bye from fGen', done: true }
console.log(it.next()); // { value: undefined, done: true }

Notice that when you return something from a generator, it automatically completes the iterator, no need to manage state. Also notice that the value of the return expression is returned only once, subsequent calls to it.next return undefined in the value. Keep in mind that if there is no explicit return statement on your function or if the execution didn't reach a logical branch with the return, then undefined is assumed to be the return value.

The yield keyword

So far we didn't do anything exciting with generators, we just used them as some more complicated functions. As said in the introduction, we can pause the execution of generators. We achieve this using the yield keyword.

The yield keyword pauses the execution of our iterator.
Whenever we call next, the generator will synchronously execute our code until a yield or a return statement is reached (assuming no errors happened, which we will see later). If the generator was in a paused state and we call next again it will resume the execution from wherever it was paused from.

function*  fooGen() {
    console.log("Begin execution");
    yield;
    console.log("End execution");
}

const it = fooGen();
it.next();
console.log("The generator is paused");
it.next();

// Begin execution
// The generator is paused
// End execution

We can use yield to allow our generator to "return" multiple values (we say the generator yields these). We do this as follows:

function*  fooGen() {
    console.log("Begin execution");
    yield "This value was yielded";
    console.log("End execution");
}

const it = fooGen();
console.log(it.next());
console.log("The generator is paused");
it.next();
// Begin execution
// { value: 'This value was yielded', done: false }
// The generator is paused
// End execution

Notice that using yield doesn't complete the generator iterator. This is very powerful. One example of where this behavior is useful is for producing (infinite) sequences in a memory efficient way, for example, let's look how we can implement Fibonacci sequence using generators.

function* fibonacciGenerator() {
    const f0 = 0;
    yield f0;
    const f1 = 1;
    yield f1;
    let previousValue = f0, currentValue = f1, nextValue;
    while(true) {
        nextValue = previousValue + currentValue;
        previousValue = currentValue;
        currentValue = nextValue;
        yield nextValue;
    }
}

const it = fibonacciGenerator();
console.log(it.next().value); // 0
console.log(it.next().value); // 1
console.log(it.next().value); // 1
console.log(it.next().value); // 2
console.log(it.next().value); // 3

Notice how the lazy nature of generators is very useful and how the ability to pause execution allows us to generate infinite elements of the sequence (let's ignore possible integer overflows) whenever we want while only needing to save the previous and the current values. Quite nice isn't it? Notice that we don't actually need to complete a generator, we may only take some values and never call next again, although I wouldn't recommend that.

Passing values to the generator

There are two ways we can pass values to our generator. One is just as we would to a function, when creating the generator iterator. Let's expand the Fibonacci example to allow us to choose where to start the sequence:

function* fibonacciGenerator(startingPosition = 1) {
    const f0 = 0;
    if(startingPosition === 1) {
        yield f0;
    }
    const f1 = 1;
    if(startingPosition <= 2) {
        yield f1;
    }
    let previousValue = f0, currentValue = f1, nextValue;
    let currentPosition = 3;
    while(true) {
        nextValue = previousValue + currentValue;
        previousValue = currentValue;
        currentValue = nextValue;
        if(currentPosition >= startingPosition){
            yield nextValue;
        } else {
            currentPosition += 1;
        }
    }
}

const it = fibonacciGenerator();
console.log(it.next().value); // 0
console.log(it.next().value); // 1
console.log(it.next().value); // 1
console.log(it.next().value); // 2
console.log(it.next().value); // 3

console.log();

const it2 = fibonacciGenerator(4);
console.log(it2.next().value); // 2
console.log(it2.next().value); // 3
console.log(it2.next().value); // 5
console.log(it2.next().value); // 8
console.log(it2.next().value); // 13

The other way to pass values to a generator is through yield. You may be confused, since until now we have been using yield to, well, yield values from the generator. The truth is that yield is an expression, meaning it evaluates to some value. To clarify, let's look at this example:

function* fooGen() {
    while(true) {
        console.log(yield);
    }
}

const it = fooGen();
it.next();
it.next(1); // 1
it.next(2); // 2
it.next("heey"); // heey

The first call of it.next() will simply initiate execution of our generator iterator. Whenever it finds the yield expression, it will simply stop execution. Whenever we do it.next(1), the yield will evaluate to the value 1 and thus we have console.log(1) and so on.

The following is allowed:

function* accumulator(startingValue = 0): Generator<number, any, number> {
    let value = startingValue;
    while(true) {
        const input = yield value;
        value += input;
    }
}

const it = accumulator();
it.next();
console.log(it.next(3).value); // 3
console.log(it.next(10).value); // 13
console.log(it.next(-3).value); // 10

First the code is executed until the yield is found, yielding value (startingValue) . Whenever we call next(3), the expression yield value evaluates to 3, so now input === 3 and then value === 3. The cycle then repeats.

A comment above about types. I had to explicitly type the generator above so that Typescript could automatically detect the type of input. The type inference of yield expressions is an ongoing struggle.

Attention: Whatever you pass to the first invocation of next will be ignored, so watch out.

Error Handling

The code of our generator is just like any other function code, meaning we can put try...catch blocks inside it:

function* fooGen() {
    try {
        throw "Hi";
    } catch(err) {
        console.log("Err caught in fooGen:", err);
    }
    return "End of execution";
}

const it = fooGen();
it.next();
console.log(it.next())

// Err caught in fooGen: Hi
// { value: "End of execution", done: true }
// { value: undefined, done: true }

Notice that after the exception was handled, the generator continued its execution. If we didn't have a try...catch inside the generator, the exception would bubble as it would normally:

function* fooGen() {
    throw "Hi";
    return "End of execution";
}

const it = fooGen();
try {
    it.next();
} catch(err) {
    console.log("Exception caught outside of generator: ", err);
}
console.log(it.next());

// Exception caught outside of generator:  Hi
// { value: undefined, done: true }

Notice that our generator was completed because of the uncaught exception and didn't reach our return statement.

We can also throw errors from outside our generator to the inside:

function* fooGen() {
    console.log("Beginning of execution");
    try {
        yield;
    } catch(err) {
        console.log("Error caught inside fooGen: ", err);
    }
    return "End of execution";
}

const it = fooGen();
it.next();
console.log(it.throw("Hi from outside"));
console.log(it.next());

// Beginning of execution
// Error caught inside fooGen:  Hi from outside
// { value: 'End of execution', done: true }
// { value: undefined, done: true }

Notice that the error was thrown at the point that the generator execution paused. If there was no try...catch at that point, then it would have bubbled as normal.

An example of where we would like to use Generator.throw is with our Fibonacci example. As it is implemented, eventually we will run into an overflow. We can avoid this by using bigInt. In our case, we just want to complete the iterator when overflow happens.

function* fibonacciGenerator() {
    const f0 = 0;
    yield f0;
    const f1 = 1;
    yield f1;
    let previousValue = f0, currentValue = f1, nextValue;
    try {
        while(true) {
            nextValue = previousValue + currentValue;
            previousValue = currentValue;
            currentValue = nextValue;
            yield nextValue;
        }
    } catch(err) {
        return;
    }
}
let flag = true;
let value: number | void;
const it = fibonacciGenerator();
while(flag) {
    value = it.next().value;
    if(value === Number.MAX_SAFE_INTEGER || !Number.isFinite(value)) {
        it.throw("overflow");
        console.log("overflow detected");
        console.log(it.next());
        flag = false;
    } else {
        console.log(value);
    }
}

Whenever we detect an overflow from outside our generator, we simply call it.throw to complete it so that no other garbage value gets generated from it.

Generator Delegation

We may compose two or more generator using generator delegation yield* syntax:

function* g1() {
    yield 2;
    yield 3;
    yield 4;
  }

function* g2() {
    yield 1;
    yield* g1();
    yield 5;
  }

const iterator = g2();
console.log(iterator.next()); // {value: 1, done: false}
console.log(iterator.next()); // {value: 2, done: false}
console.log(iterator.next()); // {value: 3, done: false}
console.log(iterator.next()); // {value: 4, done: false}
console.log(iterator.next()); // {value: 5, done: false}
console.log(iterator.next()); // {value: undefined, done: true}

What happens is that whenever a yield* is encountered, every subsequent next or throw will go to the delegated generator, g2 in this case. This happens until g2 completes and the completion value of g2 is the value of yield* g2(). The subsequent call to next on g1 after g2 completes will continue from where g1 was paused as normal. This is how you may write coroutines in Javascript.

You can actually use yield* with any iterable, such as arrays.

Conclusion

Generators are a somewhat obscure but very interesting structure in Javascript. You probably won't find a generator in the wild, however it is good to know of their existence.

You can build very cool stuff with generators, Async/Await is implemented with generators and promises. If you want to learn more, see my next post.

Any doubts or sugestions, feel free to add a comment. Stay safe and until next time :)