Iterator is a great pattern. It's extremely simple and yet very powerful. Long time ago I saw iterators literally everywhere but, when Java 8 came with streams API, things have changed. It doesn't mean that iterators disappeared fully. No, they just hid deep into libraries and frameworks you use every day. But few of them are still the most important players in modern and widely used Java SDKs.

SIP3 uses MongoDB Java Driver to read insanely big amounts of data. And as you know from my previous blog post we prefer to work with MongoCursor<Document> explicitly. Guess what? MongoCursor<Document> is nothing else but iterator 😊.

Having MongoCursor<Document> as iterator totally makes sense though. Iterator's contract forces you to call hasNext() method every time when you want to fetch a new document from MongoDB collection. So, MongoDB Java Driver just makes you read not all documents but as many as you need. This approach saves a lot of resources on data transferring cause data now can be stored in MongoDB server and pulled from there on demand in multiple batches.

In the previous blog post I showed how we work with multiple MongoDB collections partitioned by time the same way you will work with just a single collection. However, it wasn't enough to aggregate the data from all the partitions we have. That's why, inspired by streams API, we decided to introduce a few extension functions.

⚠️ WARNING: Examples below are not perfect and didn't mean to be though. So, keep in mind that calling map() and merge() methods will change a calling iterator state too.

map()

No doubts map() is the most popular function of streams API. Of course it can be a great addition to our iterator:

fun <T, R> Iterator<T>.map(transform: (T) -> R): Iterator<R> {
    val i = this

    return object : Iterator<R> {

        override fun hasNext(): Boolean {
            return i.hasNext()
        }

        override fun next(): R {
            if (!hasNext()) throw NoSuchElementException()
            return transform.invoke(i.next())
        }
    }
}

Now let's see how we can call it:

// Int: 1, 2, 3
val numbers = listOf(1, 2, 3).iterator()

// String: "1", "2", "3"
val strings = numbers.map(Any::toString) 

The cherry on the top 🍒 is that transform function will be called lazily.

merge()

Not like map() this method is not a part of streams API. However, it let's us gather data from logically partitioned in multiple collections (you can read more about it in the previous blog post). Implementation is a bit complicated.

First we need to write a helper method:

fun <T> Iterator<T>.nextOrNull(): T? {
    return if (hasNext()) next() else null
}

,and then our method extension itself:

fun <T> Iterator<T>.merge(o: Iterator<T>, comparator: Comparator<T>? = null): Iterator<T> {
    val i = this

    return object : Iterator<T> {

        var vi: T? = i.nextOrNull()
        var vo: T? = o.nextOrNull()

        override fun hasNext(): Boolean {
            return vi != null || vo != null
        }

        override fun next(): T {
            if (!hasNext()) throw NoSuchElementException()

            val v: T?
            if (vi != null && (vo == null || comparator == null || comparator.compare(vi, vo) <= 0)) {
                v = vi
                vi = i.nextOrNull()
            } else {
                v = vo
                vo = o.nextOrNull()
            }

            return v!!
        }
    }
}

Now let's see it in action:

val list1 = listOf(1, 3, 5)
val list2 = listOf(2, 4, 6)

var iterator1 = list1.iterator()
var iterator2 = list2.iterator()

// Int: 1, 3, 5, 2, 4, 6
val merged = iterator1.merge(iterator2)

iterator1 = list1.iterator()
iterator2 = list2.iterator()

// Int: 1, 2, 3, 4, 5, 6
val sorted = iterator1.merge(iterator2, Comparator.comparingInt<Int> { i -> i })

So, we just merged and even sorted two iterators 🤯. But... As Leonardo says...

Sure thing:

object IteratorUtil {

    fun <T> merge(vararg iterators: Iterator<T>): Iterator<T> {
        var i = Collections.emptyIterator<T>()
        iterators.forEach { i = i.merge(it) }
        return i
    }
}

Whaaaaat 🤯🤯🤯

Jokes aside in our case this merge exercises are very needed even though they look scary. You can always see this code in action in our Github projects.

In the post I just wanted to show all the power and beauty of Kotlin extension functions and maybe give you some motivation to use them.

👨‍💻 Happy coding,
Your SIP3 team.