Asynchronous functions and Promises in Nodejs (and javascript) are often a bit difficult for all of us to grasp when we start out learning. Frankly, i still mess it up at times, inspite of using them often. There are already a lot of detailed 'technical' explanations for the concepts, and they are certainly an excellent resource to get a really good in-depth grasp of how everything works in Nodejs. But in this post i'll be trying to explain the basic idea behind the same concepts, in a more 'casual' way, similar to how i would have wanted it explained to myself while i was starting out to learn programming. I'll not be using any actual lines of code, and will instead try to just explain the concept and flow in a "non-technical" or simple manner.
The only things you need to know about as a prerequisite :
- Tacos
- Pizza
- Fishbowl
- Some really basic Javascript (what is a js function, and how to invoke it)
- I love cats (not really relevant to anything here, but just letting you know that since we are practically friends now, i would appreciate receiving pics of your pet cats 😄 )
INTRODUCING THE NODEJS EVENT LOOP
Nodejs has a main 'controller' or 'processor' (a single thread..could be called the nodejs event loop) which actually goes about doing all the work. It works by maintaining a to-do 'list' of 'items', which we shall call the 'tick list' (the items could vaguely be called 'ticks' ,like the ticks of the seconds-hand of a clock...the needle keeps on ticking/moving to the next step). Since we have only one controller which basically goes about running all the main functions that we ask it to run, if we keep the controller stuck for a long time on completing one specific step/function , it will not be able to handle anything else. This is called 'blocking' the event loop. Our aim is ,to try and let the controller keep moving between tasks , without being stuck on one for long. We help the controller do so by using 'Promises'.
STORY TIME
To try and understand the basic flow of how Nodejs handles functions, let's take a short story as an example. Assume you are at a restaurant and have a plate of food comprising french fries 🍟, a burger 🍔, tacos 🌮 and a slice of pizza 🍕 (like a really tasty one, not the pineapple on pizza kind..but if you really like those, i won't judge you, i'll just give you an awkward look 😁 ).
ASYNCHRONOUS
You start by eating some fries, then take a bite from the pizza slice and also stuff some tacos anytime your mouth is not already chewing. Each mouthful is a different dish. In between, you have a craving for some donuts and call the waiter over and tell him your order. While he goes to get your donuts, you continue eating your food. When the donuts are ready, you receive them and immediately dig into them , along with all the other food.
SYNCHRONOUS
You start by eating your french fries, but don't move to any other food until you finish all your fries.You make a continuous chain of fries go into your mouth. Then you move to the pizza slice and don't eat anything else until it is over. Now you call the waiter and tell you want donuts. The waiter goes to get them, but you don't eat anything when the waiter goes to the kitchen. You just sit and stare blankly , wondering why you are burying your sorrows with so much junk food. The waiter takes his own sweet time and finally arrives with the donuts, releasing you from your thoughts of existential crisis. You continue eating food only after that.
HOW THE STORY RELATES TO NODEJS
In this analogy, you are the main controller , each type of food is a different function and the waiter is a 3rd party API call or a database process. In asynchronous code, the controller keeps moving to the next possible step to execute, anytime it is free. Like if you have 2 bites from the pizza slice, and then have some tacos, then get back & continue the pizza where you left off. The eating of tacos does not need the whole pizza to be over, it just needs your mouth to have a pause in between eating pizza.
Now you must be thinking : i really crave some Tacos, wonder if that place at the corner of the street would be open now 🤔 . Also, you probably have a few questions about Nodejs like :
What are promises in Nodejs ?
How does Nodejs handle so many concurrent requests ?
How to avoid blocking the event loop in Nodejs ?
How to make Nodejs functions non-blocking ?
How to use async and await in Nodejs ?
How to run cpu-intensive functions in Nodejs ?
Why did the chicken cross the road? to fulfill a nodejs promise..wait..sorry that one doesn't belong here..oops..getting back to our topic
What are Promises in Node.js ?
Promises are like their name suggests, similar to a promise that you give a friend. Promises in Nodejs are like an I.O.U slip that a function gives back immediately when it is called. The controller just keeps the slip and then goes onto processing other functions. Later on, the function gets back to the controller and replaces the I.O.U with the actual status of it's task, which could either be a success or a failure.
STORY TIME AGAIN
Let's look at another example to get a better understanding of the basic concept of promises in Nodejs. Suppose your friend gives you a fishbowl to clean the water. You get it and 'promise' your friend that you will clean it and give it back to them. Your friend goes on doing other things , while you are cleaning the fish bowl. Now, after a while there are 2 possible outcomes
you clean the fishbowl as expected
maybe some problem (error) occurs, and you are not able to complete the cleaning...let's assume the bowl broke.
So, either when you complete the cleaning, or when the bowl breaks, your work related to the fishbowl is technically over, so you inform your friend that it was cleaned (your promise was resolved/fulfilled) or that the bowl broke (your promise is rejected or not fulfilled).
Basically, you have given an update regarding the previous promise that you had given your friend. Now, your friend can decide what to do next with that information : accept the cleaned bowl and do something, or analyse the broken bowl and decide to buy a new one.
In this analogy, your friend is the main controller and you are the function that is called which returns a 'promise'. The controller just holds onto the promise and then goes about doing other tasks. It comes back to the promise when it gets a response regarding the status of the promise : resolved or rejected. This status update is referred to as the promise getting 'settled'.
The controller then sees what we have asked it to do (to decide what function it needs to run next) , to handle the data set returned , or the error message. While coding, we define the 'next steps' based on the response of the promise. So from the controller's point of view, initially the function that returns the promise is added as an 'item' in it's tick list. It immediately gets a 'promise' as a response, and then moves onto whatever item is next in it's tick list.
When the promise gets resolved/rejected , it is added as an item in the tick list and then the controller checks what we have instructed it to do. This basically keeps continuing. Even when to us it may seem like the requests are reaching Nodejs at the exact same time, most often there will be a difference of a few milliseconds between them, and one request gets added to the tick list after the other. So your Nodejs program is able to handle a large number of concurrent requests easily.
Your aim while writing codes in Nodejs, is to reduce the main controller being stuck doing some single work for a long time. Such long processes should ideally be handled by some other service like a database, separate server, 3rd party ,etc. or else, you can create 'workers' . Workers are like mini-main-controllers. Your main controller can push tasks that need intensive processing to such worker threads and continue handling other tasks. The worker and the main controller are able to communicate with each other through a few limited means, and they can use it to pass data between them.
[*sidenote : It's not that the main controller cannot handle intensive processing. It's just that if your website or app is being used by mutliple people at once, then the main controller will be stuck on one request for too long and hence unable to process anything else. This will make the server unresponsive to further requests. But, if you wanted to make some cpu-intensive program for your own personal use, you can still easily do it in Nodejs , since in that case you are willing to wait for the long processes to complete and know that you won't be making any new requests while the main controller is already busy. *]
Two common ways in which promises are handled in Nodejs are via :
then / catch
async await
THEN() , CATCH() in Nodejs
In Nodejs, one way to handle promises and specify what steps needs to be done next, is by using '.then()' and '.catch()'. then() is used to tell what needs to be done when the promise is resolved, and catch() is used to specify what should happen when a promise is rejected. It is used to instruct the controller on what it needs to do next, once the promise is settled. It is almost like an if-else condition that we are using to tell the controller exactly what it needs to do next, based on the promise's outcome.
STORY TIME YET AGAIN
We could think of it like a set of inter-dependant tasks you are assigning to your friends while you are planning for a party. One day you think you should have a mini-party and call your friends : Csaba , Faruk and Alberto , who agree to make an awesome cake. The plan is : Faruk makes the batter , Alberto bakes it and Csaba decorates it.
Now, in this analogy you are the 'spirit' that posesses each friend and makes them do the works...yeah..that's just a bit too weird ain't it...hmm...well,maybe we could also consider it as you are the **Ratatouille **that gets each person to do the work they are supposed to...yeah, That's much better.
Now if everyone did all the work simultaneously, nothing would get done. You are after all just one rat, however talented you are, and can't be everywhere at once. So while, you are making Faruk prepare the batter, Alberto and Csaba are free, since they technically can't start their work without Faruk passing the batter. As soon as Faruk makes the batter, you switch to controlling Alberto and receive the batter and keep it in the oven.
This is like the controller was told to wait for the promise of the batter, and 'then' since it was successful, it went to the next step we have told it to do, which is baking.
Now,there are two possible outcomes here as well :
- the cake is baked perfectly and Alberto takes it out
- the cake gets burnt , and the cake plan needs to get discarded or re-done
If the cake ends up being perfect, it is passed to Csaba, and 'then' he decorates it marvellously. But, if the cake ends up getting burnt, we can't give it to Csaba and instead we make Alberto put it in the garbage bin , similar to an error being caught by using .catch().
ASYNC and AWAIT in Nodejs
This is generally the most preferred method for handling promises since it is easier to understand and simpler to code. The word 'async' is added before the definition of the function , and is used to denote that the function returns a 'promise'. The 'await' keyword can be used only inside functions which have been tagged with the 'async' keyword. Adding 'await' before a function call , indicates that a promise is expected to be returned , and that the controller can make a note of it and move onto other tasks, then return once the promise is settled. (It kinda tells the controller to wait for the promise to be completed before proceeding to the next line of code) . This is especially useful when the results returned by the awaited function are needed in the lines that follow.
When the controller sees that a function is 'awaited' ,it makes a note of the promise and then goes to perform the next item in it's tick list. Once, the previously awaited promise is settled, the controller comes back to that line and then continues processing the next steps based on whether the promise was resolved or rejected. This helps us to have more control on the sequence of the functions that need to be performed, without needing to necessarily create a chain of .then() functions.
Just adding the word 'await' will not automatically make a function asynchronous. We need to make sure that the function itself is one that returns a promise and is asynchronous. Many functions in Nodejs have an 'async' version and a 'sync' version. So, choosing the async version specifically in such cases will naturally help us.
Some functions like json.parse and json.stringify make the controller stuck until their processing is over. So if we have a large json object that needs to be parsed/stringified, it would make the controller unable to handle any other requests until it is over.
Since we generally may be using only relatively small json objects at a time, processing it may not necessarily be a noticeable block of the event loop. But, depending on your use-case you may need some non-blocking option for it. This is where the concept of 'streaming' comes to our rescue. Here, kinda similar to how we stream videos on Netflix or Youtube, we get the actual data in smaller chunks. We also often use 'Buffer' for this purpose which act like temporary storages for the chunk and then pass the info.
So , for example, if we have about 1000 parameters in our large json object, instead of the controller being forced to process the whole 1000, the controller is allowed to have small breaks in between, like maybe once every 100 parameters are processed. This break lets the controller to be free to handle any other requests while also being able to get back and process the next 100 parameters of the json object.
This streaming of data concept is also useful in situations where we need to manipulate or process large data sets from a database or 3rd party REST API , etc. If for example, we wanted to process a million rows of data, handling it all at once would obviously seem like a bad idea. So , instead the data is streamed from the database to the controller in small chunks, which again allows the controller to process any other requests, while also making progress in analysing the database rows.
WHAT DID WE LEARN
- our aim while coding should be to avoid blocking the event loop
- async/await is a great way to use promises
- breaking functions into individual tasks could help to avoid blocking
- splitting data into chunks/streams is better when heavy processing is needed
- i really need to improve my writing skills...well, that's more of a learning for me than you..but still 😁
THAT'S ALL DEV FAM 😃
If you've read till here, then I want you to know that i'm grateful that you took the time to do so and proud of your willingness to read new resources while learning.
Kudos dear reader.
I hope i was able to atleast clear out some concepts related to asynchronouse functions and promises in Nodejs. (and really reallyyyyy hope i didn't make things worse for you)
** Hope you'll follow me on Twitter so that we can get to know each other , and grow together. **
Thanks again for taking out the time to read my post.
Wishing good things for you always.
Top comments (1)
what about .promise() ?