Whats are Streams?
Streams are used to process (read and write) data piece by piece (chunks) without completing the whole read and write operation and at the therefore without keeping all the data in memory.
Youtube or Netflix are good example of streams. You don't have to wait for video to fully load. The process is done piece by piece (chucks). So you can then start watching even if the entire file is not yet download
In Node.js there are Readable Streams and Writable Streams. Readable Streams can for example be a file read or a http request data.
Writable Streams is the opposite of Readable Streams so for example a http responses or file to send
Here an example of a large data file read
const fs = require('fs')
const server = require('http').createServer()
server.on('request', () => {
// No need to load the entire file to memory
// fs.readFile('data.txt', (err, data) => {
// if (err) console.log(err)
// res.end(data);
// })
// Create a Readable Streams
const readable = fs.createReadStream('data.txt')
// Pipe the Stream chunk to a writable Stream
readable.pipe(res);
})
The readable.pipe() method attaches a Writable stream to the readable, causing it to switch automatically into flowing mode and push all of its data to the attached Writable. The flow of data will be automatically managed so that the destination Writable stream is not overwhelmed by a faster Readable stream.
Conclusion
That's it for today. Tomorrow the journey continue. Stay tune!
Follow me on Twitter: Follow @justericchapman
Top comments (1)
Streams are great and all but error handling is complex with it.
I've learned with this great talk by Matteo Collina that
fs.createReadStream('data.txt').pipe(res)
isn't the right way to stream a file over HTTP because this will create a memory leak. Here Matteo explains why.