How to Create a Streamable Data Endpoint in Dotnet for Progressive Loading ?

Target Audience

I've aimed this article at people who want to learn about dotnet, next.js, streams api.

Learning Objectives

After completing this article, you will know how to do the following:

• Create a basic api that returns streamable data
• Create a frontend application where you make a request to this endpoint and show response chunk by chunk.

Advantages of using streamable endpoint ?

There are several advantages to using progressive loading, also known as lazy loading or incremental loading, for streaming large amounts of data:

1. Improved User Experience: Progressive loading can enhance the user experience by providing immediate feedback and a feeling of responsiveness as the data is being loaded. Users don't have to wait for the entire data set to be loaded before they can start viewing or interacting with the data.

2. Reduced Load Times: Progressive loading can reduce load times by breaking the data into smaller chunks and loading them gradually, instead of loading the entire data set at once. This can help to reduce the load on the server and minimize network latency, resulting in a faster and more efficient loading process.

3. Lower Memory Usage: Progressive loading can help to reduce memory usage by only loading the data that is required at any given time. This can be especially useful for large data sets that would otherwise require a lot of memory to load in their entirety.

4. Scalability: Progressive loading can help to improve the scalability of an application by allowing it to handle large amounts of data without putting excessive strain on the server or the network. This can be especially important for web applications that need to support a large number of concurrent users.

5. Robustness: Progressive loading can help to make an application more robust by allowing it to recover from errors or interruptions during the loading process. If the loading process is interrupted or the connection is lost, the application can simply resume loading from the point where it left off, instead of having to start over from the beginning.

What is Transfer Encoding ?

We will use Transfer-Encoding in http response headers to send data chunk by chunk.

Wikipedia Definition

Chunked transfer encoding is a streaming data transfer mechanism available in Hypertext Transfer Protocol (HTTP) version 1.1, defined in RFC 9112 §7.1. In chunked transfer encoding, the data stream is divided into a series of non-overlapping "chunks". The chunks are sent out and received independently of one another.

How to create a streamable endpoint in Dotnet ?

Here are the steps to create this endpoint:

1-Create a new dotnet webapi

Use the following command to create an empty web api.

dotnet new webapi -o streamable-api

2- Modify the endpoint method

 [HttpGet, Route("get/stream")]
 public async Task GetStreamAsync() {

   Response.Headers.Add("Content-Type", "text/plain");
   Response.Headers.Add("Transfer-Encoding", "chunked"); // this is required to inform the client that you need to consume chunk by chunk

   for (var i = 0; i <= 10; i++) {
     var chunk = $ "This is chunk id:{i} \n";

     _logger.LogInformation($"Chunk count: {i}");

     var bytes = System.Text.Encoding.UTF8.GetBytes(chunk);

     await Response.Body.WriteAsync(bytes, 0, bytes.Length);
     await Response.Body.FlushAsync();
     await Task.Delay(1000);
   }

 }

As you can see firstly we set the Transfer-Encoding property to inform the client. And we use Task.Delay method to demonstrate the network delay.

What is Streams API ?

Mdn Definition

The Streams API allows JavaScript to programmatically access streams of data received over the network and process them as desired by the developer.

Streaming involves breaking a resource that you want to receive over a network down into small chunks, then processing it bit by bit. This is something browsers do anyway when receiving assets to be shown on webpages — videos buffer and more is gradually available to play, and sometimes you'll see images display gradually as more is loaded.

How to consume a streamable endpoint in the frontend ?

In this demo i use React, but you can choose any javascript library or framework you want.

// here is the example for streams api

fetch('https://localhost:7059/api/get/stream')
.then(response => {
  const reader = response.body.getReader();

  function read() {
    reader.read().then(({ done, value }) => {
      if (done) {
        console.log('Response fully loaded');
        return;
      }

      console.log('Received chunk of data:');
      console.log(value);

      // Call `read` again to read the next chunk
      read();
    });
  }

  read();
})
.catch(error => {
  console.error('Error loading response:', error);
});

In this example, we use the fetch function to make a GET request to a server endpoint that returns a response in JSON format. We then use the body property of the response object to create a ReadableStream object, and call its getReader method to get a ReadableStreamDefaultReader object.

We define a read function that calls the read method of the ReadableStreamDefaultReader object to read a chunk of data from the response stream. If the done property of the result object is true, we log a message indicating that the response is fully loaded. Otherwise, we log the chunk of data and call the read function again to read the next chunk.

Note that the Fetch API is supported in most modern browsers, but not in all. If you need to support older browsers, you may need to use a different approach for loading responses chunk by chunk, such as the XMLHttpRequest API or a third-party library.

Here's an example of how you can use async/await with the Fetch API.

 const fetchData = async () => {
    const response = await fetch('https://localhost:7059/api/get/stream');
    if (!response.ok) {
      throw new Error(`Failed to load response: ${response.status} ${response.statusText}`);
    }

    if (response.body === null) {
      throw new Error("Response body is null");
    }

    const reader = response.body.getReader();

    while (true) {
      const { done, value } = await reader.read();
      if (done) {
        console.log("Read complete");
        break;
      }
      if (value !== undefined) {

        const str = new TextDecoder("utf-8").decode(value);
        console.log("-----: ", value, "----", str);    

        setValue(prev => [...prev, str]);

        console.log("Received chunk: ", value); 
      }
    }
  }

References

• https://github.com/eminvergil/streamable-tutorial
• https://developer.mozilla.org/en-US/docs/Web/API/Streams_API
• https://en.wikipedia.org/wiki/Chunked_transfer_encoding#:~:text=Chunked%20transfer%20encoding%20is%20a,received%20independently%20of%20one%20another.

You can check the example implementation here:
https://github.com/eminvergil/streamable-tutorial