DEV Community

Code Green
Code Green

Posted on

System design of REST API that facilitates the uploading of large files using chunking

This post outlines the design of a REST API that facilitates the uploading of large files using chunking, asynchronous processing, and tracking mechanisms. The API will allow clients to upload files in manageable chunks, freeing them from blocking operations during the upload process.

Asynchronous Processing

To unblock the client during file uploads, the API can implement asynchronous processing using a message queue or background worker. Here’s how it works:

  1. Client Initiates Upload: The client sends a request to initiate an upload and receives an upload ID.
  2. Client Uploads Chunks: The client uploads each chunk asynchronously.
  3. Background Processing: Each uploaded chunk is processed in the background by a worker service that assembles the chunks into a complete file.
  4. Client Checks Status: The client can check the status of the upload at any time using the status endpoint.

Chunking Strategy

  • Chunk Size: Define a maximum chunk size (e.g., 5 MB) to ensure efficient uploads.
  • Total Chunks Calculation: Clients should calculate the total number of chunks based on the file size and chunk size before initiating the upload.

API Overview

Base URL

POST /api/v1/uploads
Enter fullscreen mode Exit fullscreen mode

API Endpoints

1. Initiate Upload

Endpoint:

POST /api/v1/uploads/initiate
Enter fullscreen mode Exit fullscreen mode

Description: Initiates a file upload session and generates an upload ID for tracking.

Request Body:

    {
      "fileName": "example.txt",
      "fileSize": 104857600,  // Size in bytes (100 MB)
      "fileType": "text/plain"
    }
Enter fullscreen mode Exit fullscreen mode

Response:

    {
      "uploadId": "1234567890abcdef",
      "status": "initiated",
      "message": "Upload session initiated successfully."
    }
Enter fullscreen mode Exit fullscreen mode

2. Upload Chunk

Endpoint:

POST /api/v1/uploads/{uploadId}/chunks
Enter fullscreen mode Exit fullscreen mode

Description: Uploads a chunk of the file.

Path Parameters:

  • uploadId: The unique identifier for the upload session.

Request Body:

    {
      "chunkIndex": 0,
      "chunkData": "",
      "totalChunks": 10
    }
Enter fullscreen mode Exit fullscreen mode

Response:

    {
      "status": "chunk received",
      "message": "Chunk uploaded successfully.",
      "nextChunkIndex": 1
    }
Enter fullscreen mode Exit fullscreen mode

3. Complete Upload

Endpoint:

POST /api/v1/uploads/{uploadId}/complete
Enter fullscreen mode Exit fullscreen mode

Description: Finalizes the upload session after all chunks have been uploaded.

Path Parameters:

  • uploadId: The unique identifier for the upload session.

Response:

    {
      "status": "completed",
      "message": "File uploaded successfully.",
      "fileLocation": "/uploads/example.txt"
    }
Enter fullscreen mode Exit fullscreen mode

4. Check Upload Status

Endpoint:

GET /api/v1/uploads/{uploadId}/status
Enter fullscreen mode Exit fullscreen mode

Description: Checks the status of the ongoing upload session.

Path Parameters:

  • uploadId: The unique identifier for the upload session.

Response:

    {
      "uploadId": "1234567890abcdef",
      "status": "in progress", // or 'completed', 'failed'
      "uploadedChunks": 5,
      "totalChunks": 10
    }
Enter fullscreen mode Exit fullscreen mode

Conclusion

This REST API design provides a comprehensive solution for uploading large files efficiently. By utilizing chunking, asynchronous processing, and tracking mechanisms via upload IDs, clients can seamlessly upload large files without blocking their operations, enhancing user experience and system performance.

Top comments (0)