Introduction

In this tutorial, we’ll build a powerful downloader that allows downloading files from Google Drive and other cloud providers. With Golang’s efficient concurrency patterns, you'll be able to manage multiple downloads concurrently, stream large files, and track progress in real-time. Whether you’re downloading a few small files or handling large data sets, this project will showcase how to build a scalable and robust downloader that can easily be extended to support multiple cloud platforms.

If you're looking for a way to simplify and automate downloading large files, this tutorial is perfect for you. By the end, you’ll have a flexible and customizable Go-based downloader to suit your needs.

In a hurry?

If you're just looking to use this downloader with a UI, visit evolveasdev.com for read the full article & Go Downloader's Github. You'll find the docs to get it running fast.

What You’ll Learn

• Go Concurrency Patterns:
  Learn how to use Goroutines, channels and mutexes to handle multiple concurrent file downloads efficiently.

• Streaming Large Downloads:
  Explore how to stream large files while managing memory and system resources effectively.

• Concurrent File Downloads:
  Understand how to download files concurrently, speeding up the process and improving performance.

• Real-Time Progress Updates:
  Implement progress tracking to provide real-time feedback on download status.

• Handling Interruptions and Cancellations:
  Learn how to gracefully cancel one or all ongoing downloads.

Note: This tutorial will only focus on the core downloading logic.

Environment Setup

First before doing anything make sure to properly setup your environment to avoid potential bugs in future.

Prerequisites

• Install Go
• AIR for auto reloading
• Makefile to run complex commands
• Goose for PostgreSQL migrations

Configuring Makefile

Create a makefile at the root of the project with the following.

# Load environment variables from .env file
include ./.env

# To run the application
run: build
    @./bin/go-downloader

# Build the application
build:
    @go build -tags '!dev' -o bin/go-downloader

# Database migration status
db-status:
    @GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) status

# Run database migrations
up:
    @GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) up

# Roll back the last database migration
down:
    @GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) down

# Reset database migrations
reset:
    @GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) reset

High-Level Folder Structure Overview

go-downloader/
├── api
├── config
├── migrations
├── service
├── setting
├── store
├── types
├── util
├── .env
├── .air.toml
├── Makefile
├── go.mod
├── go.sum
└── main.go

Setting environment variables

Create a .env file in root or handle environment variables however you like, we'll use joho/godotenv package.

GOOGLE_CLIENT_ID
GOOGLE_CLIENT_SECRET
SESSION_SECRET=something-super-secret
APP_URL=http://localhost:3000
POSTGRES_USER
POSTGRES_PASSWORD
POSTGRES_DB

Creating a Web Server

We'll now start creating the web server that'll handle all the incoming requests.

Heads Up! The main part of this guide begins here. Get ready to dive in!

API Layer

To start, create the following files inside api folder api.go and route.go

route.go file setup

All API routes will be defined in this here. We create a NewRouter struct that takes the env configuration, allowing all routes and handlers to access environment variables.

package api

import (
    "github.com/gofiber/fiber/v2"
    "github.com/nilotpaul/go-downloader/config"
)

type Router struct {
    env       config.EnvConfig
}

func NewRouter(env config.EnvConfig) *Router {
    return &Router{
        env:      env,
    }
}

func (h *Router) RegisterRoutes(r fiber.Router) {
    r.Get("/healthcheck", func(c *fiber.Ctx) error {
        return c.JSON("OK")
    })
}

api.go file setup

Here, we’ll add all the necessary middlewares, such as CORS and logging, before starting the server.

type APIServer struct {
    listenAddr string
    env        config.EnvConfig
}

func NewAPIServer(listenAddr string, env config.EnvConfig) *APIServer {
    return &APIServer{
        listenAddr: listenAddr,
        env:        env,
    }
}

func (s *APIServer) Start() error {
    app := fiber.New(fiber.Config{
        AppName:      "Go Downloader",
    })

    handler := NewRouter()
    handler.RegisterRoutes(app)

    log.Printf("Server started on http://localhost:%s", s.listenAddr)

    return app.Listen(":" + s.listenAddr)
}

Main Entrypoint

This is the main package in main.go file which will act as a entrypoint to the whole.

func main() {
    // Loads all Env vars from .env file.
    env := config.MustLoadEnv()

    log.Fatal(s.Start())
}

This is enough to start up the server and test it.

Start the server

air

that's it.😊

Testing

curl http://localhost:3000/healthcheck

The response should be OK with status 200

Creating a Provider Store

We need to implement a scalable solution for adding support for multiple cloud providers if necessary.

Working on provider registry

// Better to keep it in a seperate folder.
// Specific only to OAuth Providers.
type OAuthProvider interface {
    Authenticate(string) error
    GetAccessToken() string
    GetRefreshToken() string
    RefreshToken(*fiber.Ctx, string, bool) (*oauth2.Token, error)
    IsTokenValid() bool
    GetAuthURL(state string) string
    CreateOrUpdateAccount() (string, error)
    CreateSession(c *fiber.Ctx, userID string) error
    UpdateTokens(*GoogleAccount) error
}

type ProviderRegistry struct {
    Providers map[string]OAuthProvider
}

func NewProviderRegistry() *ProviderRegistry {
    return &ProviderRegistry{
        Providers: make(map[string]OAuthProvider),
    }
}

func (r *ProviderRegistry) Register(providerName string, p OAuthProvider) {
    r.Providers[providerName] = p
}

func (r *ProviderRegistry) GetProvider(providerName string) (OAuthProvider, error) {
    p, exists := r.Providers[providerName]
    if !exists {
        return nil, fmt.Errorf("Provider not found")
    }

    return p, nil
}

The ProviderRegistry serves as a central map to hold all our OAuth providers. When we initialize our providers, we'll register them in this map. This allows us to easily access any registered provider's functionalities throughout our service.

You'll see this action later.

Initializing the Provider Store

We'll register our providers based on the environment variables provided.

func InitStore(env config.EnvConfig) *ProviderRegistry {
    r := NewProviderRegistry()

    if len(env.GoogleClientSecret) != 0 || len(env.GoogleClientID) != 0 {
        googleProvider := NewGoogleProvider(googleProviderConfig{
            googleClientID:     env.GoogleClientID,
            googleClientSecret: env.GoogleClientSecret,
            googleRedirectURL:  env.AppURL + "/callback/google",
        }, env)

        r.Register("google", googleProvider)
    }

    return r
}

Read the full article here.

Wrapping Up

We’ve laid the groundwork for the Google Drive Downloader in Go, covering key components such as setting up the project structure, handling Google OAuth, and laying the foundation for future expansion. Along the way, we touched on some important topics:

• Database migrations with Goose to ensure smooth and scalable data management.
• Building a provider registry to easily extend support for additional cloud storage providers in the future.
• Designing a flexible architecture that makes it easy to handle more complex logic down the road.

That’s more than enough for one post, as things were getting pretty long! We’ll come back in Part 2 to finish our work, where we’ll work the main downloading functionality.

Until then, feel free to explore the current implementation in my GitHub and stay tuned for the next steps. Happy downloading!