Vic Shóstak

Posted on Mar 31, 2021 • Edited on Aug 17

📈 Working with RabbitMQ in Golang by examples

#go #tutorial #beginners #rabbitmq

Introduction

Hey, DEV people! 😉 Today, I'll cover the topic of working with a Message broker called RabbitMQ in your Go projects. There will be both a theoretical part and practice.

Of course, the article is more aimed at those who just want to understand this topic. But I call on more experienced people to help cover this topic even better in the comments on this article. Together we can do a lot!

What is a message broker?

This is an architectural pattern in distributed systems, where a message broker is an application that converts a single protocol message from the source application to a protocol message from the destination application, thereby acting as an intermediary between them.

Also, the tasks of the message broker include:

Checking the message for errors;
Routing to specific receiver(s);
Splitting the message into several smaller ones, and then aggregating the receivers' responses and sending the result to the source;
Saving the messages to a database;
Calling web services;
Distributing messages to subscribers;

🤔 But what is it anyway? Well, let's translate it into our language.

If you simplify this huge description, you can portray the message broker as a post office in real life (which you have encountered many times):

A sender (user of your product) brings a parcel (any data) to the post office and specifies the addressee for receipt (another service).
A post office employee accepts the parcel and places it in the storage area (puts it in the queue to be sent) and issues a receipt that the parcel has been successfully accepted from the sender.
After some time, the parcel is delivered to the addressee (another service), and he doesn't have to be at home to accept the parcel. In this case, his parcel will wait in a mailbox until he receives it.

↑ Table of contents

What problems will be able to solve?

One of the most important problems that can be solved by using this architectural pattern is to parallelize tasks with a guaranteed result, even if the receiving service is unavailable at the time of sending the data.

With the total dominance of microservice architecture in most modern projects, this approach can maximize the performance and resilience of the entire system.

👌 It sounds a bit confusing... but let's use the post office analogy again!

Once the sender gives his parcel to the post office employee, he no longer cares how his parcel will be delivered, but he does know that it will be delivered anyway!

↑ Table of contents

Full project code

For those who want to see the project in action:

koddr / tutorial-go-fiber-rabbitmq

📖 Tutorial: Working with RabbitMQ in Golang by examples.

↑ Table of contents

Setting up RabbitMQ

As usual, let's create a new Docker Compose file:



# ./docker-compose.yml

version: "3.9"

services:

  # Create service with RabbitMQ.
  message-broker:
    image: rabbitmq:3-management-alpine
    container_name: message-broker
    ports:
      - 5672:5672   # for sender and consumer connections
      - 15672:15672 # for serve RabbitMQ GUI
    volumes:
      - ${HOME}/dev-rabbitmq/data/:/var/lib/rabbitmq
      - ${HOME}/dev-rabbitmq/log/:/var/log/rabbitmq
    restart: always
    networks:
      - dev-network

networks:
  # Create a new Docker network.
  dev-network:
    driver: bridge

☝️ Please note! For the initial introduction to RabbitMQ we will not create a cluster and use a load balancer. If you want to know about it, write a comment below.

↑ Table of contents

Setting up Fiber as a sender

To connect to the message broker, we will use the Advanced Message Queuing Protocol or AMQP for short. The standard port for RabbitMQ is 5672.

Okay, let's write a simple data sender using Fiber web framework:



// ./sender/main.go

package main

import (
    "log"
    "os"

    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/logger"
    "github.com/streadway/amqp"
)

func main() {
    // Define RabbitMQ server URL.
    amqpServerURL := os.Getenv("AMQP_SERVER_URL")

    // Create a new RabbitMQ connection.
    connectRabbitMQ, err := amqp.Dial(amqpServerURL)
    if err != nil {
        panic(err)
    }
    defer connectRabbitMQ.Close()

    // Let's start by opening a channel to our RabbitMQ
    // instance over the connection we have already
    // established.
    channelRabbitMQ, err := connectRabbitMQ.Channel()
    if err != nil {
        panic(err)
    }
    defer channelRabbitMQ.Close()

    // With the instance and declare Queues that we can
    // publish and subscribe to.
    _, err = channelRabbitMQ.QueueDeclare(
        "QueueService1", // queue name
        true,            // durable
        false,           // auto delete
        false,           // exclusive
        false,           // no wait
        nil,             // arguments
    )
    if err != nil {
        panic(err)
    }

    // Create a new Fiber instance.
    app := fiber.New()

    // Add middleware.
    app.Use(
        logger.New(), // add simple logger
    )

    // Add route.
    app.Get("/send", func(c *fiber.Ctx) error {
        // Create a message to publish.
        message := amqp.Publishing{
            ContentType: "text/plain",
            Body:        []byte(c.Query("msg")),
        }

        // Attempt to publish a message to the queue.
        if err := channelRabbitMQ.Publish(
            "",              // exchange
            "QueueService1", // queue name
            false,           // mandatory
            false,           // immediate
            message,         // message to publish
        ); err != nil {
            return err
        }

        return nil
    })

    // Start Fiber API server.
    log.Fatal(app.Listen(":3000"))
}

As you can see, at the beginning we create a new connection to RabbitMQ and a channel to send data to the queue, called QueueService1. With a GET request to localhost:3000/send, we can pass a needed text in a msg query parameter, which will be sent to the queue and next to the consumer.

Now create a new Dockerfile called Dockerfile-sender in the root of the project in which we describe the process of creating the container for sender:



# ./Dockerfile-sender

FROM golang:1.16-alpine AS builder

# Move to working directory (/build).
WORKDIR /build

# Copy and download dependency using go mod.
COPY go.mod go.sum ./
RUN go mod download

# Copy the code into the container.
COPY ./sender/main.go .

# Set necessary environment variables needed 
# for our image and build the sender.
ENV CGO_ENABLED=0 GOOS=linux GOARCH=amd64
RUN go build -ldflags="-s -w" -o sender .

FROM scratch

# Copy binary and config files from /build 
# to root folder of scratch container.
COPY --from=builder ["/build/sender", "/"]

# Command to run when starting the container.
ENTRYPOINT ["/sender"]

All that remains is to update the Docker Compose file so that it takes this Dockerfile into account when creating a container with RabbitMQ:



# ./docker-compose.yml

version: "3.9"

services:

  # ...

  # Create service with Fiber sender.
  sender:
    container_name: sender
    ports:
      - 3000:3000
    build:
      context: .
      dockerfile: Dockerfile-sender
    environment:
      AMQP_SERVER_URL: amqp://guest:guest@message-broker:5672/
    restart: always
    networks:
      - dev-network
    depends_on:
      - message-broker

  # ...

# ...

Setting up a message consumer

The message consumer should be able to accept messages from the broker's queue and output in the logs the message sent from the sender.

Let's implement such a consumer:



// ./consumer/main.go

package main

import (
    "log"
    "os"

    "github.com/streadway/amqp"
)

func main() {
    // Define RabbitMQ server URL.
    amqpServerURL := os.Getenv("AMQP_SERVER_URL")

    // Create a new RabbitMQ connection.
    connectRabbitMQ, err := amqp.Dial(amqpServerURL)
    if err != nil {
        panic(err)
    }
    defer connectRabbitMQ.Close()

    // Opening a channel to our RabbitMQ instance over
    // the connection we have already established.
    channelRabbitMQ, err := connectRabbitMQ.Channel()
    if err != nil {
        panic(err)
    }
    defer channelRabbitMQ.Close()

    // Subscribing to QueueService1 for getting messages.
    messages, err := channelRabbitMQ.Consume(
        "QueueService1", // queue name
        "",              // consumer
        true,            // auto-ack
        false,           // exclusive
        false,           // no local
        false,           // no wait
        nil,             // arguments
    )
    if err != nil {
        log.Println(err)
    }

    // Build a welcome message.
    log.Println("Successfully connected to RabbitMQ")
    log.Println("Waiting for messages")

    // Make a channel to receive messages into infinite loop.
    forever := make(chan bool)

    go func() {
        for message := range messages {
            // For example, show received message in a console.
            log.Printf(" > Received message: %s\n", message.Body)
        }
    }()

    <-forever
}

Okay, in the same way with sender, let's create a new Dockerfile called Dockerfile-consumer to describe the process of creating a container for the message consumer:



# ./Dockerfile-consumer

FROM golang:1.16-alpine AS builder

# Move to working directory (/build).
WORKDIR /build

# Copy and download dependency using go mod.
COPY go.mod go.sum ./
RUN go mod download

# Copy the code into the container.
COPY ./consumer/main.go .

# Set necessary environment variables needed 
# for our image and build the consumer.
ENV CGO_ENABLED=0 GOOS=linux GOARCH=amd64
RUN go build -ldflags="-s -w" -o consumer .

FROM scratch

# Copy binary and config files from /build 
# to root folder of scratch container.
COPY --from=builder ["/build/consumer", "/"]

# Command to run when starting the container.
ENTRYPOINT ["/consumer"]

Yes, you guessed it! 😅 Once again, we will have to put a description in the Docker Compose file to create the container for the consumer:



# ./docker-compose.yml

version: "3.9"

services:

  # ...

  # ...

  # Create service with message consumer.
  consumer:
    container_name: consumer
    build:
      context: .
      dockerfile: Dockerfile-consumer
    environment:
      AMQP_SERVER_URL: amqp://guest:guest@message-broker:5672/
    restart: always
    networks:
      - dev-network
    depends_on:
      - sender
      - message-broker

# ...

Great! Finally, we're ready to put everything together and run the project.

↑ Table of contents

Running the project

Just run this Docker Compose command:



docker-compose up --build

Wait about 1-2 minutes and make a few HTTP request to the API endpoint with different text in msg query:



curl \
    --request GET \
    --url 'http://localhost:3000/send?msg=test'

Next, go to this address: http://localhost:15672. Enter guest both as login and password. You should see the RabbitMQ user interface, like this:

To simply view the logs and metrics inside the Docker containers, I recommend to use a command line utility, called ctop:

If you turn off the consumer container, but continue to make HTTP requests, you will see that the queue starts to accumulate messages. But as soon as the consumer is started up again, the queue will clear out because the consumer will get all the messages sent.

🎊 Congratulations, you have fully configured the message broker, sender, consumer and wrapped it all in isolated Docker containers!

↑ Table of contents

Photos and videos by

Vic Shóstak https://shostak.dev
Fatos Bytyqi https://unsplash.com/photos/Agx5_TLsIf4

P.S.

If you want more articles (like this) on this blog, then post a comment below and subscribe to me. Thanks! 😻

❗️ You can support me on Boosty, both on a permanent and on a one-time basis. All proceeds from this way will go to support my OSS projects and will energize me to create new products and articles for the community.

And of course, you can help me make developers' lives even better! Just connect to one of my projects as a contributor. It's easy!

My main projects that need your help (and stars) 👇

🔥 gowebly: A next-generation CLI tool that makes it easy to create amazing web applications with Go on the backend, using htmx, hyperscript or Alpine.js and the most popular CSS frameworks on the frontend.
✨ create-go-app: Create a new production-ready project with Go backend, frontend and deploy automation by running one CLI command.

Other my small projects: yatr, gosl, json2csv, csv2api.