Learning Go by examples: part 6 - Create a gRPC app in Go

In previous articles we created an HTTP REST API server, a CLI, a Bot for Discord and even a game for Nintendo Game Boy Advance. Today let's create another type of application: a gRPC app in Go!

gRPC

First, what is gRPC?

gRPC is a modern, open source Remote Procedure Call (RPC) framework, originally developed by Google.

"gRPC is based around the idea of defining a service, specifying the methods that can be called remotely with their parameters and return types. On the server side, the server implements this interface and runs a gRPC server to handle client calls. On the client side, the client has a stub (referred to as just a client in some languages) that provides the same methods as the server."

It uses Protocol Buffers, Google’s Open Source technology for serializing and deserializing structured data.

gRPC uses HTTP/2 for the transport layer (lower latency, response multiplexing, server-side streaming, client-side streaming or even bidirectional-streaming...)

Each RPC service is declared in a protobuf file.

From this .proto file, you can generate a client in many languages.

So, one of the power of gRPC is that is language agnostic: you can have one server in Go and several clients in Java, Python, Rust, Go...

If you have microservices that need to communicate to each other, gRPC can be a solution instead of REST API interfaces.

Initialization

We created our Git repository in the previous article, so now we just have to retrieve it locally:

$ git clone https://github.com/scraly/learning-go-by-examples.git
$ cd learning-go-by-examples

We will create a folder go-gopher-grpc for our CLI application and go into it:

$ mkdir go-gopher-grpc
$ cd go-gopher-grpc

Now, we have to initialize Go modules (dependency management):

$ go mod init github.com/scraly/learning-go-by-examples/go-gopher-grpc
go: creating new go.mod: module github.com/scraly/learning-go-by-examples/go-gopher-grpc

This will create a go.mod file like this:

module github.com/scraly/learning-go-by-examples/go-gopher-grpc

go 1.16

Before to start our super gRPC application, as good practices, we will create a simple code organization.

Create the following folders organization:

.
├── README.md
├── bin
├── go.mod
└── test-results

That's it? Yes, the rest of our code organization will be created shortly ;-).

Create our CLI application

Like the second article, we will create a CLI (Command Line Interface) application.
If you don't know Cobra I recommend you to read the CLI article before to go further.

Install Cobra:

$ go get -u github.com/spf13/cobra@latest

Generate our CLI application structure and imports:

$ cobra init --pkg-name github.com/scraly/learning-go-by-examples/go-gopher-grpc
Your Cobra application is ready at
/Users/aurelievache/git/github.com/scraly/learning-go-by-examples/go-gopher-grpc

Our application is initialized, a main.go file and a cmd/ folder has been created, our code organization is now like this:

.
├── LICENSE
├── bin
├── cmd
│   └── root.go
├── go.mod
├── go.sum
├── main.go
└── test-results

Like in the CLI article, Viper is used in root.go so we need to install it:

$ go get github.com/spf13/viper@v1.8.1

Let's create our gRPC client and server

We want a gRPC application so the first things that we need to do is to create a server and a client command:

$ cobra add client
client created at /Users/aurelievache/git/github.com/scraly/learning-go-by-examples/go-gopher-grpc

$ cobra add server
server created at /Users/aurelievache/git/github.com/scraly/learning-go-by-examples/go-gopher-grpc

Now the cmd/ folder code organisation should contains these files:

cmd
├── client.go
├── root.go
└── server.go

At this time, the go.mod file should have these following imports:

module github.com/scraly/learning-go-by-examples/go-gopher-grpc

go 1.16

require (
    github.com/spf13/cast v1.4.0 // indirect
    github.com/spf13/cobra v1.2.1
    github.com/spf13/viper v1.8.1
    golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c // indirect
    golang.org/x/text v0.3.6 // indirect
)

In order to explain to the users the goal and the usage of our app, we need to edit the root.go file:

// rootCmd represents the base command when called without any subcommands
var rootCmd = &cobra.Command{
    Use:   "go-gopher-grpc",
    Short: "gRPC app in Go",
    Long:  `gRPC application written in Go.`,
}

It's time to execute our application:

$ go run main.go
gRPC application written in Go.

Usage:
  go-gopher-grpc [command]

Available Commands:
  client      A brief description of your command
  completion  generate the autocompletion script for the specified shell
  help        Help about any command
  server      A brief description of your command

Flags:
      --config string   config file (default is $HOME/.go-gopher-grpc.yaml)
  -h, --help            help for go-gopher-grpc
  -t, --toggle          Help message for toggle

Use "go-gopher-grpc [command] --help" for more information about a command.

By default, an usage message is displayed, perfect!

Let's test our client and server commands:

$ go run main.go client
client called

$ go run main.go server
server called

OK, the client and server commands answered too.

Let's create our proto

Like we said, by default, gRPC uses Protocol Buffers.

The first step when working with Protocol Buffers is to define the structure for the data you want to serialize in a .proto file.

Let's create a gopher.proto file under a new folder pkg/gopher/:

syntax = "proto3";
package gopher;

option go_package = "github.com/scraly/learning-by-examples/go-gopher-grpc";

// The gopher service definition.
service Gopher {
  // Get Gopher URL
  rpc GetGopher (GopherRequest) returns (GopherReply) {}
}

// The request message containing the user's name.
message GopherRequest {
  string name = 1;
}

// The response message containing the greetings
message GopherReply {
  string message = 1;
}

Let's explain it.
This .proto file exposes our Gopher service which have a GetGopher function which can be called by any gRPC client written in any language.

gRPC is supported by many programming languages, so microservices that need to interact with your gRPC server can generate their own code with the .proto file in output.

option go_package line is required in order to generate Go code, the Go package's import path must be provided for every .proto file.

Generate Go code from proto

Now, we need to install Protocol Buffers v3.

For MacOs:

$ brew install protoc

Check protoc is correctly installed:

$ protoc --version
libprotoc 3.17.3

Now we need to generate the Go gRPC code thanks to protoc tool:

$ protoc --go_out=plugins=grpc:. --go_opt=paths=source_relative pkg/gopher/gopher.proto

You should have one new file in pkg/gopher folder:

pkg/gopher
├── gopher.pb.go
└── gopher.proto

gopher.go file contains generated code that we will import in our server.go file in order to register our gRPC server to Gopher service.

Let's create our gRPC server

It's time to create our gRPC server, for that we need to edit our server.go file.

First, we initialize the package, called cmd, and all dependencies/librairies we need to import:

package cmd

import (
    "context"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "log"
    "net"
    "net/http"
    "strings"

    "github.com/spf13/cobra"
    "golang.org/x/xerrors"

    pb "github.com/scraly/learning-go-by-examples/go-gopher-grpc/pkg/gopher"
    "google.golang.org/grpc"
)

Then, we initialize our constants:

const (
    port         = ":9000"
    KuteGoAPIURL = "https://kutego-api-xxxxx-ew.a.run.app"
)

We define two structs, one for our server and one for our Gopher data.

// server is used to implement gopher.GopherServer.
type Server struct {
    pb.UnimplementedGopherServer
}

type Gopher struct {
    URL string `json: "url"`
}

We improve our serverCmd run function that initialize a gRPC server, register to RPC service and start our server:

// serverCmd represents the server command
var serverCmd = &cobra.Command{
    Use:   "server",
    Short: "Starts the Schema gRPC server",

    Run: func(cmd *cobra.Command, args []string) {
        lis, err := net.Listen("tcp", port)
        if err != nil {
            log.Fatalf("failed to listen: %v", err)
        }

        grpcServer := grpc.NewServer()

        // Register services
        pb.RegisterGopherServer(grpcServer, &Server{})

        log.Printf("GRPC server listening on %v", lis.Addr())

        if err := grpcServer.Serve(lis); err != nil {
            log.Fatalf("failed to serve: %v", err)
        }
    },
}

Finally, we implement GetGopher method.

Wait, what do we want?
Oups, excuse me I forget to expain what our server will serve ^^.

Our gRPC should implement a GetGopher method that will:

• check that request is not nil and contains a not empty Gopher's name
• ask to KuteGo API information about the Gopher
• return Gopher's URL

// GetGopher implements gopher.GopherServer
func (s *Server) GetGopher(ctx context.Context, req *pb.GopherRequest) (*pb.GopherReply, error) {
        res := &pb.GopherReply{}

    // Check request
    if req == nil {
        fmt.Println("request must not be nil")
        return res, xerrors.Errorf("request must not be nil")
    }

    if req.Name == "" {
        fmt.Println("name must not be empty in the request")
        return res, xerrors.Errorf("name must not be empty in the request")
    }

    log.Printf("Received: %v", req.GetName())

    //Call KuteGo API in order to get Gopher's URL
    response, err := http.Get(KuteGoAPIURL + "/gophers?name=" + req.GetName())
    if err != nil {
        log.Fatalf("failed to call KuteGoAPI: %v", err)
    }
    defer response.Body.Close()

    if response.StatusCode == 200 {
        // Transform our response to a []byte
        body, err := ioutil.ReadAll(response.Body)
        if err != nil {
            log.Fatalf("failed to read response body: %v", err)
        }

        // Put only needed informations of the JSON document in our array of Gopher
        var data []Gopher
        err = json.Unmarshal(body, &data)
        if err != nil {
            log.Fatalf("failed to unmarshal JSON: %v", err)
        }

        // Create a string with all of the Gopher's name and a blank line as separator
        var gophers strings.Builder
        for _, gopher := range data {
            gophers.WriteString(gopher.URL + "\n")
        }

        res.Message = gophers.String()
    } else {
        log.Fatal("Can't get the Gopher :-(")
    }

    return res, nil
}

Don't forget the existing original init function:

func init() {
    rootCmd.AddCommand(serverCmd)
}

Install our dependencies

As usual, if you use external depencencies, you need to install them:

$ go get google.golang.org/grpc
$ go get golang.org/x/xerrors

Let's create our gRPC client

Now, we can create our gRPC client, for that we need to edit our client.go file.

We initialize the package, called cmd, and all dependencies/librairies we need to import:

package cmd

import (
    "context"
    "log"
    "os"
    "time"

    "google.golang.org/grpc"

    pb "github.com/scraly/learning-go-by-examples/go-gopher-grpc/pkg/gopher"
    "github.com/spf13/cobra"
)

Define our constants:

const (
    address     = "localhost:9000"
    defaultName = "dr-who"
)

We improve our clientCmd run function that:

• initialize a gRPC client
• connect to gRPC server
• call the GetGopher function with the Gopher's name
• return "URL:" + the message returned by the gRPC call

// clientCmd represents the client command
var clientCmd = &cobra.Command{
    Use:   "client",
    Short: "Query the gRPC server",

    Run: func(cmd *cobra.Command, args []string) {
        var conn *grpc.ClientConn
        conn, err := grpc.Dial(address, grpc.WithInsecure())
        if err != nil {
            log.Fatalf("did not connect: %s", err)
        }
        defer conn.Close()

        client := pb.NewGopherClient(conn)

        var name string

        // Contact the server and print out its response.
        // name := defaultName
        if len(os.Args) > 2 {
            name = os.Args[2]
        }
        ctx, cancel := context.WithTimeout(context.Background(), time.Second)
        defer cancel()
        r, err := client.GetGopher(ctx, &pb.GopherRequest{Name: name})
        if err != nil {
            log.Fatalf("could not greet: %v", err)
        }
        log.Printf("URL: %s", r.GetMessage())
    },
}

And don't forget the existing init method:

func init() {
    rootCmd.AddCommand(clientCmd)
}

Test it!

Let's start our gRPC server:

$ go run main.go server
2021/08/07 14:57:27 GRPC server listening on [::]:9000

Then, in another tab of your terminal, launch the gRPC client that call our GetGopher method with "gandalf" parameter:

$ go run main.go client gandalf
2021/08/07 14:57:35 URL: https://raw.githubusercontent.com/scraly/gophers/main/gandalf.png

Our application works properly, it answers "URL:" + the URL of the wanted Gopher.

Built it!

Your application is now ready, you just have to build it.
For that, like the previous articles, we will use Taskfile in order to automate our common tasks.

So, for this app too, I created a Taskfile.yml file with this content:

version: "3"

tasks:
    build:
        desc: Build the app
        cmds:
        - GOFLAGS=-mod=mod go build -o bin/gopher-grpc main.go 

    run: 
        desc: Run the app
        cmds:
        - GOFLAGS=-mod=mod go run main.go

    generate:
        desc: Generate Go code from protobuf
        cmds:
        - protoc --go_out=plugins=grpc:. --go_opt=paths=source_relative pkg/gopher/gopher.proto

    test:
        desc: Execute Unit Tests
        cmds:
        - gotestsum --junitfile test-results/unit-tests.xml -- -short -race -cover -coverprofile test-results/cover.out ./...

Thanks to this, we can build our app easily:

$ task build
task: [build] GOFLAGS=-mod=mod go build -o bin/gopher-grpc main.go

Let's test it again with our fresh executable binary:

$ ./bin/gopher-grpc server
2021/08/07 15:07:20 GRPC server listening on [::]:9000

And in another tab of your terminal:

$ ./bin/gopher-grpc client yoda-gopher
2021/08/07 15:07:34 URL: https://raw.githubusercontent.com/scraly/gophers/main/yoda-gopher.png

Cool, the URL of our cute Yoda Gopher! :-)

Unit tests?

Now, I can deploy my gRPC server/microservice in production environment, cool, thanks, bye!

Uh... wait for it, before that, as you know it's important to test our applications, in order to know if our app is working like we want to, before to deploy it.
Unit Tests are a powerful practice and in Go you can even create Unit Tests for gRPC apps.

With Golang, you don't need to import an external package, like JUnit in Java. It's integrated in core package with the command go test.

Let's execute our Unit Tests:

$ go test
?       github.com/scraly/learning-go-by-examples/go-gopher-grpc    [no test files]

As you can see, 0 unit test were run successfully, normal ^^
We will deal with them in the next section, but before that, we'll discover a useful tool gotestsum.

Gotestsum

Gotestsum, what is this new tool? Go test is not enough?

Let's answer this question. One of the benefits of Go is its ecosystem of tools that allow us to make our lives easier.

Like we saw, the test tool is integrated with Go. This is convenient, but not very user-friendly and integrable in all CI/CD solutions, for example.

That's why gotestsum, a small Go utility, designed to run tests with go test improves the display of results, making a more human-readable, practical report with possible output directly in JUnit format. And it's one of the good practice given by this article ;-).

Install it:

$ go get gotest.tools/gotestsum

Let's execute our task test command that use gotestsum tool:

$ task test
task: [test] gotestsum --junitfile test-results/unit-tests.xml -- -short -race -cover -coverprofile test-results/cover.out ./...
∅  . (3ms)
∅  cmd
∅  pkg/gopher

DONE 0 tests in 1.409s

The code above shows that we use the gotestsum tool to run our unit tests and that test results are exported in JUnit format in a file, named test-results/unit-tests.xml.

Here an example of a generated test result file in JUnit format:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite tests="0" failures="0" time="0.000000" name="github.com/scraly/learning-go-by-examples/go-gopher-grpc" timestamp="2021-08-11T14:23:36+02:00">
        <properties>
            <property name="go.version" value="go1.16.5 darwin/amd64"></property>
        </properties>
    </testsuite>
    <testsuite tests="0" failures="0" time="0.000000" name="github.com/scraly/learning-go-by-examples/go-gopher-grpc/cmd" timestamp="2021-08-11T14:23:36+02:00">
        <properties>
            <property name="go.version" value="go1.16.5 darwin/amd64"></property>
        </properties>
    </testsuite>
    <testsuite tests="0" failures="0" time="0.000000" name="github.com/scraly/learning-go-by-examples/go-gopher-grpc/pkg/gopher" timestamp="2021-08-11T14:23:36+02:00">
        <properties>
            <property name="go.version" value="go1.16.5 darwin/amd64"></property>
        </properties>
    </testsuite>
</testsuites>

How to Test gRPC?

Our app is a gRPC client/server, so this means that when we call the getGopher method, a client/server communication is triggered, but no question to test the gRPC calls in our unit tests. We will only test the intelligence of our application.

As we have seen, our gRPC server is based on a protobuf file named pkg/gopher/gopher.proto.

The standard Go library provides us a package that allows us to test our Go program. A test file in Go must be placed in the same folder as the file we want to test and finished with the _test.go extension. This formalism must be followed so that the Go executable recognizes our test files.

The first step is to create a server_test.go file that is placed next to server.go.

We are going to name the package of this test file cmd_test and we will start by importing the testing package and creating the function we are going to test, like that:

package cmd_test

import "testing"

func TestGetGopher(t *testing.T) {

}

/!\ Warning: Each test function must be written as funcTest***(t *testing.T), where *** represents the name of the function we want to test.

Let’s Write Tests With Table-Driven Tests

In our application, we will not test everything, but we will start by testing our business logic, the intelligence of our application. In our app, what interests us is what is inside server.go, especially the GetGopher function:

func (s *server) GetGopher(ctx context.Context, req *pb.GopherRequest) (*pb.GopherReply, error) {
    res := &pb.GopherReply{}
...

As you can see, in order to cover the maximum amount of our code, we will have to test at least three cases:

• The request is nil.
• The request is empty (the name field is empty).
• The name field is filled in the request.

Table Driven Tests

Instead of creating a test case method, and copying-and-pasting it, we're going to follow Table Driven Tests, which will make life a lot easier.

Writing good tests is not easy, but in many situations, you can cover a lot of things with table driven tests: each table entry is a complete test case with the inputs and the expected results. Sometimes additional information is provided. The test output is easily readable. If you usually find yourself using copy and paste when writing a test, ask yourself if refactoring in a table-driven test may be a better option.

Given a test case table, the actual test simply scans all entries in the table and performs the necessary tests for each entry. The test code is written once and is depreciated on all table entries. It is therefore easier to write a thorough test with good error messages.

First, install needed external dependency:

$ go get github.com/onsi/gomega

Let's define our package and dependencies:

package cmd_test

import (
    "context"
    "testing"

    cmd "github.com/scraly/learning-go-by-examples/go-gopher-grpc/cmd"
    pb "github.com/scraly/learning-go-by-examples/go-gopher-grpc/pkg/gopher"

    . "github.com/onsi/gomega"
)

Then, we define our test case in the TestGetGopher function:

func TestGetGopher(t *testing.T) {
    s := cmd.Server{}

    testCases := []struct {
        name        string
        req         *pb.GopherRequest
        message     string
        expectedErr bool
    }{
        {
            name:        "req ok",
            req:         &pb.GopherRequest{Name: "yoda-gopher"},
            message:     "https://raw.githubusercontent.com/scraly/gophers/main/yoda-gopher.png\n",
            expectedErr: false,
        },
        {
            name:        "req with empty name",
            req:         &pb.GopherRequest{},
            expectedErr: true,
        },
        {
            name:        "nil request",
            req:         nil,
            expectedErr: true,
        },
    }

The good practice is to provide a name for our test case, so if an error occurs during its execution the name of the test case will be written and we will see easily where is our error.

Then, I loop through all the test cases. I call my service and depending on whether or not I wait for an error, I test its existence, otherwise I test if the result is that expected:

    for _, tc := range testCases {
        testCase := tc
        t.Run(testCase.name, func(t *testing.T) {
            t.Parallel()
            g := NewGomegaWithT(t)

            ctx := context.Background()

            // call
            response, err := s.GetGopher(ctx, testCase.req)

            t.Log("Got : ", response)

            // assert results expectations
            if testCase.expectedErr {
                g.Expect(response).ToNot(BeNil(), "Result should be nil")
                g.Expect(err).ToNot(BeNil(), "Result should be nil")
            } else {
                g.Expect(response.Message).To(Equal(testCase.message))
            }
        })
    }
}

Aurélie, your code is nice! But why creating a new variable, testCase, which takes a value, tc, when you could have used tc directly?

In short, without this line, there is a bug with the t.Parallel() well known to Gophers — we use a closure that is in a go routine. So, instead of executing three test cases: "req ok", "req with empty name", and "nil request", there would be three tests runs but always with the values of the first test case :-(.

And, what is Gomega?

Gomega is a Go library that allows you to make assertions. In our example, we check if what we got is null, not null, or equal to an exact value, but the gomega library is much richer than that.

Let's run our Unit Tests!

To run your newly created Unit Tests, if you use VisualStudio Code, you can directly run them in your IDE; it's very convenient:

First, open the server_test.go file.

Then, click in the “run package tests” link:

Now, open the server.go file:

The code highlighted in green is the code that is covered by the tests — super! And red lines are code not covered by our Unit Tests ;-).

Otherwise, we can run all the unit tests of our project in the command line thanks to our marvelous Taskfile:

$ task test
task: [test] gotestsum --junitfile test-results/unit-tests.xml -- -short -race -cover -coverprofile test-results/cover.out ./...
∅  . (1ms)
✓  cmd (1.388s) (coverage: 41.5% of statements)
∅  pkg/gopher

DONE 4 tests in 7.787s

Cool, it's the begining of Unit Testing journey :-).

If you're in the habit of copying paste when writing your test cases, I think you'll have to seriously take a look at Table Driven Tests :-). It's really a good practice to follow when writing unit tests and as As we have seen, writing unit tests that cover our code becomes child's play.

Conclusion

As you have seen in this article and previous articles, it's possible to create multiple different applications in Go... and to write Unit Tests without copying and pasting code from StackOverFlow ;-).

All the code of our gRPC app in Go is available in: https://github.com/scraly/learning-go-by-examples/tree/main/go-gopher-grpc

In the following articles we will create others kind/types of applications in Go.

Hope you'll like it.