DEV Community

Aceld
Aceld

Posted on • Edited on

12.Data Transmission Protocol: Protocol Buffers

#zinx #go

[Zinx]

<1.Building Basic Services with Zinx Framework>
<2. Zinx-V0.2 Simple Connection Encapsulation and Binding with Business>
<3.Design and Implementation of the Zinx Framework's Routing Module>
<4.Zinx Global Configuration>
<5.Zinx Message Encapsulation Module Design and Implementation>
<6.Design and Implementation of Zinx Multi-Router Mode>
<7. Building Zinx's Read-Write Separation Model>
<8.Zinx Message Queue and Task Worker Pool Design and Implementation>
<9. Zinx Connection Management and Property Setting>

[Zinx Application - MMO Game Case Study]

<10. Application Case Study using the Zinx Framework>
<11. MMO Online Game AOI Algorithm>
<12.Data Transmission Protocol: Protocol Buffers>
<13. MMO Game Server Application Protocol>
<14. Building the Project and User Login>
<15. World Chat System Implementation>
<16. Online Location Information Synchronization>
<17. Moving position and non-crossing grid AOI broadcasting>
<18. Player Logout>
<19. Movement and AOI Broadcast Across Grids>

MMO-GAME Source Code

https://github.com/aceld/zinx/tree/master/zinx_app_demo/mmo_game

In the current section of the server-side project, the data protocol used for communication between the client and the server is represented using Protocol Buffers. This section will provide a separate introduction to the installation and basic usage of Protocol Buffers. If readers are already familiar with Protocol Buffers syntax and related protocols, they can skip this section.

12.1 Introduction to Protocol Buffers

Google Protocol Buffers, commonly referred to as Protobuf, is a lightweight and efficient structured data storage format developed by Google. It is platform-agnostic, language-agnostic, extensible, and can be used in various domains such as communication protocols and data storage. This makes it suitable for data storage and usage across different applications. It is also well-suited for data exchange between different languages. By implementing the same protocol format, typically defined in a ".proto" file, it can be compiled into different language versions and integrated into respective projects. This enables different languages to parse data serialized using Protobuf. Currently, the official website provides support for languages including C/C++, Python, Java, Go, and more. Google open-sourced Protocol Buffers on July 7, 2008.

12.2 Data Exchange Format

ProtoBuf is, in essence, a data exchange format, which serves as a layer of data protocol built upon application protocols. This is distinct from the previously discussed TLV protocol in this book. Data exchange formats possess superior readability and data assembly capabilities. However, the trade-off is that their parsing performance tends to be relatively poorer. ProtoBuf has been designed with optimizations that specifically target performance aspects.

ProtoBuf's relative advantages include:

(1) Serialization results in smaller data volumes compared to Json and XML, making it suitable for network transmission.
(2) Support for cross-platform and multi-language usage.
(3) Good message format upgrade and compatibility.
(4) Fast serialization and deserialization speed, faster than Json processing speed.

ProtoBuf's relative disadvantages include:

(1) Less widespread adoption compared to XML and JSON.
(2) Binary format leads to poorer readability.
(3) Lack of self-description.

12.3 Installing Protobuf Environment

This section primarily covers the installation of the Protobuf environment for the Golang programming language on the Linux (Ubuntu) platform. For other operating systems, you can refer to other resources or official documentation.

12.3.1 Installing Protobuf Compilation Tools

Step 1: Download the Protobuf source code:

Step 1: Download the Protobuf source code:
Enter fullscreen mode Exit fullscreen mode

Alternatively, you can directly extract the compressed package:

unzip protobuf.zip
Enter fullscreen mode Exit fullscreen mode

Step 2: Install the required library dependencies:

sudo apt-get install autoconf automake libtool curl make g++ unzip libffi-dev -y
Enter fullscreen mode Exit fullscreen mode

Step 3: Generate the Configure configuration file:

cd protobuf/
./autogen.sh
Enter fullscreen mode Exit fullscreen mode

Step 4: Configure the environment:

./configure
Enter fullscreen mode Exit fullscreen mode

Step 5: Compile the source code (this process may take a while):

make
Enter fullscreen mode Exit fullscreen mode

Step 6: Install the Protobuf-related executables:

sudo make install
Enter fullscreen mode Exit fullscreen mode

Step 7: Refresh the dynamic libraries required by Protobuf. This step is crucial to avoid dynamic library path association failures that might prevent Protobuf-related command programs from starting:

sudo ldconfig
Enter fullscreen mode Exit fullscreen mode

Step 8: Test whether Protobuf is installed successfully by executing the following command:

protoc -h
Enter fullscreen mode Exit fullscreen mode

If it displays the expected information without any errors, the installation is successful.

12.3.2 Installing Protobuf Go Language Plugin

Since Protobuf doesn't directly support the Go programming language, developers need to manually install the relevant plugin.

Step 1: Obtain the proto Package (Go Language's proto API Interface)

go get -v -u github.com/golang/protobuf/proto
go get -v -u github.com/golang/protobuf/protoc-gen-go
Enter fullscreen mode Exit fullscreen mode

Step 2: Compile the Code for Golang Support Plugin

cd $GOPATH/src/github.com/golang/protobuf/protoc-gen-go/
go build
Enter fullscreen mode Exit fullscreen mode

Step 3: Place the Generated protoc-gen-go Executable

Place the generated protoc-gen-go executable in the /bin directory (or any other $PATH directory):

sudo cp protoc-gen-go /bin/
Enter fullscreen mode Exit fullscreen mode

12.4 Protobuf Syntax

Protobuf commonly refers to user-defined struct types as "messages". The definition of a Protobuf message is typically written in a file with the ".proto" extension.

12.4.1 A Simple Example

Below is an example of a Protobuf file. You can name it "file.proto". The content of the file is as follows:

//file.proto

syntax = "proto3";            // Specify version information, an error will occur if not specified
package pb;                   // Package name for generating Go files later

// 'message' is a keyword, used to define a message type
message Person {
    string name = 1;           // Name
    int32 age = 2;              // Age
    repeated string emails = 3; // Email addresses (repeated indicates the field allows duplicates)
    repeated PhoneNumber phones = 4;   // Phone numbers
}

// 'enum' is a keyword, used to define an enumeration type
enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
}

// 'message' is a keyword, used to define a message type that can be nested within another message type
message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
}
Enter fullscreen mode Exit fullscreen mode

A Proto file must have several essential elements:

At the top of the file, there needs to be a syntax version information. In the current project example, the Proto version used is "proto3".
A package declaration, which specifies the package name for generating Go files. This is typically relevant in scenarios using Golang applications.

12.4.2 Message Format Explanation

A message consists of fields, and the format of each field in a message is as follows:
(Field modifier +) Data type + Field name + Unique numeric tag

The unique numeric tags, like 1 and 2 in PhoneNumber:

message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
}
Enter fullscreen mode Exit fullscreen mode

represent a unique identifier for each field within the message. These numeric tags are used to identify fields in the binary format of the message and cannot be repeated within the same message. Once a message is defined, these tags cannot be changed.

12.4.3 Data Types

The data types supported by the Protobuf protocol and their corresponding types in Golang are explained in Table 21-1 below:

Table 12-1: Protobuf Data Types

.proto Type Go Type Notes
double float64 64-bit floating-point number
float float32 32-bit floating-point number
int32 int32 Variable-length encoding; inefficient for negative values; use sint64 for negative values if possible
uint32 uint32 Variable-length encoding
uint64 uint64 Variable-length encoding
sint32 int32 Variable-length encoding; much more efficient than int32 for negative values
sint64 int64 Variable-length encoding; signed integer value; more efficient encoding than standard int64
fixed32 uint32 4 bytes; more efficient than uint32 if the value is always greater than 228
fixed64 uint64 8 bytes; more efficient than uint64 if the value is always greater than 256
sfixed32 int32 4 bytes
sfixed64 int64 8 bytes
bool bool 1 byte
string string The string must be UTF-8 or 7-bit ASCII encoded text
bytes []byte May contain arbitrary sequences of bytes

In the above table, if you need to define a data type in Golang, you can refer to the corresponding ".proto Type" defined in the Proto protocol file.

12.4.4 Default Values

When a message is being parsed and the encoded information does not include a specific element, the corresponding field in the parsed object is set to a default value. Different data types have default values as specified in Table 12-2:

Table 12-2: Protobuf Default Values

Data Type Range Default Value
String types Default is an empty string
Binary types Default is an empty byte array
Boolean types Default is false
Numeric types Default is 0

12.5 Compiling Protobuf

You can compile ".proto" files into code using the protocol compiler as follows:

$ protoc --proto_path=IMPORT_PATH --go_out=DST_DIR path/to/file.proto
Enter fullscreen mode Exit fullscreen mode

In the above command, the "protoc" instruction is the compiler of the previously installed Protobuf protocol. The two parameters are defined as follows:

(1) --proto_path specifies the path for importing packages from ".proto" files. Multiple paths can be specified. If ignored, the current directory is assumed by default.
(2) --go_out specifies the folder where the generated Go language code files will be placed.

Developers can also compile multiple .proto files simultaneously using the following approach:

protoc --go_out=./ *.proto
Enter fullscreen mode Exit fullscreen mode

During compilation, the Protobuf compiler will generate ".pd.go" files from ".proto" files. These files should not be modified by developers. If changes are needed, the ".proto" file should be edited and then recompiled. The ".pd.go" files can be directly imported and referenced in program code. They include predefined data structures and related methods.

12.6 Golang Programming with Protobuf Protocol

In the previous section, the Protobuf file defined the package as "pb," and the generated pb files were placed in the "protocolbuffer_excise" directory under the current path. Now, you can write Golang code to utilize the defined data protocol. The code is as shown below:

package main

import (
    "fmt"
    "github.com/golang/protobuf/proto"
    "protocolbuffer_excise/pb"
)

func main() {
    person := &pb.Person{
        Name:   "Aceld",
        Age:    16,
        Emails: []string{"https://github.com/aceld", "https://yuque.com/aceld"},
        Phones: []*pb.PhoneNumber{
            &pb.PhoneNumber{
                Number: "13113111311",
                Type:   pb.PhoneType_MOBILE,
            },
            &pb.PhoneNumber{
                Number: "14141444144",
                Type:   pb.PhoneType_HOME,
            },
            &pb.PhoneNumber{
                Number: "19191919191",
                Type:   pb.PhoneType_WORK,
            },
        },
    }

    data, err := proto.Marshal(person)
    if err != nil {
        fmt.Println("marshal err:", err)
    }

    newdata := &pb.Person{}
    err = proto.Unmarshal(data, newdata)
    if err != nil {
        fmt.Println("unmarshal err:", err)
    }

    fmt.Println(newdata)
}
Enter fullscreen mode Exit fullscreen mode

The above code can now use the defined "Person" data protocol. The Marshal() method encodes the in-memory "Person" struct into serialized Proto data. The Unmarshal() method parses the serialized binary data into a "Person" struct for use in business code.

MMO-GAME Source Code

https://github.com/aceld/zinx/tree/master/zinx_app_demo/mmo_game

[Zinx]

<1.Building Basic Services with Zinx Framework>
<2. Zinx-V0.2 Simple Connection Encapsulation and Binding with Business>
<3.Design and Implementation of the Zinx Framework's Routing Module>
<4.Zinx Global Configuration>
<5.Zinx Message Encapsulation Module Design and Implementation>
<6.Design and Implementation of Zinx Multi-Router Mode>
<7. Building Zinx's Read-Write Separation Model>
<8.Zinx Message Queue and Task Worker Pool Design and Implementation>
<9. Zinx Connection Management and Property Setting>

[Zinx Application - MMO Game Case Study]

<10. Application Case Study using the Zinx Framework>
<11. MMO Online Game AOI Algorithm>
<12.Data Transmission Protocol: Protocol Buffers>
<13. MMO Game Server Application Protocol>
<14. Building the Project and User Login>
<15. World Chat System Implementation>
<16. Online Location Information Synchronization>
<17. Moving position and non-crossing grid AOI broadcasting>
<18. Player Logout>
<19. Movement and AOI Broadcast Across Grids>

Author:
discord: https://discord.gg/xQ8Xxfyfcz
zinx: https://github.com/aceld/zinx
github: https://github.com/aceld
aceld's home: https://yuque.com/aceld

Top comments (0)

Read next

pankaj892 profile image

Devops pipeline on a Golang Web App with Gitops and Kubernetes

pankaj892 -

faizanabbas62 profile image

Leveraging Go Tailwind Template (GoTTH) for Efficient Microservices Architecture

Faizan Abbas -

patricia_chebet profile image

Setting up and utilizing private modules in Go

Patricia Chebet -

daviducolo profile image

Building an Efficient Text Compression Algorithm Inspired by Silicon Valley’s Pied Piper

Davide Santangelo -