DEV Community

Cover image for GO Protobuf Encoding/Decoding integrated with C
brundhasv
brundhasv

Posted on

GO Protobuf Encoding/Decoding integrated with C

Protobuf messages are gaining popularity across many domains. There are libraries like nanopb in C to support the protobuf encoding/decoding. However they are slow and also pose significant challenge in implementation. GoLang protobuf encoding/decoding is fast and provides support of unmarshalling data from JSON to proto format.

We will be creating a C shared library containing functions that read from json file, protobuf encode and decode.

Full Code can be found here:

GitHub logo brundhasv / GoC-Protobuf

Go functions integrated with C

GoC Protobuf Encoding/Decoding

Go functions for Protobuf Encoding/decoding integrated with C

Usage:

protoc --go_out=. Student.proto

go build -o student.so -buildmode=c-shared Student.pb.go student_en_dc.go

gcc -o student student_cgo.c ./student.so

Go program can also be run in standalone mode:

go run student_en_dc.go Student.pb.go

Explanation of Code here:

https://dev.to/brundhasv/go-protobuf-encoding-decoding-integrated-with-c-3ig0




GoLang Code

Lets start with Student.proto. I am using proto3 for this example:

syntax="proto3";

package main;

message Parents {
      string dad=1;
      string mom=2;
}

enum Gender {
    Female = 0;
    Male = 1;
}

message Student {
      string name = 1;
      int32 age = 2;
      Parents parents=3;
      Gender gender=4;
}

message Students {
      repeated bytes StudentEntry=1;
}
Enter fullscreen mode Exit fullscreen mode

As you can see above proto includes enum, nested proto and repeated bytes. Student represents a student's data, each student's data will be encoded and added to Students StudentEntry and all the encoded entries will be encoded again.

Below is input JSON file student_list.json which is used to populate proto fields. We have to use the same names and nested levels for the unmarshal of Json to proto.

{
"0" : {
    "Name": "Elliot",
    "Age" : 15,
    "Parents" : {
            "Dad" : "George",
            "Mom" : "Elena"
    },
    "Gender" : 0
  },
"1" : {
    "Name": "David",
    "Age" : 14,
    "Parents" : {
            "Dad" : "Keith",
            "Mom" : "Sarah"
    },
    "Gender" : 1
   }
}
Enter fullscreen mode Exit fullscreen mode

Encoding Protobuf from JSON

To integrate with C, library C is used to convert GoData to C datatypes. Comment export is used to export functions to C lib.
We will create a new GoLang program file student_en_dc.go and add encode and decode functions to it.
Json file is input parameter of this function get_student_enbuf passed from C. This function returns encoded buffer and its length to C.

//export get_student_enbuf
func get_student_enbuf(student_json_file *C.char) (unsafe.Pointer,C.int) {
       .
       .
       return C.CBytes(students_en),C.int(int32(len(students_en)))
Enter fullscreen mode Exit fullscreen mode

First step in encoding is to Unmarshall JSON to proto. As there are more than one student entries, I have added entries using number keys. This cannot be directly unmarshalled, we would have to extract json data of each student and then unmarshal. I am first unmarshalling json file data to an interface. Below is the code:

jsonbytes, _ := ioutil.ReadAll(jsonFile)
var parsed_proto map[string]interface{}
json.Unmarshal(jsonbytes, &parsed_proto)
Enter fullscreen mode Exit fullscreen mode

Then parse each interface to obtain student json data and unmarshall it into our proto. Aftermath proto is encoded as shown.

    for i=0;i<len(parsed_proto);i++ {
            selectedjb, err := json.Marshal(parsed_proto[strconv.Itoa(i)])
            if err != nil {
                log.Println("Error in reading json index",err)
                return nil,0
            }
            var parsed_jf map[string]interface{}
            //Each student data to proto
            json.Unmarshal(selectedjb, &parsed_jf)
            student := &Student{}
            //Proto Encoding of student
            json.Unmarshal(selectedjb, student)

Enter fullscreen mode Exit fullscreen mode

Encoded data is added to StudentEntry of Students. Then Students data is encoded. We have the encoded buffer.

            students.StudentEntry=append(students.StudentEntry,student_en)
    }
    students_en, err := proto.Marshal(students)
Enter fullscreen mode Exit fullscreen mode

Decoding protobuf

Students is decoded first following all the student entries.

Function decode_student_enbuf takes encoded data and its length as input passed from C and returns decoded buffer to C.

//export decode_student_enbuf
func decode_student_enbuf(data *C.char,leng C.int) *C.char {
           .
           .
           return C.CString(decoded_student)
}
Enter fullscreen mode Exit fullscreen mode

Before decode, *char is converted to GoBytes and then unmarshalled to proto.

new_students := &Students{}
student_en:=C.GoBytes(unsafe.Pointer(data),leng)
err := proto.Unmarshal(student_en, new_students)
Enter fullscreen mode Exit fullscreen mode

Decoded students data is used to obtain each StudentEntry and is decoded. Decoded student is appended to a string and is returned to C.

for i=0;i<len(new_students.StudentEntry);i++ {
        new_student := &Student{}
        err = proto.Unmarshal(new_students.StudentEntry[i], new_student)
        decoded_student=decoded_student+"\n"+proto.MarshalTextString(new_student)
Enter fullscreen mode Exit fullscreen mode

Building C-Shared file

Lets compile proto to generate Student.pb.go.Then build c-shared file.

protoc --go_out=. Student.proto
go build  -o student.so -buildmode=c-shared Student.pb.go student_en_dc.go
Enter fullscreen mode Exit fullscreen mode

student.so and student.h files are generated.
Following lines would be present in the generated header file. Golang function get_student_enbuf returns tuple(more than one return parameters), this is converted to struct when header files are generated as seen below.

/* Return type for get_student_enbuf */
struct get_student_enbuf_return {
        void* r0;
        int r1;
};

extern struct get_student_enbuf_return get_student_enbuf(char* p0);

extern char* decode_student_enbuf(char* p0, int p1);
Enter fullscreen mode Exit fullscreen mode

C Code

From C side, generated header file is included. Students list JSON file is passed to function get_student_enbuf, returned encoded buffer and its length is populated in struct stu. The same are passed to decode function decode_student_enbuf, the returned decoded string is printed.
Finally both encoded and decoded buffers are freed.

#include <stdio.h>
#include "student.h"
#include <string.h>
#include <stdlib.h>
#include <stdbool.h>

struct get_student_enbuf_return stu;

int main() {
    stu = get_student_enbuf("student_list.json");
    int stu_list_len = stu.r1;
    if(stu.r0!=NULL) {
        char *decoded_stu = decode_student_enbuf((char*)stu.r0,stu.r1);
        if(decoded_stu != NULL){
                printf("%s",decoded_stu);
                free((char*)decoded_stu);
        }
    free((char*)stu.r0);
    }
    return 0;
}
Enter fullscreen mode Exit fullscreen mode

Compiling and executing C

Link with the created student.so, create object student and run the object.

gcc -o student student_cgo.c ./student.so

./student
name: "Elliot"
age: 15
parents: <
  dad: "George"
  mom: "Elena"
>

name: "David"
age: 14
parents: <
  dad: "Keith"
  mom: "Sarah"
>
gender: Male

Enter fullscreen mode Exit fullscreen mode

Full Code can be found here:

GitHub logo brundhasv / GoC-Protobuf

Go functions integrated with C

GoC Protobuf Encoding/Decoding

Go functions for Protobuf Encoding/decoding integrated with C

Usage:

protoc --go_out=. Student.proto

go build -o student.so -buildmode=c-shared Student.pb.go student_en_dc.go

gcc -o student student_cgo.c ./student.so

Go program can also be run in standalone mode:

go run student_en_dc.go Student.pb.go

Explanation of Code here:

https://dev.to/brundhasv/go-protobuf-encoding-decoding-integrated-with-c-3ig0






Top comments (0)