DEV Community

Cover image for Relaying a Microservice JSON Response to the Client by Unmarshalling Go Structs
Sebastian Roy
Sebastian Roy

Posted on • Originally published at sebastianroy.de

Relaying a Microservice JSON Response to the Client by Unmarshalling Go Structs

#go #api

In this post we will transform a JSON response from a microservice by adding metadata like a name, date or a unique identifier and relaying it to the user client. This is part of my journey in building AI SaaS app on iOS for researchers, as an example we will pretend to send data to an AI object identifier for zoo animals.

The past week I have been working on connecting an image analysis microservice to the user frontend. We can send an image via post request and retrieve an array of x and y-coordinates as well as a label for the identified object and a confidence parameter.

{"success": true,
 "predictions": [
        {
            "confidence": 0.69146144,
            "label": "Lion",
            "y_min": 682,
            "x_min": 109,
            "y_max": 707,
            "x_max": 186
        },
        {
            "confidence": 0.7,
            "label": "Pinguin",
            "y_min": 782,
            "x_min": 209,
            "y_max": 717,
            "x_max": 196
        }]
}
Enter fullscreen mode Exit fullscreen mode

In this post I will talk about transforming the JSON response and adding metadata like a name, date or a unique identifier. You could also convert the data, so instead of forwarding x_min and x_max coordinates, we could relay X-Min as coordinate and width instead of x_max. In order to distinguish on the client if an object was detected by an algorithm or a human we add a new field method. This could also be a more complicated object to store the version of the AI algorithm. The relayed response should look like this

{"name": "Zoo Image Analysis",
 "xid": "9m4e2mr0ui3e8a215n4g"
 "predictions": [
        {
            "confidence": 0.69146144,
            "label": "Lion",
            "x": 682,
            "y": 109,
            "width": 77, 
            "height": 25,
            "method": "AI"
        },
        {
            "confidence": 0.7,
            "label": "Penguin",
            "x": 782,
            "y": 209,
            "width":  -13,
            "height": -65,
            "method": "AI"
        }]
}
Enter fullscreen mode Exit fullscreen mode

To do this we will create a ResponseAdapter struct that mimics the response structure of the AI microservice as well as a PredictionAdapter. Then we create a Response and a Prediction struct that contains the fields and methods we want to relay to our client.

  1. Print unknown JSON data with an empty interface
  2. Convert JSON data into a usable struct to access data fields
  3. Rewrite into testable functions
  4. Implementing custom UnmarshalJSON
  5. Conclusion

When retrieving some JSON data from an API, we cannot call those properties like with data.name. Converting this data into structs takes some unmarshalling, mapping and interfacing. So let’s play with some JSON handling examples to step by step arrive at our goal, which is making data from an API call usable.

You can follow these steps using your local tooling in VS Code and go-tools or just use Go Playground without any installation.

Printing unknown JSON data with an empty interface

To have some sample data, we read some JSON data from an AI microservice as a multiline string into a byte array.

var data = []byte(`{"success": true,
    "predictions": [
        {
            "confidence": 0.69146144,
            "label": "Lion",
            "y_min": 682,
            "x_min": 109,
            "y_max": 707,
            "x_max": 186
        },
        {
            "confidence": 0.7,
            "label": "Penguin",
            "y_min": 782,
            "x_min": 209,
            "y_max": 717,
            "x_max": 196
        }]
    }`)
Enter fullscreen mode Exit fullscreen mode

Next we create a temporary empty interface

var temp map[string]interface{}
Enter fullscreen mode Exit fullscreen mode

Then we unmarshall the data into this variable temp. This requires an import of „encoding/json“.

json.Unmarshal(data, &temp)
Enter fullscreen mode Exit fullscreen mode

Catch any unmarshalling errors

if err != nil {
        fmt.Println(err)
}
Enter fullscreen mode Exit fullscreen mode

Then we can print out temp.

fmt.Println(temp)
Enter fullscreen mode Exit fullscreen mode

The result should be

map[predictions:[map[confidence:0.69146144 label:Lion x_max:186 x_min:109 y_max:707 y_min:682] map[confidence:0.7 label:Penguin x_max:196 x_min:209 y_max:717 y_min:782]] success:true]
Enter fullscreen mode Exit fullscreen mode

Great. If we want to access just the title

fmt.Println(temp.predictions[0])
Enter fullscreen mode Exit fullscreen mode

we get an error.

./prog.go:40:19: temp.predictions undefined (type map[string]interface{} has no field or method predictions)
Enter fullscreen mode Exit fullscreen mode

Ok, let’s take care about this next. The listing for this section.

package main

import (
    "encoding/json"
    "fmt"
)

func main() {

    var data = []byte(`{"success": true,
    "predictions": [
        {
            "confidence": 0.69146144,
            "label": "Lion",
            "y_min": 682,
            "x_min": 109,
            "y_max": 707,
            "x_max": 186
        },
        {
            "confidence": 0.7,
            "label": "Penguin",
            "y_min": 782,
            "x_min": 209,
            "y_max": 717,
            "x_max": 196
        }]
    }`)

    var temp map[string]interface{}

    //var prediction PredictionsAdapter

    err := json.Unmarshal(data, &temp)

    if err != nil {
        fmt.Println(err)
    }

    //fmt.Println(temp.predictions[0])
}
Enter fullscreen mode Exit fullscreen mode

Accessing data fields

Assuming we know the expected data structure in advance, lets create a struct.

type ResponseAdapter struct {
    Success     bool                 `json:"success"`
    Predictions []PredictionsAdapter `json:"predictions"`
}
Enter fullscreen mode Exit fullscreen mode

Scroll up to the first JSON response to see where this is coming from. This is mirroring the response we get from the AI microservice. To be exported, the names must start with a capital letter. []PredictionsAdapter means that we expect multiple objects or an array of type PredictionsAdapter.

type PredictionsAdapter struct {
    Confidence float64 `json:"confidence"`
    Label      string  `json:"label"`
    YMin       int     `json:"y_min"`
    XMin       int     `json:"x_min"`
    YMax       int     `json:"y_max"`
    XMax       int     `json:"x_max"`
}
Enter fullscreen mode Exit fullscreen mode

We extend the PredictionsAdapter with two functions Width() and Height(), since we need this information later for our client response with Predictions.

func (p PredictionsAdapter) Width() int {

    return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

    return p.YMax - p.YMin
}
Enter fullscreen mode Exit fullscreen mode

Instead of unmarshalling it into an empty interface, we use a variable of type ResponseAdapter.

var responseAdapter ResponseAdapter

json.Unmarshall(data, &responseAdapter)

fmt.Println(responseAdapter.Predictions[0])
Enter fullscreen mode Exit fullscreen mode

The & implies that we are referring to the address in memory. And thats how we can access JSON data from go as structs :).

{0.69146144 Lion 682 109 707 186}
Enter fullscreen mode Exit fullscreen mode

The full listing

package main

import (
    "encoding/json"
    "fmt"
)

type ResponseAdapter struct {
    Success     bool                 `json:"success"`
    Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
    Confidence float64 `json:"confidence"`
    Label      string  `json:"label"`
    YMin       int     `json:"y_min"`
    XMin       int     `json:"x_min"`
    YMax       int     `json:"y_max"`
    XMax       int     `json:"x_max"`
}

func (p PredictionsAdapter) Width() int {

    return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

    return p.YMax - p.YMin
}

func main() {

    var data = []byte(`{"success": true,
    "predictions": [
        {
            "confidence": 0.69146144,
            "label": "Lion",
            "y_min": 682,
            "x_min": 109,
            "y_max": 707,
            "x_max": 186
        },
        {
            "confidence": 0.7,
            "label": "Penguin",
            "y_min": 782,
            "x_min": 209,
            "y_max": 717,
            "x_max": 196
        }]
    }`)

    //var temp map[string]interface{}

    var responseAdapter ResponseAdapter

    err := json.Unmarshal(data, &responseAdapter)

    if err != nil {
        fmt.Println(err)
    }

    fmt.Println(responseAdapter.Predictions[0])
}
Enter fullscreen mode Exit fullscreen mode

Testing

I find it always motivating to write something that works and see some result first. To adapt good practices let’s now rewrite everything into a testable function and get rid of main(). If you use Go Playground, it will automatically run the test functions in absence of main().

First import "testing" and "github.com/google/go-cmp/cmp". It is a library to compare expectations want with actual result got of a function. It checks if both are equal with if !cmp.Equal(want, got) and returns an error with the differences otherwise.

The declaration of datawe plug out of the functional context such that it is accessible from each testing function.

Most of what has been happening in mainwe will rename to ParseImageAnalysisResponse which will receive the JSON content of type []byte and return a ResponseAdapter and an error.

The test function has the same name with prefix TestXxx and a reference to *testing.T which enables automated testing in go with go test. The first line then enables parallel test execution with t.Parallel().

Then we declare our expected result. This time we don’t use the JSON format, since we expect data as a go struct. Thats why we state the type before opening curly parenthesises want := ResponseAdapter{... .}

want := ResponseAdapter{
        Success: true,
        Predictions: []PredictionsAdapter{
            {
                Confidence: 0.69146144,
                Label:      "Lion",
                YMin:       682,
                XMin:       109,
                YMax:       707,
                XMax:       186,
            },
            {
                Confidence: 0.7,
                Label:      "Penguin",
                YMin:       782,
                XMin:       209,
                YMax:       717,
                XMax:       196,
            },
        },
}
Enter fullscreen mode Exit fullscreen mode

Then we have to call the function with our data and catch possible errors. got will be of type ResponseAdapter, since this is what ParseImageAnalysisResponseAdapter(…) returns.

got, err := ParseImageAnalysisResponseAdapter(data)
if err != nil {
        t.Fatal(err)
}
Enter fullscreen mode Exit fullscreen mode

And last we compare expectation and result.

if !cmp.Equal(want, got) {
    t.Error(cmp.Diff(want, got))
}
Enter fullscreen mode Exit fullscreen mode

If everything is correct we can use Run on Go Playground or go test in the command line and the test should pass or return an error.

=== RUN   TestParseImageAnalysisResponse
=== PAUSE TestParseImageAnalysisResponse
=== CONT  TestParseImageAnalysisResponse
{0.69146144 Lion 682 109 707 186}
--- PASS: TestParseImageAnalysisResponse (0.00s)
PASS
Enter fullscreen mode Exit fullscreen mode

Full listing

package main

import (
    "encoding/json"
    "fmt"
    "github.com/google/go-cmp/cmp"
    "testing"
)

type ResponseAdapter struct {
    Success     bool                 `json:"success"`
    Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
    Confidence float64 `json:"confidence"`
    Label      string  `json:"label"`
    YMin       int     `json:"y_min"`
    XMin       int     `json:"x_min"`
    YMax       int     `json:"y_max"`
    XMax       int     `json:"x_max"`
}

var data = []byte(`{"success": true,
    "predictions": [
        {
            "confidence": 0.69146144,
            "label": "Lion",
            "y_min": 682,
            "x_min": 109,
            "y_max": 707,
            "x_max": 186
        },
        {
            "confidence": 0.7,
            "label": "Penguin",
            "y_min": 782,
            "x_min": 209,
            "y_max": 717,
            "x_max": 196
        }]
    }`)

func ParseImageAnalysisResponseAdapter(data []byte) (ResponseAdapter, error) {

    var responseAdapter ResponseAdapter

    err := json.Unmarshal(data, &responseAdapter)

    if err != nil {
        fmt.Println(err)
    }

    fmt.Println(responseAdapter.Predictions[0])

    return responseAdapter, nil
}

func TestParseImageAnalysisResponseAdapter(t *testing.T) {
    t.Parallel()

    want := ResponseAdapter{
        Success: true,
        Predictions: []PredictionsAdapter{
            {
                Confidence: 0.69146144,
                Label:      "Lion",
                YMin:       682,
                XMin:       109,
                YMax:       707,
                XMax:       186,
            },
            {
                Confidence: 0.7,
                Label:      "Penguin",
                YMin:       782,
                XMin:       209,
                YMax:       717,
                XMax:       196,
            },
        },
    }

    got, err := ParseImageAnalysisResponseAdapter(data)
    if err != nil {
        t.Fatal(err)
    }

    if !cmp.Equal(want, got) {
        t.Error(cmp.Diff(want, got))
    }

}
Enter fullscreen mode Exit fullscreen mode

Now we have the data available in such a format that we can actually use it. We could even convert it to JSON and pass it to our user client. But sometimes the implementation logic requires that we add or remove some fields and provide the data in a different format. If that is what makes most sense, we should do it to maintain a well designed data workflow, even if it takes some additional work.

In the next section we will define the types Response and Predictions in a way we actually need them and transform them from our Adapter types using custom UnmarshallJSON. First we will add some custom metadata. Adding or changing fields in []PredictionsAdapter however requires looping through each element in the array.

Implementing custom UnmarshallJSON

Let’s create the Test for ParseImageAnalysisResponse that we are going to implement next (ignore any errors). Out want will be a Response that contains some metadata like name and Xid and the array of predictions. We use the data to pass it to ParseImageAnalysisResponse and than see if our expectation was met.

func TestParseImageAnalysisResponse(t *testing.T) {
    t.Parallel()

    want := Response{
        Name: "Test Image",
        Xid:  "9m4e2mr0ui3e8a215n4g",
        Predictions: []Prediction{
            {
                Confidence: 0.69146144,
                Label:          "Lion",
                X:              109,
                Y:              682,
                Width:          77,
                Height:         25,
                Method:         "AI",
            },
            {
                Confidence:     0.7,
                Label:          "Penguin",
                X:              209,
                Y:              782,
                Width:          -13,
                Height:         -65,
                Method:         "AI",
            },      
        },
    }

    got, err := ParseImageAnalysisResponse(data)
    if err != nil {
        t.Fatal(err)
    }

    if !cmp.Equal(want, got) {
        t.Error(cmp.Diff(want, got))
    }

}
Enter fullscreen mode Exit fullscreen mode

Next we define types that match the JSON API response we want to relay to our client, that could be a web app or a mobile app for Android or iOS.

type Prediction struct {
    Confidence  float64 `json:"confidence"`
    Label       string  `json:"label"`
    X           int     `json:"x"`
    Y           int     `json:"y"`
    Width       int     `json:"width"`
    Height      int     `json:"height"`
    Method      string  `json:"method"` // AI, Manual, Predicted(?)
}
Enter fullscreen mode Exit fullscreen mode

The Response will contain many of those predictions, since many objects can be detected in an image. But further more it will include meta data like a Name and an Xid. A Xid is a unique identifier that is a related but shorter version of UUID. We might need to change the type later, but for now let’s use string.

type Response struct {
    Name        string       `json:"name"`
    Xid         string       `json:"xid"`
    Predictions []Prediction `json:"predictions"`
}
Enter fullscreen mode Exit fullscreen mode

Now we will adapt the ParseImageAnalysisResponse function. It will still use the data of type []byte as an argument, but instead of a ResponseAdapter it will now return our new Response. Therefore we use a variable response of type Response and use this when unmarshalling. We then check for errors and return the response.

func ParseImageAnalysisResponse(data []byte) (Response, error) {

    var response Response
    // Use custom UnmarshalJSON method to obtain Response
    err := json.Unmarshal(data, &response)

    if err != nil {
        fmt.Println(err)
    }

    return response, nil

}
Enter fullscreen mode Exit fullscreen mode

Since there is no easy 1-to-1 mapping from the ResponseAdapter to the Response, we need to do some manual work. Instead of the default UnmarshalJSON which we don’t have to specify, we will now use a customized version of it that explicitly states how we can derive our Response to the client from the data in the ResponseAdapter.
The data we send to ParseImageAnalysisResponse is the one we want to unmarshall. We know that we can unmarshall it into a ResponseAdapter, so thats what we will do first, but not into the pointer of the response object r, but into a temporary variable tmp.

var tmp ResponseAdapter
err := json.Unmarshal(data, &tmp)

if err != nil {
    fmt.Println(err)
}
Enter fullscreen mode Exit fullscreen mode

This variable also has the Predictions in form of a PredictionsAdapter. We now cast them into a new Predictions-Array.

var p []Prediction
Enter fullscreen mode Exit fullscreen mode

We do this by looping though each element PredictionsAdapter

for _, element := range tmp.Predictions {
Enter fullscreen mode Exit fullscreen mode

and create a temporary corresponding predictions element.
This predictions element also defines the data for any added field or it could also leave out fields from PredictionsAdapter. Then we append the Predictions element to the Predictions Array p, that was defined before the loop. I did variable creation and appending kind of in one step here and closed the loop.

p = append(p, Prediction{
            Confidence: element.Confidence,
            Method:     "AI",
            Label:      element.Label,
            XMin:       element.XMin,
            XMax:       element.XMax,
            YMin:       element.YMin,
            YMax:       element.YMax,
    })
}
Enter fullscreen mode Exit fullscreen mode

Then we set the Response fields. That includes metadata like the name and Xid and setting the predictions array p.

r.Name = "Test Image"
r.Xid = "9m4e2mr0ui3e8a215n4g"
r.Predictions = p
Enter fullscreen mode Exit fullscreen mode

And last since UnmarshalJSON requires us to return an error, we do that as well.

return err
Enter fullscreen mode Exit fullscreen mode

Now lets see how this function looks in its full glory.

func (r *Response) UnmarshalJSON(data []byte) error {


    var tmp ResponseAdapter
    err := json.Unmarshal(data, &tmp)

    if err != nil {
        fmt.Println(err)
    }

    var p []Prediction
    for _, element := range tmp.Predictions {

        p = append(p, Prediction{
            Confidence: element.Confidence,
            Method:     "AI",
            Label:      element.Label,
            X:          element.XMin,
            Y:          element.YMin,
            Width:      element.Width(),
            Height:     element.Height(),
        })

    }

    r.Name = "Test Image"
    r.Xid = "9m4e2mr0ui3e8a215n4g"
    r.Predictions = p

    return err
}
Enter fullscreen mode Exit fullscreen mode

Conclusion

Great. We can now take an image that was annotated with an AI microservice and make this analysis useful for relaying it the user client by enhancing it with more relevant data for a JSON response. On the networking side how to exactly send JSON responses from a HTTP request to a client was explained . I talked about how to send POST requests to the AI micrososervice in a previous post.

Last, let’s look at the full listing, which you can find also on Go Playground. The tests could also be written into a separate file of course.

Thanks for reading, I hope you found this useful. This has been one of the first five posts on my new blog, so I appreciate any comments and suggestions.

package main

import (
    "encoding/json"
    "fmt"
    "github.com/google/go-cmp/cmp"
    "testing"
)

type ResponseAdapter struct {
    Success     bool                 `json:"success"`
    Predictions []PredictionsAdapter `json:"predictions"`
}

type PredictionsAdapter struct {
    Confidence float64 `json:"confidence"`
    Label      string  `json:"label"`
    YMin       int     `json:"y_min"`
    XMin       int     `json:"x_min"`
    YMax       int     `json:"y_max"`
    XMax       int     `json:"x_max"`
}


func (p PredictionsAdapter) Width() int {

    return p.XMax - p.XMin
}

func (p PredictionsAdapter) Height() int {

    return p.YMax - p.YMin
}


type Prediction struct {
    Confidence float64 `json:"confidence"`
    Label      string  `json:"label"`
    X          int     `json:"x"`
    Y          int     `json:"y"`
    Width      int     `json:"width"`
    Height     int     `json:"height"`
    Method     string  `json:"method"` // AI, Manual, Predicted(?)
}


type Response struct {
    Name        string       `json:"name"`
    Xid         string       `json:"xid"`
    Predictions []Prediction `json:"predictions"`
}


func (r *Response) UnmarshalJSON(data []byte) error {


    var tmp ResponseAdapter
    err := json.Unmarshal(data, &tmp)

    if err != nil {
        fmt.Println(err)
    }

    var p []Prediction
    for _, element := range tmp.Predictions {

        // fmt.Println("element", element, "at index", index)

        p = append(p, Prediction{
            Confidence: element.Confidence,
            Method:     "AI",
            Label:      element.Label,
            X:          element.XMin,
            Y:          element.YMin,
            Width:      element.Width(),
            Height:     element.Height(),
        })

    }

    r.Name = "Test Image"
    r.Xid = "9m4e2mr0ui3e8a215n4g"
    r.Predictions = p

    return err
}

var data = []byte(`{"success": true,
    "predictions": [
        {
            "confidence": 0.69146144,
            "label": "Lion",
            "y_min": 682,
            "x_min": 109,
            "y_max": 707,
            "x_max": 186
        },
        {
            "confidence": 0.7,
            "label": "Penguin",
            "y_min": 782,
            "x_min": 209,
            "y_max": 717,
            "x_max": 196
        }]
    }`)

func ParseImageAnalysisResponseAdapter(data []byte) (ResponseAdapter, error) {

    var responseAdapter ResponseAdapter

    err := json.Unmarshal(data, &responseAdapter)

    if err != nil {
        fmt.Println(err)
    }

    fmt.Println(responseAdapter.Predictions[0])

    return responseAdapter, nil
}

func ParseImageAnalysisResponse(data []byte) (Response, error) {

    var response Response
    // Use custom UnmarshalJSON method to obtain Response
    err := json.Unmarshal(data, &response)

    if err != nil {
        fmt.Println(err)
    }
    // fmt.Println("Response")
    // fmt.Println(response)

    return response, nil

}


func TestParseImageAnalysisResponseAdapter(t *testing.T) {
    t.Parallel()

    want := ResponseAdapter{
        Success: true,
        Predictions: []PredictionsAdapter{
            {
                Confidence: 0.69146144,
                Label:      "Lion",
                YMin:       682,
                XMin:       109,
                YMax:       707,
                XMax:       186,
            },
            {
                Confidence: 0.7,
                Label:      "Penguin",
                YMin:       782,
                XMin:       209,
                YMax:       717,
                XMax:       196,
            },
        },
    }

    got, err := ParseImageAnalysisResponseAdapter(data)
    if err != nil {
        t.Fatal(err)
    }

    if !cmp.Equal(want, got) {
        t.Error(cmp.Diff(want, got))
    }

}

func TestParseImageAnalysisResponse(t *testing.T) {
    t.Parallel()

    want := Response{
        Name: "Test Image",
        Xid:  "9m4e2mr0ui3e8a215n4g",
        Predictions: []Prediction{
            {
                Confidence: 0.69146144,
                Label:          "Lion",
                X:              109,
                Y:              682,
                Width:          77,
                Height:         25,
                Method:         "AI",
            },
            {
                Confidence:  0.7,
                Label:          "Penguin",
                X:              209,
                Y:              782,
                Width:          -13,
                Height:         -65,
                Method:         "AI",
            },
        },
    }

    got, err := ParseImageAnalysisResponse(data)
    if err != nil {
        t.Fatal(err)
    }

    if !cmp.Equal(want, got) {
        t.Error(cmp.Diff(want, got))
    }

}
Enter fullscreen mode Exit fullscreen mode

Links

https://bitfieldconsulting.com/golang/map-string-interface
https://jhall.io/posts/go-json-tricks-array-as-structs/
https://stackoverflow.com/questions/42377989/unmarshal-json-array-of-arrays-in-go

Top comments (0)

Read next

lucasdeataides profile image

Why Clean Architecture Struggles in Golang and What Works Better

Lucas de Ataides -

dumebii profile image

API Documentation Tools: Using Hoppscotch for API Documentation

Dumebi Okolo -

crusty0gphr profile image

Understanding sync.Cond in Go: Synchronizing Goroutines in Producer-Consumer Scenarios

Harutyun Mardirossian -

refifauzan profile image

Make your response consistent in NestJS using Interceptors

Refi Fauzan -