After spending a lot of the week finalizing the team service with an in-memory data provider and feeling like I was reasonably feature-complete I made the decision to get down in the details.

The details in this case, being AWS DynamoDB.

A quick side note, if anybody is following along with the repo just switch over the below two lines in main.go to bring back an in-memory database.

// teamInteractor.TeamRepository = infrastructure.NewInMemTeamRepo()
teamInteractor.TeamRepository = infrastructure.NewDynamoDbRepo()

DynamoDB

For anybody who hasn't heard of Dynamo Db, here it is as described by Amazon themselves.

Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. It's a fully managed, multi-region, multimaster, durable database with built-in security, backup and restores, and in-memory caching for internet-scale applications. DynamoDB can handle more than 10 trillion requests per day and can support peaks of more than 20 million requests per second.

So it does a lot. The Lehmans translation of the above

Amazon DynamoDB is a NoSQL database in the cloud

Because of the commitment to Clean Architecture from the start of this project, substituting a new database provider becomes extremely easy.

We already have an interface defined for the required methods on the repository, so as long as the Dynamo Db repository implements the same methods the two are completely interchangeable.

Let's remind ourselves of that interface

// TeamRepository handles the persistence of teams.
type TeamRepository interface {
    FindByID(id string) *Team
    Store(team *Team) string
    Update(team *Team) *Team
    Search(searchTerm string) []Team
}

Nice and easy, let's get on with this implementation.

Implementing Dynamo Db in GoLang

As with a lot of the AWS services, they have a set of pre-built SDK's for most languages. Go is no exception. Docs here.

When I have worked with Dynamo Db in the past, I've always created the table within the AWS UI and then posted data in from there. In this case, I wanted to keep definitions closer to the codebase.

The reason being two-fold:

1. It allows the ease of local development for me and others using the repo (DynamoDB can be launched with Docker, more on that shortly)
2. If there was ever a need to rebuild the AWS infrastructure the service would quickly handle the creation of it's tables

// NewDynamoDbRepo creates a new DynamoDb Repository
func NewDynamoDbRepo() *DynamoDbRepository {
    // Initialize a session that the SDK will use to load
    // credentials from the shared credentials file ~/.aws/credentials
    // and region from the shared configuration file ~/.aws/config.
    sess := session.Must(session.NewSessionWithOptions(session.Options{
        SharedConfigState: session.SharedConfigEnable,
    }))

    // Create DynamoDB client
    svc := dynamodb.New(sess, &aws.Config{Endpoint: aws.String("http://localhost:8000")})

    tableInput := &dynamodb.CreateTableInput{
        AttributeDefinitions: []*dynamodb.AttributeDefinition{
            {
                AttributeName: aws.String("id"),
                AttributeType: aws.String("S"),
            },
        },
        KeySchema: []*dynamodb.KeySchemaElement{
            {
                AttributeName: aws.String("id"),
                KeyType:       aws.String("HASH"),
            },
        },
        ProvisionedThroughput: &dynamodb.ProvisionedThroughput{
            ReadCapacityUnits:  aws.Int64(10),
            WriteCapacityUnits: aws.Int64(10),
        },
        TableName: aws.String("Teams"),
    }

    _, tableErr := svc.CreateTable(tableInput)

    if tableErr != nil {
        fmt.Println("Got error calling CreateTable:")
        fmt.Println(tableErr.Error())
    }

    return &DynamoDbRepository{
        dynamoSvc: svc,
    }
}

The above code is initializing an AWS DynamoDb service using a hardcoded endpoint and then runs a CreateTable command to ensure the Teams table exists.

The usage of localhost:8000 has some relevance. There is a fantastic Docker image called dwmkerr/dynamodb which runs a local instance of DynamoDb. It isn't completely feature-rich, but it covers most of the key bits of functionality.

At this point, I'll start up the Docker container ready for the first test of the Go table creation code.

docker run -p 8000:8000 dwmkerr/dynamodb

Starting the app now gives the standard log messages and no errors, which is a fantastic sign.

Note: Stopping and starting the app will write a Cannot create preexisting table error. That's fine for the moment.

Now that we have a working Dynamo service let's try and actually persist some data.

DynamoDb Store Item in GoLang

When defining a table in Go, it's possible to define each and every property of the schema. In this instance, I've chosen to not define the schema. This just eases the maintenance of needing to manage a schema definition going forward.

Here's the complete working code to store an item.

// Store creates a new team in the in memory storage.
func (r *DynamoDbRepository) Store(team *domain.Team) string {
    team.ID = xid.New().String()

    av, err := dynamodbattribute.MarshalMap(team)

    input := &dynamodb.PutItemInput{
        Item:      av,
        TableName: aws.String("Teams"),
    }

    _, err = r.dynamoSvc.PutItem(input)

    if err != nil {
        fmt.Println("Got error calling CreateTable:")
        fmt.Println(err.Error())
        os.Exit(1)
    }

    return team.ID
}

Keeping the same line of code from the in-memory service, I first assign a new unique identifier to the team record.

The MarshalMap object transforms a GoLang struct into a Dynamo Db array of attribute maps. Attribute maps are the way the table schema handled. It stores the data value and it's requisite data types in one object.

The attribute maps can be created manually, but the MarshalMap is a nice little helper method to abstract all this complication away.

Following that, we simply define the PutItemInput parameters (Item to be inserted and the TableName to insert into).

Running the PutItem method from the dynamo service should hopefully return no errors and we are cooking.

Creation ... Check!

Reading data from Dynamo Db

Persisting data in a database is a fantastic feature, but it's kind of redundant without being able to retrieve it. So that is where we go next.

func (r *DynamoDbRepository) FindByID(teamID string) *domain.Team {
    filt := expression.Name("id").Equal(expression.Value(teamID))

    expr, _ := expression.NewBuilder().WithFilter(filt).Build()

    // Build the query input parameters
    params := &dynamodb.ScanInput{
        ExpressionAttributeNames:  expr.Names(),
        ExpressionAttributeValues: expr.Values(),
        FilterExpression:          expr.Filter(),
        TableName: aws.String("Teams"),
    }

    // Make the DynamoDB Query API call
    result, _ := r.dynamoSvc.Scan(params)

    if len(result.Items) > 0 {
        item := &domain.Team{}

        dynamodbattribute.UnmarshalMap(result.Items[0], &item)

        for _, res := range result.Items {
            for fieldName, playerRes := range res {
                if fieldName == "players" && playerRes.S != nil {
                    s := *playerRes.S

                    json.Unmarshal([]byte(s), &item.Players)
                }
            }
        }

        return item
    }

    return nil
}

Querying from dynamo db using the SDK is just as simple as storing an item. We first set up a new filter expression, using the id that we are looking for. Note: It's important to note the usage of the property name "id" instead of "ID". When using MarshalMap any JSON formatting from the struct is applied.

From there, we build a new set of scan parameters and run a table scan against the Teams table in the database.

Once data has been returned, the items are looped and the first item is taken (a query for a specific ID should only ever return a single record).

The player data is stored as a JSON string rather than an object model, so on the return of the data that JSON data needs to be Unmarshalled into the Player array.

This persistence layer is starting to be useful.

TDD with the persistence layer

I love test-driven development. This whole project is about building up my knowledge of both TDD and micro-service design in GoLang.

That said, a persistence layer is NOTORIOUSLY difficult to unit test. A unit test should have no external dependencies... localhost:8000 is definitely an external dependency.

One of my next tasks with the teams-service repository is to set up some integration tests in preparation for the CI/CD pipeline.

For the moment though, I feel this application covers all the bases for initial deployment. AWS here we come.

Event Buses

Any eagle-eyed followers of the commits in the git repo will notice the addition of an event bus file.

There are also a set of calls added to the usecases for publishing events.

Sticking with the clean architecture principles of staying away from details, all event bus interactions are handled through an interface.

The use cases know there is a thing called an EventHandler, and that objects can be published. Aside from that though, no other detail is known.

For the moment, the event handler is very dumb. Write Line = job done.

// Publish sends a new message to the event bus.
func (ev MockEventBus) Publish(evt domain.Event) error {
    println(string(evt.AsEvent()))

    return nil
}

GoLang thoughts so far

I didn't get as much done on the project as I would have liked this week, but progress is still being made.

But GoLang is an incredible language, and I can see why there is such a buzz around it.

I'm still trying to wrap my head around the type system and the way inheritance works. But that aside it's a great language I can see me sticking with for a long time after the end of this project.

See you next week :)