I’ve recently done some work with Go and DynamoDB and needed to test my work. Luckily Amazon provides DynamoDB local, which means that I don’t need to provision any real infrastructure in AWS and can run my tests offline. This post will walk through a simple example of interacting with DynamoDB with Go and how to test this code with Dynamo Local. All of the code for this example can be found in GitHub.
I use the guregu/dynamo library to interact with Dynamo as I find it provides a nice abstraction to the DynamoDB API.
To start with I create a struct representing the Items I wish to store in Dynamo:
// item/item.go
type Item struct {
Id string `dynamo:"item_id,hash"`
Name string `dynamo:"name"`
Description string `dynamo:"description"`
CreatedAt time.Time `dynamo:"created_at"`
UpdatedAt time.Time `dynamo:"updated_at"`
}
Notice the tags on the struct fields, these provide some addition information to the guregu/dynamo library on how to marshal and unmarshal data to and from Dynamo. For instance, the ‘Id’ field will be shown in DynamoDb as ‘item_id’ and it will be a hash (primary key). The hash is only required here when we want to create Dynamo Tables using this type, as we will later.
Next, I’ll create an ItemService which will hold the DynamoDB client and for which we can write methods to interact with Dynamo, such as adding items to the database.
// item/item.go
type ItemService struct {
itemTable dynamo.Table
}
Next, come a couple of methods, one that creates a new dynamo.Table, which is a client for communicating with Dynamo and one that creates a new ItemService
// item/item.go
func newDynamoTable(tableName, endpoint string) (dynamo.Table, error) {
if tableName == "" {
return dynamo.Table{}, fmt.Errorf("you must supply a table name")
}
cfg := aws.Config{}
cfg.Region = aws.String("eu-west-2")
if endpoint != "" {
cfg.Endpoint = aws.String(endpoint)
}
sess := session.Must(session.NewSession())
db := dynamo.New(sess, &cfg)
table := db.Table(tableName)
return table, nil
}
There are a few things going on here. First we’re checking that we have been provided with a tableName (the name of the table we’re connecting to). We make this function private, so that we’re in control of when it’s used — we only need to create a dynamo client when we’re setting up our ItemService, or testing.
Next we’re setting up an AWS session with some configuration. In a real-world use case we’d also pass in the AWS region, rather than hard-coding it.
If we’ve been provided with an endpoint we’ll point the client to that, rather than allowing the client to use it’s default endpoints. In normal usage, we won’t supply an endpoint, but for testing it allows us to point to Dynamo Local.
// item/item.go
func NewItemService(itemTableName string) (*ItemService, error) {
dynamoTable, err := newDynamoTable(itemTableName, "")
if err != nil {
return nil, err
}
return &ItemService{
itemTable: dynamoTable,
}, nil
}
This function takes the name of our item table, sets up a client using the ‘newDynamoTable’ function we discussed above and finally returns a new ItemService which holds the dynamo client. Notice that we always send an empty string as the endpoint, this means we can’t accidentally send an invalid endpoint when we’re in production, but it does mean we can’t use this function in our tests.
Now we need some methods to interact with DynamoDB to read and write Items. These methods will be associated with the ItemService so that they have access to the Dynamo client.
// item/item.go
func (i *ItemService) CreateItem(item *Item) error {
now := time.Now()
item.CreatedAt = now
item.UpdatedAt = now
item.Id = xid.New().String()
return i.itemTable.Put(item).Run()
}
Here we add a new Item to the database. We set the created and updated times to the current time, generate an Id for the item and the use the Put method to write to the Dynamo Table.
// item/item.go
func (i ItemService) GetItem(item Item) error {
return i.itemTable.Get("item_id", item.Id).One(item)
}
Here’s an example of reading from DynamoDB.
Note that these are basic examples and don’t have any verification on the Items we’re operating on or any detailed error handling. For instance we would probably want to check that the Item we’re are creating has a Name and Description before we try and actually create it. There may also be errors we can handle. For instance, if we were throttled by Dynamo, we could retry with an incremental backoff (wait for a longer period of time between each retry).
Next, it’s time to test the code. In each test I create a new, randomly named, Dynamo table that each test can run against, meaning that two tests won’t clash if they are run in parallel.
I created a test_utils package that creates a new table, using an interface as a schema.
// test_utils/dynamo-local.go
func CreateTable(table interface {}) (string, error) {
cfg := aws.Config{
Endpoint: aws.String("http://localhost:9000"),
Region: aws.String("eu-west-2"),
}
sess := session.Must(session.NewSession())
db := dynamo.New(sess, &cfg)
tableName := xid.New().String()
err := db.CreateTable(tableName, table).Run()
if err != nil {
return"", err
}
return tableName, nil
}
The ‘table interface{}’ that the function accepts in our case will be the ‘Item’ type we created at the start of this post. By taking an interface we can reuse this function to create any table. Of course, this function will fail if you pass an interface that cannot be made to represent a Dynamo Table . For instance, a struct that does not have a tag defining the hash key — (dynamo:”hash”).
type OtherItem struct {
Id string `dynamo:"item_id,hash"`
}
The above would be sufficient to create a table that we can use to test the Item type. The other fields will be added to the table when we create Item’s in the database. As long as the keys match there isn’t a problem.
Now in each test of ItemService methods, we can use this function to set up a table for us. To save some repetition I’ve created a ‘newItemService’ function in item_test.go that will call ‘CreateTable’ and set up an ItemService configured to use that table and to use the Dynamo Local endpoint (we’ll set up Dynamo Local later).
// item/item_test.go
func newItemService() (*ItemService, error) {
tableName, err := test\_utils.CreateTable(Item{})
if err != nil {
return nil, fmt.Errorf("failed to set up table. %s", err)
}
db, err := newDynamoTable(tableName, "http://localhost:9000")
if err != nil {
return nil, err
}
service := ItemService{
itemTable: db,
}
return &service, nil
}
Tests for CreateItem and GetItem follow the same pattern:
- Setup a slice of test conditions (table tests)
- Setup the ItemService using ‘newItemService’
- Run through each test condition and evaluate if it passes or fails.
// item/item\_test.go
func TestItemService\_CreateItem(t *testing.T) {
cases := [] struct {
name string
item Item
err bool // Whether we expect an error back or not
}{
{
name: "created successfully",
item: &Item{
Name: "spoon",
Description: "shiny",
},
},
}
service, err := newItemService()
if err != nil {
t.Fatal(err)
}
for _, c := range cases {
t.Run(c.name, func (t testing.T) {
err := service.CreateItem(c.item)
if c.err {
assert.Error(t, err)
} else {
assert.NoError(t, err)
assert.NotEqual(t, time.Time{}, c.item.CreatedAt)
assert.NotEqual(t, time.Time{}, c.item.UpdatedAt)
}
})
}
}
We create a slice of an anonymous struct, to which we add a name (or short description of the test), the item we want to test and whether we expect it to succeed. You could also add the error you expect to receive, if you want to test specific error conditions.
We then create out ItemService and fail the test (t.fatal) if any of this process fails.
Next, we iterate through each test case, with each case being a subtest (t.Run). We use the name we defined for each test case as the name parameter to t.Run, which helps us identify which test failed and what we were trying to test with that test case.
For each case, we use assert to check for errors. If we expected to get an error we assert that we did receive one and visa versa if we don’t expect an error. If we didn’t expect to receive an error we also check that the item has the time set correctly (remember we set this time in the CreateItem method).
Now we have tests that will set up tables in Dynamo Local and test our code using those tables, but we don’t have an instance of Dynamo Local to run against… We’ll use Docker to set this up.
First we’ll set up a ‘Dockerfile’
FROM openjdk:7
RUN mkdir -p opt/dynamodb
WORKDIR /opt/dynamodb
RUN wget https://s3.eu-central-1.amazonaws.com/dynamodb-local-frankfurt/dynamodb\_local\_latest.tar.gz -q -O - | tar -xz
EXPOSE 8000
ENTRYPOINT ["java", "-jar", "DynamoDBLocal.jar"]
This builds on top of the openjdk container, setups up a directory for dynamo local, downloads and extracts the jar, open a port for it to listen on and then sets the entrypoint to ensure that Dynamo Local starts when the container does.
Next, we’ll set up a docker-compose.yml which will allow us to use docker-compose to build, start and stop our container for us.
dynamo:
build: .
ports:
- 9000:8000
command:
-sharedDb
This builds from The Dockerfile we created above in the same directory, maps our local port 9000 to the containers port 8000 (You may have noticed that we used localhost:9000 as the endpoint for DynamoDB in our code).
Finally, it sends the -sharedDb flag to Dynamo Local when we start the container. If we don’t use this flag then each request will use a different table, which means we can’t do things like reading the Item we just created to check if the create worked.
Next up is to create a Makefile which will run docker-compose to setup dynamo local, run the tests and the use docker-compose to stop dynamo local.
TEST?=$$(go list ./... |grep -v 'vendor')
GOFMT\_FILES?=$$(find . -name '\*.go' |grep -v vendor)
default: test
fmt:
gofmt -w $(GOFMT\_FILES)
test: fmt
docker-compose down
docker-compose up -d --build --force-recreate
go test -i $(TEST) || exit 1
echo $(TEST) | \
xargs -t -n4 go test -v
docker-compose down
Now I can just run make and my tests will run, hopefully successfully.
Top comments (0)