How to Structure a Go Project: Start Simple, Refactor Later

Once upon a codebase, in a kingdom of endless debates, a lone developer pondered the age-old question: “What's the perfect project structure?” Spoiler alert: there isn't one. Let's embark on a quest to discover the charm of simplicity and the art of Go project structuring.

The perennial question about what directory structure should I use echoes across various social platforms now and then. This subject was discussed many times. I’ve been programming in Go for a couple of years now and asked myself this question every time I had to start a new project. If you’re asking it yourself, here are my 2 cents for you.

Why is project structure important?

"A well-organized directory structure is the scaffolding upon which a robust codebase stands tall."

A project structure is important because it affects readability & maintainability. Think about your future self, six months from now reading your code when it’s not fresh in your memory. If you have structured it right (and there are many ways to do it) it would be easier to jump in. It would also be easier for collaboration as other developers can navigate through it more easily. Most of the time we read and maintain existing code, not writing it from scratch.

What are the common pitfalls of project structure?

There is no such thing as a perfect, one-size-fits-all project structure, so stop looking. It’s like asking for the perfect car. It depends on who you ask, and what are their standards and requirements. You won’t find a single answer, because like many things in computer science — it depends. While there’s no one-size-fits-all solution, some general guidelines can be followed.

Some of the common pitfalls:

• Over-engineering: Trying to anticipate all the possible scenarios and use-cases before writing any code. This leads to unnecessary complexity and abstractions, which makes your code harder to read and maintain
• Under-engineering: Writing all the code in a single package, without any structure or organization. This leads to spaghetti code base, which makes it harder to test and debug
• Copy-pasta: Blindly following the structure of another project, without putting any thoughts or understanding the trade-offs behind it.

How to start simple and refactor later?

My advice is to start simple and let the packages grow organically as you’re writing the code. Don’t worry about creating the perfect packages and abstractions at this point. Just start by writing your code in a single package — main and see how it works. A simple directory tree is easier to read, which reflects that it is easier to maintain and collaborate with a team or individuals.

As you write more code, you’ll notice some patterns emerge and repetitions that you can extract to a different package. This will make your code more readable, maintainable, and testable.

I find it much harder to do the right abstraction before writing code. How do you know which packages do you need? Start writing your business logic. It is easier to refactor code when you see it in front of you. It’s usually easier to abstract some logic from a package, rather than refactoring multiple packages because you got it wrong the first time.

The key is to write code and refactor it later.

The cmd package pattern

A powerful pattern but make sure to ask yourself if you need it

This is a great pattern when necessary. Do you write a library? Or an app with a single binary? Then you probably don’t need to use the cmd package. Ask yourself what this abstraction allows you. Remember — keep it simple.
This pattern is useful when your code base is compiled to multiple binaries, for example, if you have a Server and a CLI.

Inside this package, you put the sub-directories that contain your main.go files. Each sub-directory is compiled into its own binary. This is the entry point of your app. You should look into spf13/cobra: A Commander for modern Go CLI interactions project, which is a widely used library for creating powerful modern CLI applications. It blends well with spf13/viper: Go configuration with fangs project, which handles loading configurations nicely.

The internal package

The internal package has a special meaning in Go. Packages that reside under internal/ may not be imported by packages outside the source subtree in which they reside. Therefore, these are said to be internal packages. The code placed here is internal to the project, and can’t be used outside of it.

There used to be times people grouped external packages under pkg/, but I find it meaningless. A directory that exists only to hold other packages is a potential code smell.
Instead, give your packages descriptive names. Try to describe what they do, while avoiding ambiguous names like utils or common.

The config package

The config package is responsible for creating and managing the configuration object that your app depends on. It simplifies the process of importing configurations from a single source and accessing them from anywhere in your code base. It also avoids the circular dependency problem that can arise from importing configurations from multiple places.

Let the config package handle your app's settings, harmonizing inputs from flags, environment variables, or files into a cohesive configuration object.

The API package

The API package defines the interface of your app with the outside world. It contains the schema definition (openapi) and the models (which can be generated from schema) that represent the data types and endpoints of your app. Unless a struct is internal to a specific package and is not used outside of it, it should be placed here. The API package should include any struct that is used by more than one package, or that is exposed to the client or the storage.

The API package is the protocol that enables communication between different components of your app.

Controllers, handlers, and storage

Now, let's delve into structuring controllers, handlers, and storage. Two approaches stand out:

Approach 1: Storage and Handlers (or controllers) packages

With the first approach, you write your storage and service/controller layers in separate packages where the service imports the storage. This allows you to decouple your business logic from your data access layer and use different storage implementations (postgres, sqlite, redis, etc.) without changing your service code. Here’s a simple view of this approach:

.
└── project/
    ├── api/
    │   ├── models.go
    │   └── openapi3.yaml
    ├── cmd/
    │   ├── server/
    │   │   └── main.go
    │   └── cli/
    │       └── main.go
    └── internal/
        ├── config/
        │   └── config.go
        ├── storage/
        │   └── postgres/
        │       ├── users.go
        │       └── ...
        ├── handlers/
        │   ├── users.go
        │   ├── users_test.go
        │   ├── storage.go
        │   └── ..
        └── server/
            ├── server.go
            ├── server_test.go
            └── ..

Approach 2: Domain Entity Package

With this approach, you encapsulate logic by functionality so that both your service and storage code reside in the same package. This follows the principle of domain-driven design, where you model your code around the business domain and its entities. Each package represents a domain entity (such as user, post, comment, etc.) and contains all the code related to it (such as internal structs, methods, handlers, queries, etc.)

This is a topic for a separate blog post I might do later on, but in the meantime for more details about this type of architecture check out introduction to DDD or watch Kat Zien great talk from GopherCon 2018.. Here’s a simple view of this approach:

.
└── project/
    ├── api/
    │   ├── models.go
    │   └── openapi3.yaml
    ├── cmd/
    │   ├── server/
    │   │   └── main.go
    │   └── cli/
    │       └── main.go
    └── internal/
        ├── config/
        │   └── config.go
        ├── users/
        │   ├── postgres/
        │   │   └── db.go
        │   ├── handlers.go
        │   ├── handlers_test.go
        │   ├── storage.go
        │   └── ..
        └── server/
            ├── server.go
            ├── server_test.go
            └── ..

Conclusion

In conclusion, there is no single ‘perfect’ way to structure your Go project. There isn’t a one-size-fits-all. There are multiple ways, and it depends on your use-case, preferences, and the trade-offs you’re willing to take. Experiment based on your needs and requirements.

Embrace simplicity initially. Write code in a single package, observe emerging patterns, and refactor as needed. The key here is to write code first, and ‘perfect’ structure later.

I hope this post gave you some insights and ideas on how to structure your next project. Happy coding!