Go best practices, six years in

(This article was originally a talk at QCon London 2016. Video and slides here.)

In 2014, I gave a talk at the inaugural GopherCon titled Best Practices in Production Environments. We were early adopters at SoundCloud, and by that point had been writing, running, and maintaining Go in production in one form or another for nearly 2 years. We had learned a few things, and I tried to distill and convey some of those lessons.

Since then, I’ve continued working in Go full-time, later on the activities and infrastructure teams at SoundCloud, and now at Weaveworks, on Weave Scope and Weave Mesh. I’ve also been working hard on Go kit, an open-source toolkit for microservices. And all the while, I’ve been active in the Go community, meeting lots of developers at meetups and conferences throughout Europe and the US, and collecting their stories—both successes and failures.

With the 6th anniversary of Go’s release in November of 2015, I thought back to that first talk. Which of those best practices have stood the test of time? Which have become outmoded or counterproductive? Are there any new practices that have emerged? In March, I had the opportunity to give a talk at QCon London where I reviewed the best practices from 2014 and took a look at how Go has evolved in 2016. Here’s the meat of that talk.

I’ve highlighted the key takeaways as linkable Top Tips.

  Top Tip — Use Top Tips to level up your Go game.

And a quick table of contents…

  1. Development environment
  2. Repository structure
  3. Formatting and style
  4. Configuration
  5. Program design
  6. Logging and instrumentation
  7. Testing
  8. Dependency management
  9. Build and deploy
  10. Conclusion


Go has development environment conventions centered around the GOPATH. In 2014 I advocated strongly for a single global GOPATH. My positioned has softened a bit. I still think that’s the best idea, all else equal, but depending on your project or team, other things may make sense, too.

If you or your organization produces primarily binaries, you might find some advantages with a per-project GOPATH. There’s a new tool, gb, from Dave Cheney and contributors, which replaces the standard go tooling for this use-case. A lot of people are reporting a lot of success with it.

Some Go developers use a two-entry GOPATH, e.g. $HOME/go/external:$HOME/go/internal. The go tool has always known how to deal with this: go get will fetch into the first path, so it can be useful if you need strict separation of third-party vs. internal code.

One thing I’ve noticed some developers forget to do: put GOPATH/bin into your PATH. This allows you to easily run binaries you get via go get, and makes the (preferred) go install mechanism of building code easier to work with. No reason not to do it.

  Top Tip — Put $GOPATH/bin in your $PATH, so installed binaries are easily accessible.

Regarding editors and IDEs, there’s been a lot of steady improvement. If you’re a vim warrior, life has never been better: thanks to the tireless and extremely capable efforts of Fatih Arslan, the vim-go plugin is in an absolutely exceptional state, best-in-class. I’m not as familiar with emacs, but Dominik Honnef’s go-mode.el is still the big kahuna there.

Moving up the stack, lots of folks are still using and having success with Sublime Text + GoSublime. And it’s hard to beat the speed. But more attention seems to be paid lately to the Electron-powered editors. Atom + go-plus has many fans, especially those developers that have to frequently switch languages to JavaScript. The dark horse has been Visual Studio Code + vscode-go, which, while slower than Sublime Text, is noticably faster than Atom, and has excellent default support for important-to-me features, like click-to-definition. I’ve been using it daily for about half a year now, after being introduced to it by Thomas Adam. Lots of fun.

In terms of full IDEs, the purpose-built LiteIDE has been receiving regular updates and certainly has its share of fans. And the IntelliJ Go plugin has been consistently improving as well.


Update: Ben Johnson has written an excellent article titled Standard Package Layout with great advice for typical line-of-business applications.

Update: Tim Hockin’s go-build-template, adapted slightly, has proven to be a better general model. I’ve adapted this section since its original publication.

We’ve had a lot of time for projects to mature, and some patterns have emerged. While I believe there is no single best repo structure, I think there is a good general model for many types of projects. It’s especially useful for projects that provide both binaries and libraries, or combine Go code with other, non-Go assets.

The basic idea is to have two top-level directories, pkg and cmd. Underneath pkg, create directories for each of your libraries. Underneath cmd, create directories for each of your binaries. All of your Go code should live exclusively in one of these locations.

github.com/peterbourgon/foo/
  circle.yml
  Dockerfile
  cmd/
    foosrv/
      main.go
    foocli/
      main.go
  pkg/
    fs/
      fs.go
      fs_test.go
      mock.go
      mock_test.go
    merge/
      merge.go
      merge_test.go
    api/
      api.go
      api_test.go

All of your artifacts remain go gettable. The paths may be slightly longer, but the nomenclature is familiar to other Go developers. And you have space and isolation for non-Go assets. For example, Javascript can live in a client or ui subdirectory. Dockerfiles, continuous integration configs, or other build helpers can live in the project root or in a build subdirectory. And runtime configuration like Kubernetes manifests can have a home, too.

  Top Tip — Put library code under a pkg/ subdirectory. Put binaries under a cmd/ subdirectory.

Of course, you’ll still use fully-qualified import paths. That is, the main.go in cmd/foosrv should import "github.com/peterbourgon/foo/pkg/fs". And beware of the ramifications of including a vendor dir for downstream users.

  Top Tip — Always use fully-qualified import paths. Never use relative imports.

This little bit of structure makes us play nice in the broader ecosystem, and hopefully continues to ensure our code is easy to consume.


Things have stayed largely the same here. This is one area that Go has gotten quite right, and I really appreciate the consensus in the community and stability in the language.

The Code Review Comments are great, and should be the minimum set of critera you enforce during code review. And when there are disputes or inconsistencies in names, Andrew Gerrand’s idiomatic naming conventions are a great set of guidelines.

  Top Tip — Defer to Andrew Gerrand’s naming conventions.

And in terms of tooling, things have only gotten better. You should configure your editor to invoke gofmt—or, better, goimports—on save. (At this point, I hope that’s not in any way controversial.) The go vet tool produces (almost!) no false positives, so you might consider making it part of your precommit hook. And check out the excellent gometalinter for linting concerns. This can produce false positives, so it’s not a bad idea to encode your own conventions somehow.


Configuration is the surface area between the runtime environment and the process. It should be explicit and well-documented. I still use and recommend package flag, but I admit at this point I wish it were less esoteric. I wish it had standard, getopts-style long- and short-form argument syntax, and I wish its usage text were much more compact.

12-factor apps encourage you to use environment vars for configuration, and I think that’s fine, provided each var is also defined as a flag. Explicitness is important: changing the runtime behavior of an application should happen in ways that are discoverable and documented.

I said it in 2014 but I think it’s important enough to say again: define and parse your flags in func main. Only func main has the right to decide the flags that will be available to the user. If your library code wants to parameterize its behavior, those parameters should be part of type constructors. Moving configuration to package globals has the illusion of convenience, but it’s a false economy: doing so breaks code modularity, makes it more difficult for developers or future maintainers to understand dependency relationships, and makes writing independent, parallelizable tests much more difficult.

  Top Tip — Only func main has the right to decide which flags are available to the user.

I think there’s a great opportunity for a well-scoped flags package to emerge from the community, combining all of these characteristics. Maybe it already exists; if so, please let me know. I’d certainly use it.


In the talk, I used configuration as a jumping-off point, to discuss a few other issues of program design. (I didn’t cover this in the 2014 version.) To start, let’s take a look at constructors. If we are properly parameterizing all of our dependencies, our constructors can get quite large.

foo, err := newFoo(
    *fooKey,
    bar,
    100 * time.Millisecond,
    nil,
)
if err != nil {
    log.Fatal(err)
}
defer foo.close()

Sometimes this kind of construction is best expressed with a config object: a struct parameter to a constructor that takes optional parameters to the constructed object. Let’s assume fooKey is a required parameter, and everything else either has a sensible default or is optional. Often, I see projects construct config objects in a sort of piecemeal way.

// Don't do this.
cfg := fooConfig{}
cfg.Bar = bar
cfg.Period = 100 * time.Millisecond
cfg.Output = nil

foo, err := newFoo(*fooKey, cfg)
if err != nil {
    log.Fatal(err)
}
defer foo.close()

But it’s considerably nicer to leverage so-called struct initialization syntax to construct the object all at once, in a single statement.

// This is better.
cfg := fooConfig{
    Bar:    bar,
    Period: 100 * time.Millisecond,
    Output: nil,
}

foo, err := newFoo(*fooKey, cfg)
if err != nil {
    log.Fatal(err)
}
defer foo.close()

No statements go by where the object is in an intermediate, invalid state. And all of the fields are nicely delimited and indented, mirroring the fooConfig definition.

Notice we construct and then immediately use the cfg object. In this case we can save another degree of intermediate state, and another line of code, by inlining the struct declaration into the newFoo constructor directly.

// This is even better.
foo, err := newFoo(*fooKey, fooConfig{
    Bar:    bar,
    Period: 100 * time.Millisecond,
    Output: nil,
})
if err != nil {
    log.Fatal(err)
}
defer foo.close()

Nice.

  Top Tip — Use struct literal initialization to avoid invalid intermediate state. Inline struct declarations where possible.

Let’s turn to the subject of sensible defaults. Observe that the Output parameter is something that can take a nil value. For the sake of argument, assume it’s an io.Writer. If we don’t do anything special, when we want to use it in our foo object, we’ll have to first perform a nil check.

func (f *foo) process() {
    if f.Output != nil {
        fmt.Fprintf(f.Output, "start\n")
    }
    // ...
}

That’s not great. It’s much safer, and nicer, to be able to use output without having to check it for existence.

func (f *foo) process() {
     fmt.Fprintf(f.Output, "start\n")
     // ...
}

So we should provide a usable default here. With interface types, one good way is to pass something that provides a no-op implementation of the interface. And it turns out that the stdlib ioutil package comes with a no-op io.Writer, called ioutil.Discard.

  Top Tip — Avoid nil checks via default no-op implementations.

We could pass that into the fooConfig object, but that’s still fragile. If the caller forgets to do it at the callsite, we’ll still end up with a nil parameter. So, instead, we can create a sort of safety within the constructor.

func newFoo(..., cfg fooConfig) *foo {
    if cfg.Output == nil {
        cfg.Output = ioutil.Discard
    }
    // ...
}

This is just an application of the Go idiom make the zero value useful. We allow the zero value of the parameter (nil) to yield good default behavior (no-op).

  Top Tip — Make the zero value useful, especially in config objects.

Let’s revisit the constructor. The parameters fooKey, bar, period, output are all dependencies. The foo object depends on each of them in order to start and run successfully. If there’s a single lesson I’ve learned from writing Go code in the wild and observing large Go projects on a daily basis for the past six years, it is this: make dependencies explicit.

  Top Tip — Make dependencies explicit!

An incredible amount of maintenance burden, confusion, bugs, and unpaid technical debt can, I believe, be traced back to ambiguous or implicit dependencies. Consider this method on the type foo.

func (f *foo) process() {
    fmt.Fprintf(f.Output, "start\n")
    result := f.Bar.compute()
    log.Printf("bar: %v", result) // Whoops!
    // ...
}

fmt.Printf is self-contained and doesn’t affect or depend on global state; in functional terms, it has something like referential transparency. So it is not a dependency. Obviously, f.Bar is a dependency. And, interestingly, log.Printf acts on a package-global logger object, it’s just obscured behind the free function Printf. So it, too, is a dependency.

What do we do with dependencies? We make them explicit. Because the process method prints to a log as part of its work, either the method or the foo object itself needs to take a logger object as a dependency. For example, log.Printf should become f.Logger.Printf.

func (f *foo) process() {
    fmt.Fprintf(f.Output, "start\n")
    result := f.Bar.compute()
    f.Logger.Printf("bar: %v", result) // Better.
    // ...
}

We’re conditioned to think of certain classes of work, like writing to a log, as incidental. So we’re happy to leverage helpers, like package-global loggers, to reduce the apparent burden. But logging, like instrumentation, is often crucial to the operation of a service. And hiding dependencies in the global scope can and does come back to bite us, whether it’s something as seemingly benign as a logger, or perhaps another, more important, domain-specific component that we haven’t bothered to parameterize. Save yourself the future pain by being strict: make all your dependencies explicit.

  Top Tip — Loggers are dependencies, just like references to other components, database handles, commandline flags, etc.

Of course, we should also be sure to take a sensible default for our logger.

func newFoo(..., cfg fooConfig) *foo {
    // ...
    if cfg.Logger == nil {
        cfg.Logger = log.New(ioutil.Discard, ...)
    }
    // ...
}

Update: for more detail on this and the subject of magic, see the June 2017 blog post on a theory of modern Go.


To speak about the problem generally for a moment: I’ve had a lot more production experience with logging, which has mostly just increased my respect for the problem. Logging is expensive, more expensive than you think, and can quickly become the bottleneck of your system. I wrote more extensively on the subject in a separate blog post, but to re-cap:

Where logging is expensive, instrumentation is cheap. You should be instrumenting every significant component of your codebase. If it’s a resource, like a queue, instrument it according to Brendan Gregg’s USE method: utilization, saturation, and error count (rate). If it’s something like an endpoint, instrument it according to Tom Wilkie’s RED method: request count (rate), error count (rate), and duration.

If you have any choice in the matter, Prometheus is probably the instrumentation system you should be using. And, of course, metrics are dependencies, too!

Let’s use loggers and metrics to pivot and address global state more directly. Here are some facts about Go:

These facts are convenient in the small, but awkward in the large. That is, how can we test the log output of components that use the fixed global logger? We must redirect its output, but then how can we test in parallel? Just don’t? That seems unsatisfactory. Or, if we have two independent components both making HTTP requests with different requirements, how do we manage that? With the default global http.Client, it’s quite difficult. Consider this example.

func foo() {
    resp, err := http.Get("http://zombo.com")
    // ...
}

http.Get calls on a global in package http. It has an implicit global dependency. Which we can eliminate pretty easily.

func foo(client *http.Client) {
    resp, err := client.Get("http://zombo.com")
    // ...
}

Just pass an http.Client as a parameter. But that is a concrete type, which means if we want to test this function we also need to provide a concrete http.Client, which likely forces us to do actual HTTP communication. Not great. We can do one better, by passing an interface which can Do (execute) HTTP requests.

type Doer interface {
    Do(*http.Request) (*http.Response, error)
}

func foo(d Doer) {
    req, _ := http.NewRequest("GET", "http://zombo.com", nil)
    resp, err := d.Do(req)
    // ...
}

http.Client satisfies our Doer interface automatically, but now we have the freedom to pass a mock Doer implementation in our test. And that’s great: a unit test for func foo is meant to test only the behavior of foo, it can safely assume that the http.Client is going to work as advertised.

Speaking of testing…


In 2014, I reflected on our experience with various testing frameworks and helper libraries, and concluded that we never found a great deal of utility in any of them, recommending the stdlib’s approach of plain package testing with table-based tests. Broadly, I still think this is the best advice. The important thing to remember about testing in Go is that it is just programming. It is not sufficiently different from other programming that it warrants its own metalanguage. And so package testing continues to be well-suited to the task.

TDD/BDD packages bring new, unfamiliar DSLs and control structures, increasing the cognitive burden on you and your future maintainers. I haven’t personally seen a codebase where that cost has paid off in benefits. Like global state, I believe these packages represent a false economy, and more often than not are the product of cargo-culting behaviors from other languages and ecosystems. When in Go, do as Gophers do: we already have a language for writing simple, expressive tests—it’s called Go, and you probably know it pretty well.

With that said, I do recognize my own context and biases. Like with my opinions on the GOPATH, I’ve softened a bit, and defer to those teams and organizations for whom a testing DSL or framework may make sense. If you know you want to use a package, go for it. Just be sure you’re doing it for well-defined reasons.

Another incredibly interesting topic has been designing for testing. Mitchell Hashimoto recently gave a great talk on the subject here in Berlin (SpeakerDeck, YouTube) which I think should be required viewing.

In general, the thing that seems to work the best is to write Go in a generally functional style, where dependencies are explicitly enumerated, and provided as small, tightly-scoped interfaces whenever possible. Beyond being good software engineering discipline in itself, it feels like it automatically optimizes your code for easy testing.

  Top Tip — Use many small interfaces to model dependencies.

As in the http.Client example just above, remember that unit tests should be written to test the thing being tested, and nothing more. If you’re testing a process function, there’s no reason to also test the HTTP transport the request came in on, or the path on disk the results get written to. Provide inputs and outputs as fake implementations of interface parameters, and focus on the business logic of the method or component exclusively.

  Top Tip — Tests only need to test the thing being tested.


Ever the hot topic. In 2014, things were nascent, and about the only concrete advice I could give was to vendor. That advice still holds today: vendoring is still the solution to dependency management for binaries. In particular, the GO15VENDOREXPERIMENT and its concomittant vendor/ subdirectory have become default in Go 1.6. So you’ll be using that layout. And, thankfully, the tools have gotten a lot better. Some I can recommend:

  Top Tip — Use a top tool to vendor dependencies for your binary.

A big caveat for libraries. In Go, dependency management is a concern of the binary author. Libraries with vendored dependencies are very difficult to use; so difficult that it is probably better said that they are impossible to use. There are many corner cases and edge conditions that have played out in the months since vendoring was officially introduced in 1.5. (You can dig in to one of these forum posts if you’re particularly interested in the details.) Without getting too deep in the weeds, the lesson is clear: libraries should never vendor dependencies.

  Top Tip — Libraries should never vendor their dependencies.

You can carve out an exception for yourself if your library has hermetically sealed its dependencies, so that none of them escape to the exported (public) API layer. No dependent types referenced in any exported functions, method signatures, structures—anything.

If you have the common task of maintaining an open-source repository that contains both binaries and libraries, unfortunately, you are stuck between a rock and a hard place. You want to vendor your deps for your binaries, but you shouldn’t vendor them for your libraries, and the GO15VENDOREXPERIMENT doesn’t admit this level of granularity, from what appears to me to be regrettable oversight.

Bluntly, I don’t have an answer to this. The etcd folks have hacked together a solution using symlinks which I cannot in good faith recommend, as symlinks are not well-supported by the go toolchain and break entirely on Windows. That this works at all is more a happy accident than any consequence of design. I and others have raised all of these concerns to the core team, and I hope something will happen in the near term.


Regarding building, one important lesson learned, with a hat tip to Dave Cheney: prefer go install to go build. The install verb caches build artifacts from dependencies in $GOPATH/pkg, making builds faster. It also puts binaries in $GOPATH/bin, making them easier to find and invoke.

  Top Tip — Prefer go install to go build.

If you produce a binary, don’t be afraid to try out new build tools like gb, which may significantly reduce your cognitive burden. Conversely, remember that since Go 1.5 cross-compilation is built-in; just set the appropriate GOOS and GOARCH environment variables, and invoke the appropriate go command. So there’s no need for extra tools here anymore.

Regarding deployment, we Gophers have it pretty easy compared to languages like Ruby or Python, or even the JVM. One note: if you deploy in containers, follow the advice of Kelsey Hightower and do it FROM scratch. Go gives us this incredible opportunity; it’s a shame not to use it.

As more general advice, think carefully before choosing a platform or orchestration system—if you even choose one at all. Likewise for jumping onto the microservices bandwagon. An elegant monolith, deployed as an AMI to an autoscaling EC2 group, is a very productive setup for small teams. Resist, or at least carefully consider, the hype.


The Top Tips:

  1. Put $GOPATH/bin in your $PATH, so installed binaries are easily accessible.  link
  2. Put library code under a pkg/ subdirectory. Put binaries under a cmd/ subdirectory.  link
  3. Always use fully-qualified import paths. Never use relative imports.  link
  4. Defer to Andrew Gerrand’s naming conventions.  link
  5. Only func main has the right to decide which flags are available to the user.  link
  6. Use struct literal initialization to avoid invalid intermediate state.  link
  7. Avoid nil checks via default no-op implementations.  link
  8. Make the zero value useful, especially in config objects.  link
  9. Make dependencies explicit!  link
  10. Loggers are dependencies, just like references to other components, database handles, commandline flags, etc.  link
  11. Use many small interfaces to model dependencies.  link
  12. Tests only need to test the thing being tested.  link
  13. Use a top tool to vendor dependencies for your binary.  link
  14. Libraries should never vendor their dependencies.  link
  15. Prefer go install to go build.  link

Go has always been a conservative language, and its maturity has brought relatively few surprises and effectively no major changes. Consequently, and predictably, the community also hasn’t dramatically shifted its stances on what’s considered best practice. Instead, we’ve seen a reification of tropes and proverbs that were reasonably well-known in the early years, and a gradual movement “up the stack” as design patterns, libraries, and program structures are explored and transformed into idiomatic Go.

Here’s to another 6 years of fun and productive Go programming. 🏌


________
Go back to my website, or follow me on Twitter.