Today I Learned — Fix: go get private repository return error reading sum.golang.org/lookup

Fixing error: go get gomodule on private module with error message pattern verifying git-host… reading sum.golang.org/lookup … 410 gone

Today on a bright day, Sunday Funday, but ruined by something that made me furious for an hour. Actually, I’ve stuck with this issue since last night (Saturday Night), but since I’m really tired, I decide to stop and rest and continue again on Sunday.

So, I’ve been working on Mabar (https://mabar.id) as one of my side projects. It’s still on the beta version, still lack of many features, you can try it on Android Play Store here

I have a high expectation on this Mabar, it should be a platform, that available on many interfaces (Mobile, Desktop, Web). For the tech-stack, I use Golang as the backend, Kubernetes as the infrastructure, and Digital Ocean behind them as the Server.

Problems

So the issues is, I have a private module (Golang module), a simple library that will be imported by my backend API. But somehow, I can’t get the module and always bring errors when I do the go get command.

Let’s say the package name is lucifer. And it always throw error on the terminal. That really made me mad on it.

$ go get -v bitbucket.org/compay/lucifer
go: finding bitbucket.org/compay/lucifer latest
go: downloading bitbucket.org/compay/lucifer v0.0.0-20190921175342-61a76c096369
**verifying bitbucket.org/compay/lucifer@v0.0.0-20190921175342-61a76c096369: bitbucket.org/compay/lucifer@v0.0.0-20190921175342-61a76c096369: reading** [**https://sum.golang.org/lookup/bitbucket.org/**](https://sum.golang.org/lookup/bitbucket.org/gokar/lucifer@v0.0.0-20190921175342-61a76c096369) **compay** [**/lucifer@v0.0.0-20190921175342-61a76c096369**](https://sum.golang.org/lookup/bitbucket.org/gokar/lucifer@v0.0.0-20190921175342-61a76c096369) **: 410 Gone**

If you look the message, it says the package is gone or not available in sum.golang.org.

At first, I think this happens because I forgot to enforce SSH on bitbucket, as I ever write it here: https://medium.com/easyread/today-i-learned-fix-go-get-private-repository-return-error-terminal-prompts-disabled-8c5549d89045

But, it still not worked well. It still keeps returning an error when I do the go get commands, even I force it only using SSH.

Root Causes

So, after searching for this issue on the internet, I found the root-causes. This only happens on Golang from version 1.13. I can verify this after reading this release https://golang.org/doc/go1.13#modules

So, this happened because there’s a new feature about proxy in this Golang version.

Solutions

Actually there are a few solutions that we can choose.

• Using GOPRIVATE

As stated in the release doc of Go 1.13,

The new GOPRIVATE environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level GONOPROXY and GONOSUMDB variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database.

Means, to solve the issue above, we can just fill the GOPRIVATE variable in our system. Add this command in the ~/.bashrc. *Change the export value based on your company/org name.

**export**  **GOPRIVATE** ="gitlab.com/ **idmabar** ,bitbucket.org/ **idmabar** ,github.com/ **idmabar**"

And to verify if this worked, you can do the go env command. It’s should be more like this.

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/imantumorang/Library/Caches/go-build"
GOENV="/Users/imantumorang/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"  
GOOS="darwin"
GOPATH="/Users/imantumorang/go"
**GOPRIVATE="gitlab.com/idmabar,bitbucket.org/idmabar,github.com/idmabar"**
GOPROXY="[https://proxy.golang.org,direct](https://proxy.golang.org,direct)"
GOROOT="/usr/local/Cellar/go/1.13/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.13/libexec/pkg/tool/darwin\_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO\_ENABLED="1"
GOMOD=""
CGO\_CFLAGS="-g -O2"
CGO\_CPPFLAGS=""
CGO\_CXXFLAGS="-g -O2"
CGO\_FFLAGS="-g -O2"
CGO\_LDFLAGS="-g -O2"

And now I can do the go get command for my private repository.

$ go get bitbucket.org/company/lucifer
go: finding bitbucket.org/company/lucifer latest
go: downloading bitbucket.org/company/lucifer v0.0.0-20190921175342-61a76c096369
go: extracting bitbucket.org/company/lucifer v0.0.0-20190921175342-61a76c096369

So this env-variable will tell the go get command to use the private host proxy to retrieve the package.

• Using GONOSUMDB

Another solution, maybe using the GONOSUMDB variable. I still don’t try this, but it seems works after reading this proposal https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md

So you can set this in your environment variable:

**export**  **GONOSUMDB** ="gitlab.com/idmabar,bitbucket.org/idmabar,github.com/idmabar"

Actually this issue only happens in new Golang version 1.13 and afterward. So before updating your Golang version, make sure to set this environment variable.

Here a few links that may be related to this issue, thanks to noveaustack for finding this and posting this in Stackoverflow, I just reposted this because I just know this issue and as a new thing I learned.

References

• Stackoverflow answer: https://stackoverflow.com/a/57887036/4075313
• Proposal about Go Sum DB for go module: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md
• Proxying Checksum DB: https://docs.gomods.io/configuration/sumdb/
• Github Issues related to this issue: #33985 and #32291