build Archives - Woohoo Services Blog!

Summary

If you have landed here, it is most likely because you attempted to implement the subject and ran into errors. In my use case, I ran into errors when splitting up my Golang app into packages and added unit testing to it. To compound it, I was automatically deploying via triggers with CloudBuild.

This builds upon the article IP Subnet Calculator on Google App Engine. Reading that article will help give you a good reference point for this article.

This article is not a “How To” on actual unit testing but getting it to work under this unique combination of tools. A great unit testing article I found is here – https://blog.alexellis.io/golang-writing-unit-tests/

How To Setup

Before we talk about what might have gone wrong, let’s talk about how to set this up properly. There has been a lot of confusion since Golang went to version 1.11 and started supporting modules. After supporting modules it appears using GOPATH is a less of a supported method. With that said some tools still look for it.

Directory Structure

To get a final state visual of my directory structure, it ended up as follows

tools.woohoosvcs.com
tools.woohoosvcs.com/app.yaml
tools.woohoosvcs.com/go.mod
tools.woohoosvcs.com/subnetcalculator
tools.woohoosvcs.com/subnetcalculator/subnetcalculator.go
tools.woohoosvcs.com/subnetcalculator/subnetcalculator_test.go
tools.woohoosvcs.com/cloudbuild.yaml
tools.woohoosvcs.com/html
tools.woohoosvcs.com/html/root.html
tools.woohoosvcs.com/html/view.html
tools.woohoosvcs.com/README.md
tools.woohoosvcs.com/static
tools.woohoosvcs.com/static/submitdata.js
tools.woohoosvcs.com/static/subnetcalculator.css
tools.woohoosvcs.com/root
tools.woohoosvcs.com/root/root.go
tools.woohoosvcs.com/main.go

Modules

Modules help with the dependency needs of Golang applications. Previously, private packages were difficult to manage as well as specific versioned public packages. Modules helps many of these issues. Here is a good article on modules – https://blog.golang.org/using-go-modules

The key to my issues seemed to be involving modules. Google App Engine’s migrating to 1.11 guide recommends the following.

The preferred method is to manually move all http.HandleFunc() calls from your packages to your main()function in your main package.
Alternatively, import your application’s packages into your main package, ensuring each init() function that contains calls to http.HandleFunc() gets run on startup.
https://cloud.google.com/appengine/docs/standard/go111/go-differences

My app’s directory is “tools.woohoosvcs.com” so I named the root, the same. Run “go mod init”

$ go mod init tools.woohoosvcs.com
go: creating new go.mod: module tools.woohoosvcs.com
dwcjr@Davids-MacBook-Pro tools.woohoosvcs.com % cat go.mod
module tools.woohoosvcs.com

go 1.13

Importing Private Packages

This then lets us to refer to packages and modules under it via something similar.

import ( "tools.woohoosvcs.com/subnetcalculator" )

CloudBuild

My cloudbuild.yaml ended up as follows

steps:
  - id: subnetcalculator_test
    name: "gcr.io/cloud-builders/go"
    args: ["test","tools.woohoosvcs.com/subnetcalculator"]
    #We use modules but this docker wants GOPATH set and they are not compatible.
    env: ["GOPATH=/fakepath"]
  - name: "gcr.io/cloud-builders/gcloud"
    args: ["app", "deploy"]

Interesting tidbit is the “name” is a docker image from Google Cloud Repository or gcr.io

The first id runs the “go” image and runs “go test tools.woohoosvcs.com/subnetcalculator”. It seems the go image wants GOPATH but go test fails with it set so I had to set it to something fake.

It then uses the gcloud deployer which consumes app.yaml

How I Got Here

Unit tests drove me to wanting to split out logic, particularly the subnetcalculator (check it out via https://tools.woohoosvcs.com/subnetcalculator )

Before implementing modules I could get it to run locally by importing ./subnetcalculator but then “gcloud app deploy” would fail.

2019/11/22 01:39:13 Failed to build app: Your app is not on your GOPATH, please move it there and try again.
building app with command '[go build -o /tmp/staging/usr/local/bin/start ./...]', env '[PATH=/go/bin:/usr/local/go/bin:/builder/google-cloud-sdk/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ace17fcba136 HOME=/builder/home BUILDER_OUTPUT=/builder/outputs DEBIAN_FRONTEND=noninteractive GOROOT=/usr/local/go/ GOPATH=/go GOPATH=/tmp/staging/srv/gopath]': err=exit status 1, out=go build: cannot write multiple packages to non-directory /tmp/staging/usr/local/bin/start.

The error was vague but I noticed it was related to the import path. I tried moving the folder to $GOPATH/src and it seemed to deploy via “gcloud app deploy” but then failed via CloudBuild automated trigger.

------------------------------------ STDERR ------------------------------------
2019/11/21 21:31:39 staging for go1.13
2019/11/21 21:31:39 GO111MODULE=auto, but no go.mod found, so building with dependencies from GOPATH
2019/11/21 21:31:39 Staging second-gen Standard app (GOPATH mode): failed analyzing /workspace: cannot find package "tools.woohoosvcs.com/subnetcalculator" in any of:
	($GOROOT not set)
	/builder/home/go/src/tools.woohoosvcs.com/subnetcalculator (from $GOPATH)
GOPATH: /builder/home/go
--------------------------------------------------------------------------------

ERROR
ERROR: build step 0 "gcr.io/cloud-builders/gcloud" failed: exit status 1

It was like a balancing act with a 3 legged chair! Once I initialized the module though and adjusted the imports it worked great

Final Worlds

Here we worked through a best practice of using modules and an internal package to do automated build deploy on Google App Engine. Unit testing with automated deploys are important so that broken builds do not get pushed to production.

Tag: build

Unit Testing Golang on App Engine