Upgrade Go1.14 to Go 1.17

[KyleJu]

WARNING: Go 1.14 is no longer supported by the Go community as of February 2021. We recommend you to upgrade to the latest version of Go runtime as soon as possible. For details on upgrading, see https://cloud.google.com/appengine/docs/standard/go/runtime.

[KyleJu]

https://github.com/web-platform-tests/wpt.fyi/pull/2767, https://github.com/web-platform-tests/wpt.fyi/pull/2766 should be merged after golang upgrade. Closed them for now

[KyleJu]

#2780

[foolip]

@KyleJu we're currently using Go 1.14, is this bug about upgrading to 1.15?

[KyleJu]

@KyleJu we're currently using Go 1.14, is this bug about upgrading to 1.15?

We will upgrade Go to 1.17. I believe thats the most wildly accepted version at the moment

[foolip]

It looks like AppEngine only support 1.16 still, right?

[KyleJu]

#2798

[KyleJu]

It looks like AppEngine only support 1.16 still, right?

Looking at here, you are right. Appengine only supports 1.16 at the moment. The error messages I am getting are:

# golang.org/x/net/http2
/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:417:45: undefined: os.ErrDeadlineExceeded
note: module requires Go 1.17

I found a related thread here: https://github.com/golang/go/issues/45950. It appears to a downloading issue? And the fix is to upgrade it to Go 1.15, IIUC

[foolip]

If we have some dependency that requires Go 1.17, can't we keep using whatever the current version is, since that must support 1.14?

[KyleJu]

Yea we can do that for now, as I closed the update for https://github.com/web-platform-tests/wpt.fyi/pull/2767, https://github.com/web-platform-tests/wpt.fyi/pull/2766, https://github.com/web-platform-tests/wpt.fyi/pull/2798. But it will be a good idea to upgrade because Go 1.14 is no longer supported by the Go community as of February 2021

[KyleJu]

#2833

[KyleJu]

Golang 1.16 upgrade doesn't seem to solve the issue. Stil gets the same errors after running

go get -t -u -d ./...
go mod tidy

Error:

# golang.org/x/net/http2
/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:417:45: undefined: os.ErrDeadlineExceeded
note: module requires Go 1.17

After some search again, I believe upgrading Go1.17 can solve this dependency upgrade issue. We have to wait until App Engine supports Go1.17

[jcscottiii]

A possibility would be to move the webapp to use a custom runtime.

In that instance, you provide a Dockerfile at the same level as the app.yaml. That Dockerfile can use the latest version of golang from Docker.

---

Pros vs Cons

Benefits of using a custom runtime:

• The same image that is used in app engine, is the same one used locally.
• Can control the

Cons of using a custom runtime:

• You need to maintain the security of the docker image over time

---

Reasons to why should we move the webapp to using the App Engine custom runtime

1. Searchcache and ResultsProcessor already use a custom runtime configuration. That means we are already doing some maintenance.

Results Processor

https://github.com/web-platform-tests/wpt.fyi/blob/75fd16bdd086ff29459c0fd41fe8e86908af5347/results-processor/app.yaml#L1-L3

https://github.com/web-platform-tests/wpt.fyi/blob/b1296e17801c43991445df58e1026e371763ad2d/results-processor/Dockerfile#L1

Search Cache

https://github.com/web-platform-tests/wpt.fyi/blob/207813b3ed18bae81068934caa478daffd782d36/api/query/cache/service/app.yaml#L1-L3

https://github.com/web-platform-tests/wpt.fyi/blob/207813b3ed18bae81068934caa478daffd782d36/api/query/cache/service/Dockerfile#L4

---

Way to implement

1. Change Webapp to use a custom runtime
2. In the respective Dockerfiles for the search cache and web app, change the base image to use the official golang image from docker.

• Example of an gcp app doing this: https://github.com/GoogleCloudPlatform/golang-samples/blob/f607bc269bb2c01861ff29b243ee50b1f4e67d10/cloudsql/mysql/database-sql/Dockerfile#L18
• Preferably version 1.18

---

Other notes

Things to keep in mind:

Make sure webapp conforms to the rules of the custom runtime: https://cloud.google.com/appengine/docs/flexible/custom-runtimes/build#code

[KyleJu]

A possibility would be to move the webapp to use a custom runtime.

In that instance, you provide a Dockerfile at the same level as the app.yaml. That Dockerfile can use the latest version of golang from Docker.

Pros vs Cons

Benefits of using a custom runtime:

• The same image that is used in app engine, is the same one used locally.
• Can control the

Cons of using a custom runtime:

• You need to maintain the security of the docker image over time

Reasons to why should we move the webapp to using the App Engine custom runtime

1. Searchcache and ResultsProcessor already use a custom runtime configuration. That means we are already doing some maintenance.

Results Processor

https://github.com/web-platform-tests/wpt.fyi/blob/75fd16bdd086ff29459c0fd41fe8e86908af5347/results-processor/app.yaml#L1-L3

https://github.com/web-platform-tests/wpt.fyi/blob/b1296e17801c43991445df58e1026e371763ad2d/results-processor/Dockerfile#L1

Search Cache

https://github.com/web-platform-tests/wpt.fyi/blob/207813b3ed18bae81068934caa478daffd782d36/api/query/cache/service/app.yaml#L1-L3

https://github.com/web-platform-tests/wpt.fyi/blob/207813b3ed18bae81068934caa478daffd782d36/api/query/cache/service/Dockerfile#L4

Way to implement

1. Change Webapp to use a custom runtime
2. In the respective Dockerfiles for the search cache and web app, change the base image to use the official golang image from docker.

• Example of an gcp app doing this: https://github.com/GoogleCloudPlatform/golang-samples/blob/f607bc269bb2c01861ff29b243ee50b1f4e67d10/cloudsql/mysql/database-sql/Dockerfile#L18
• Preferably version 1.18

Other notes

Things to keep in mind:

Make sure webapp conforms to the rules of the custom runtime: https://cloud.google.com/appengine/docs/flexible/custom-runtimes/build#code

@foolip I don't have enough experience on build to comment on it but will look into it. WDYT?

[jcscottiii]

I was able to get a working implementation of my suggestion. However, through the process of moving from App Engine standard to App Engine flexible, I learned that users of App Engine flexible need to implement their own caching for static resources. (Source: Documentation of serving static content in App Engine flexible environment (new way) vs App Engine standard environment (current way))

As a result, there would need to be a bigger change than worth the value at the moment to accommodate that.

Recommended that we will continue to leverage what the latest that App Engine standard environment supports, Go1.16. I looked at #2846 and was able to make the changes to get it to work. Changes include:

• Fixed the blocking issue by changing the builder image for searchcache from gcr.io/gcp-runtimes/go1-builder:1.14 to golang:1.16-buster
  • Why: Searchcache leverages the bring your own container in flexible App Engine.
  • Also, I changed the final application image for searchcache from gcr.io/distroless/base:latest to gcr.io/distroless/static-debian11. Distroless maintains a list of images for users. The former base image is meant to be an intermediate image. Also, added Go build flags which are needed. The same flags were retrieved from the distroless Go example
• Adding docs for future upgrades

Staging link

Outstanding issues: We still will have the issue of dependencies that depend on Go1.17 will need to wait until it becomes available in the standard App Engine environment. But this will get us off of the unsupported Go 1.14 version

[KyleJu]

As a result, there would need to be a bigger change than worth the value at the moment to accommodate that.

@jcscottiii Thanks for the investigation and a great attempt for the migration https://github.com/web-platform-tests/wpt.fyi/pull/2905! (personally I am jaded by any sort of migrations because I have done so many with GCP).

I agree with the conclusion here - the engineering hours that could spend on this migration need to be justified. There are a list of high-level comparisons between flex vs standard. The notable implications are scaling and cost. The standard environment have the ability to handle extreme spikes of traffic and scale from 0 to many instances, back to 0 within seconds, whereas flex has at least one a long-running daemon and expects a consistent traffic. Moreover, flex environment has much higher cost. The billing for wpt.fyi is something we have been keeping an eye on.

On the developer side, I found that the standard is easier to test and develop locally, while I had difficulty building flex locally.

[jcscottiii]

We can close this 🎉 with #3067 and #3070