microservices-demo icon indicating copy to clipboard operation
microservices-demo copied to clipboard

improvement: fix the cloud debugger errors in nodejs services

Open Shabirmean opened this issue 3 years ago • 1 comments
trafficstars

Describe the bug

  • There are several insufficient scopes authentication errors in the logs for the nodejs services in the CI clusters (onlineboutique-pr and onlineboutique-master)
  • This is because that the Cloud Debugger API is not accessible from these clusters as these clusters were not created with the necessary access scopes.
  • Thus, the nodes that make up these clusters don't have access to the Cloud Debugger API

To Reproduce

  • Create a PR on this repo or you can use an existing PR
  • Navigate to the PR specific namespace on the CI cluster onlineboutique-pr
  • Open up one of the two nodejs services currencyservice or paymentsservice and view the logs
  • Filter logs by ERROR logs

Logs & Screenshots

image

Environment

  • Existing CI clusters for this repo

Additional context

  • This was discovered when debugging the memory leak issue #538

Solution ideas:

  • There were multiple thoughts around handling this

    • i.e. whether we should disable debugger by default and have is disabled in the CI clusters as well
    • see chat for more details
  • I suggest we have the debugger switched on in the CI clusters so that it enables us to debug and experiment a full feature set on the applications whenever a change is made.

  • We can do this by creating the clusters with the --scopes=https://www.googleapis.com/auth/cloud_debugger,gke-default scopes and updating the CI yamls to point to the new cluster, then decommissioning the old CI clusters. I have a PR that achieves this.

  • But there may be extra work that needs to be done according to what the team decides.

    • If the team decides that we want to disable debugger by default on the sample then we might have to modify some of the yamls and ensure that it is only enabled for the CI clusters.
    • We need to have a discussion around whether this will create confusion for external stakeholders, and how we can have parity between the release yamls and the dev yamls.
  • So I have left the PR closed for someone who picks this issue to have a starting point (if they want to use it)

Exposure

  • Internal/Local

Shabirmean avatar Nov 29 '21 22:11 Shabirmean

This issue might not be relavant with the Cloud Debugger sunset being announced as explained in #836

Shabirmean avatar Jun 16 '22 16:06 Shabirmean

Closing this issue now, because:

  • Since https://github.com/GoogleCloudPlatform/microservices-demo/releases/tag/v0.3.5, we turned off by default this Logging/Debugging environment variables
  • The use of Google Cloud Debugger will be deprecated as per https://github.com/GoogleCloudPlatform/microservices-demo/issues/836

In the meantime, if the issue is faced, the proper values for the --scopes parameter when creating the GKE cluster is required, see description of this issue.

mathieu-benoit avatar Aug 31 '22 13:08 mathieu-benoit