bug: google-cloud-logging plugin refresh_access_token(): failed to refresh google oauth access token

[zkl94]

Current Behavior

it is not able to send request logs to google cloud logging now due to this bug.

Expected Behavior

The logs should be sent to google cloud logging successfully

Error Logs

2024/04/18 19:03:44 [error] 55#55: *67159 [lua] oauth.lua:62: refresh_access_token(): failed to refresh google oauth access token, 20: unable to get local issuer certificate, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443 2024/04/18 19:03:44 [error] 55#55: *67159 [lua] batch-processor.lua:95: Batch Processor[google-cloud-logging] failed to process entries: failed to get google oauth token, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443

Steps to Reproduce

1. run apisix helm chart https://artifacthub.io/packages/helm/apisix/apisix on k8s 1.27 on GKE
2. create the below ApisixGlobalRule CR (the GCP SA has owner access just in case of permission issue)

apiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
  name: google-cloud-logging
spec:
  plugins:
    - name: google-cloud-logging
      enable: true
      config:
        auth_config:
          client_email: "xxxxxx"
          private_key_id: "xxxxx"
          client_id: "xxxxx"
          private_key: "xxxxx"
          project_id: "xxxxx"
          token_uri: "https://oauth2.googleapis.com/token"
          entries_uri: "https://logging.googleapis.com/v2/entries:write"
          auth_uri: "https://accounts.google.com/o/oauth2/auth"
          ssl_verify: false
        batch_max_size: 1
        retry_delay: 1
        inactive_timeout: 10
        max_retry_count: 10

3. just in case it is not applied globally, I added the google-cloud-logging section to ApisixRoute as well:

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: apisix-dashboard
spec:
  http:
    - name: dashboard
      match:
        hosts:
          - xxxx.example.com
        paths:
          - /*
      backends:
        - serviceName: apisix-dashboard
          servicePort: 80
      plugins:
        - name: redirect
          enable: true
          config:
            http_to_https: true
        - name: google-cloud-logging
          enable: true
          config:
            auth_config:
              client_email: "xxxxx"
              private_key_id: "xxxx"
              client_id: "xxx"
              private_key: "xxxx"
              project_id: "xxxxx"
              token_uri: "https://oauth2.googleapis.com/token"
              entries_uri: "https://logging.googleapis.com/v2/entries:write"
              auth_uri: "https://accounts.google.com/o/oauth2/auth"
              ssl_verify: false
            batch_max_size: 1
            retry_delay: 1
            inactive_timeout: 10
            max_retry_count: 10

4. visit xxxx.example.com on browser multiple times
5. check the logs of apisix pods inside kubernetes and we get the error logs:

2024/04/18 19:03:53 [error] 55#55: *67552 [lua] batch-processor.lua:95: Batch Processor[google-cloud-logging] failed to process entries: failed to get google oauth token, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443
2024/04/18 19:03:53 [error] 55#55: *67552 [lua] batch-processor.lua:104: Batch Processor[google-cloud-logging] exceeded the max_retry_count[11] dropping the entries, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443
2024/04/18 19:03:53 [error] 55#55: *67554 [lua] oauth.lua:62: refresh_access_token(): failed to refresh google oauth access token, 20: unable to get local issuer certificate, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443
2024/04/18 19:03:53 [error] 55#55: *67554 [lua] batch-processor.lua:95: Batch Processor[google-cloud-logging] failed to process entries: failed to get google oauth token, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443

6. I checked google-cloud-logging as well, the request logs are not received

Environment

• APISIX version (run apisix version): 3.8.0
• Operating system (run uname -a): (I run it inside the apisix pod) Linux apisix-6bc8f67474-mk5xj 5.15.133+ SMP Wed Nov 8 17:30:28 UTC 2023 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.21.4.2
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[nitishfy]

Thanks for raising this issue! It seems like the SSL/TLS certificate of the server issuing the OAuth token cannot be verified.

[zkl94]

@nitishfy Could you suggest your solution? According to my understanding, Google is the entity issuing the OAuth token. What is the wrong configuration that you see here? Are you saying that google SSL/TLS certificate cannot be verified?

[balaji-n4]

@nitishfy could you explain how to fix the issue or any work around for the issue?