Token exchange failed with status 400

[or4dx]

Currently running the fluentd-plugin-gcs on a kops kubernetes cluster where I have the following error;

2024-01-12 14:17:56 +0000 [error]: #0 unexpected error error_class=RuntimeError error="Token exchange failed with status 400"

Below is my configuration;

              <match kube-apiserver-audit>
                @type gcs

                project {{ $.Values.global.cluster.gcp_project_id }}
                bucket cluster-access-{{ $.Values.global.cluster.name }}-{{ $.Values.global.cluster.domain | replace "." "-" }}
                path k8s-audit/

                <buffer tag,time>
                  @type file
                  path /var/log/fluent/gcs
                  timekey 600 # 10 minute partition
                  timekey_wait 10m
                  timekey_use_utc true # use utc
                  chunk_limit_size 256m
                </buffer>
              </match>

I currently have workload identity setup so I did not specify the keyfile path.