service account instructions don't seem to work

[rspier]

What happened?

Got redirected to Google to log in.

What did you expect to happen?

Accessed backend.

How'd it happen?

Following https://www.pomerium.io/docs/topics/impersonation.html#using-the-command-line-interface

#  FP/S1REDACTED= is config.sharedSecret from the helm values.yaml

$ docker run -ti --entrypoint "/bin/pomerium-cli"     pomerium/pomerium:latest     service-account \
    --email [email protected] \
    --aud prometheus.int.mydomain.com     \
    --expiry 1h     \
    --iss authenticate.int.mydomain.com \
    FP/S1REDACTED=
eyJREDACTEDaw
$ jwt="eyJREDACTEDaw"
$ curl https://prometheus.int.mydomain.com     -H "Authorization: Pomerium $jwt"
Login

# using curl -v -L shows a chain of redirects all the way to accounts.google.com/ServiceLogin

What's your environment like?

• Pomerium version (retrieve with pomerium --version or /ping endpoint): 0.10.1
• Server Operating System/Architecture/Cloud: Kubernetes v1.17.9

What's your config.yaml?

address: ":443"
grpc_address: ":443"
certificate_file: "/pomerium/cert.pem"
certificate_key_file: "/pomerium/privkey.pem"
certificate_authority_file: "/pomerium/ca.pem"
authenticate_service_url: https://authenticate.int.mydomain.com
authorize_service_url: https://pomerium-authorize.pomerium.svc.cluster.local
cache_service_url: https://pomerium-cache.pomerium.svc.cluster.local
idp_provider: google
idp_scopes: 
idp_provider_url: 
cookie_secret: RDREDACTED=
shared_secret: FPREDACTED=
idp_client_id: REDACTED.apps.googleusercontent.com
idp_client_secret: REDACTED
databroker_storage_type: memory    
policy:
  - allowed_users:
    - [email protected]
    - [email protected]
    from: https://prometheus.int.mydomain.com
    to: http://prometheus.monitoring.svc.cluster.local:9090
  # and a bunch more allowed-users blocks

What did you see in the logs?

{"level":"info","X-Forwarded-For":["6.7.8.9,10.42.8.119"],"X-Forwarded-Proto":["https"],"ip":"127.0.0.1","user_agent":"curl/7.68.0","request-id":"a64697ab-beb4-48a7-8ef7-803293a0602e","error":"Bad Request: internal/sessions: session is not found","time":"2020-08-25T23:54:52Z","message":"authenticate: session load error"}
{"level":"info","service":"envoy","upstream-cluster":"pomerium-control-plane-http","method":"GET","authority":"10.42.7.17:443","path":"/ping","user-agent":"kube-probe/1.17","referer":"","forwarded-for":"1.2.3.4","request-id":"c506cee5-55d5-425c-8b6b-13c07999e264","duration":0.770931,"size":62,"response-code":200,"response-code-details":"via_upstream","time":"2020-08-25T23:54:53Z","message":"http-request"}
{"level":"info","service":"envoy","upstream-cluster":"pomerium-control-plane-http","method":"GET","authority":"authenticate.int.mydomain.com","path":"/.pomerium/sign_in?pomerium_expiry=1598399992&pomerium_issued=1598399692&pomerium_redirect_uri=https%3A%2F%2Fprometheus.int.mydomain.com%2F&pomerium_signature=REDACTED%3D","user-agent":"curl/7.68.0","referer":"","forwarded-for":"6.7.8.9,10.42.8.119","request-id":"a64697ab-beb4-48a7-8ef7-803293a0602e","duration":1.261382,"size":874,"response-code":302,"response-code-details":"via_upstream","time":"2020-08-25T23:54:53Z","message":"http-request"}
{"level":"info","service":"envoy","upstream-cluster":"pomerium-control-plane-http","method":"GET","authority":"10.42.7.17:443","path":"/ping","user-agent":"kube-probe/1.17","referer":"","forwarded-for":"1.2.3.4","request-id":"f9acfb87-bb37-4366-9e2c-c5223ff72f79","duration":0.934169,"size":62,"response-code":200,"response-code-details":"via_upstream","time":"2020-08-25T23:54:56Z","message":"http-request"}

Anything else?

My end goal is to be able to have automation that can transit our Pomerium proxy to access the backends.

[rspier]

Some further debugging:

jwt.io doesn't seem to like the generated signature, although it's not clear that's useful.

Playing around with decoding the session JWT myself, go-jose.jwt seems to want a proper key for the shared secret, and not just random bytes. But I may be holding it wrong.

[desimone]

Investigating.

[volvotrax]

I'm facing the same problem. Pomerium version v0.10.0 Tested with pomerium-cli 0.10.0, 0.10.6 and 0.12.2-1 The generated JWT signature is not valid. I see that this feature has been removed from the latest version (v0.13 - Breaking change client-side service accounts removed). It would be very useful to us to run automated end-to-end tests in our CI pipeline. Is there another way to bypass the Google sign-in and run the tests? Thank you for your help.

[nareddyt]

One thing to note is that you must use a persistent data broker to support service accounts. https://0-12-0.docs.pomerium.io/docs/topics/data-storage.html#persistence

Otherwise Pomerium will lose the service account session and redirect you to the sign in screen.

Using a persistent data store helped. Instead of a redirect, I now get a 403 Forbidden directly. Probably means i configured the user or audience incorrectly when generating the JWT. But this confirms it's actually reading the JWT.

[desimone]

@volvotrax / @nareddyt ;

As of version v0.13.0, client side service accounts and impersonation have been removed in an effort limit the attack surface area. Sorry for not circling back on this issue sooner @rspier !

Otherwise Pomerium will lose the service account session and redirect you to the sign in screen.

That's exactly right.

Using a persistent data store helped. Instead of a redirect, I now get a 403 Forbidden directly. Probably means i configured the user or audience incorrectly when generating the JWT. But this confirms it's actually reading the JWT.

@nareddyt -- how are you adding the service account? Is this in v0.13?

[nareddyt]

This is in v0.12.2. I'll try to debug it more next weekend, but this is what I've gathered so far.

Create JWT with the command:

docker run -it \
    --env POMERIUM_SHARED_KEY=${POMERIUM_SHARED_SECRET} \
    --network "pomerium" \
    --entrypoint "/bin/pomerium-cli" \
    pomerium/pomerium:v0.12.2 \
    service-account \
    --email "[email protected]" \
    --aud "SERVICE.DOMAIN.com" \
    --sub "USER" \
    --user "USER" \
    --expiry 1h \
    --iss "https://authenticate.DOMAIN.com" \
    --databroker-url "http://pomerium:5443"

Request with Authorization: Pomerium ${jwt} fails with 403 Forbidden as mentioned above. Info logs:

{"level":"info","service":"databroker","type":"type.googleapis.com/user.User","id":"USER","time":"2021-03-29T04:15:15Z","message":"get"}
{"level":"warn","error":"rpc error: code = NotFound desc = record not found","time":"2021-03-29T04:15:15Z","message":"failed to get user from databroker"}
{"level":"info","service":"authorize","request-id":"d6333edf-3633-4ea9-bed3-e45368176574","check-request-id":"5c16c7aa-a1c7-4424-bd5e-175e4e4ab5f8","method":"GET","path":"/_cat/","host":"SERVICE.DOMAIN.com","query":"","allow":false,"status":403,"message":"Forbidden","user":"","groups":[],"time":"2021-03-29T04:15:15Z","message":"authorize check"}
{"level":"info","service":"envoy","upstream-cluster":"","method":"GET","authority":"SERVICE.DOMAIN.com","path":"/_cat/","user-agent":"PostmanRuntime/7.6.0","referer":"","forwarded-for":"192.168.29.1","request-id":"5c16c7aa-a1c7-4424-bd5e-175e4e4ab5f8","duration":10.579643,"size":9,"response-code":403,"response-code-details":"ext_authz_denied","time":"2021-03-29T04:15:16Z","message":"http-request"}

[rspier]

As of version v0.13.0, client side service accounts and impersonation have been removed in an effort limit the attack surface area. Sorry for not circling back on this issue sooner @rspier !

So what's the recommended mechanism for robot access through Pomerium? Using curl to authenticate as a Google account is non-trivial and will be fragile.

[anjmao]

I'm looking for this too. @desimone Maybe it would sense to have static api tokens provider to generate api tokens and share for programatic access. For example I want to access argo cd api from gitlab CI job, this is how configuration could look to support both google idp and static tokens via Authorization header.

authenticate:
  idp:
    provider: google
    clientID: client
    clientSecret: secret
  static_tokens:
    - name: gitlab-ci-argo-token
      value: token123

config:
  rootDomain: argocd.domain.com
  policy:
    - from: https://argocd.domain.com
      to: http://argocd-server.argo.svc.cluster.local:80
      allowed_domains:
        - domain.com
      allowed_static_tokens:
        - gitlab-ci-argo-token

[desimone]

Thanks for the suggestion, @anjmao . Let me think it over. Could you elaborate a bit more on how these tokens would be ideally used? What would be your expectations on those tokens have identity as part of the assertion header (or not?).

[anjmao]

Hi, now I'm thinking that static tokens would not solve the problem for all cases as some applications still expects identity headers and claims.

One way to access internal services programatically bypassing pomerium would be to proxy via k8s api server which actually doesn't require any changes in pomerium.

[rspier]

It looks like programmatic access solves some of the use case here, but there's no obvious way to authenticate as a non-real account. For example, as a Google Cloud Service Account (@...gserviceaccount.com) which could be granted specific access to a specific backend. It's also not clear how long the programmatic access jwt lasts.

The k8s api server approach proposed by @anjmao #issuecomment-823055196 but adds a whole other inbound path with it's own inherent risks and complexities.

One tangible use case (that I'm trying to implement right now): I have a prometheus behind pomerium. I want to meta-monitor it from outside of the k8s cluster. This requires some sort of long-term token.

[rspier]

My prometheus behind pomerium scraping worked for ~10 days, and then the JWT appears to have expired or otherwise become invalid.

Poking at the logs, it looks like maybe the session on the Google side expired. (No manual action was taken to do this.)

[rspier]

Actually, the Pomerium JWT has the 10 day expiration.

Looks like what I'm looking for is the M2M functionality which is part of the enterprise product. see https://github.com/pomerium/pomerium/issues/2307