iam
iam copied to clipboard
indigo-iam
Please allow exchanged and refreshed tokens to have the same durations/lifetimes as their originating access-tokens
Assuming the duration/lifetime of an access-token is:
exp - nbf
In order to reduce confusion, one should be able to configure IAM to give exchanged and refreshed access-tokens the same duration/lifetime as their originating access-token.
The following "full WLCG lifecycle" test shows that it is very confusing when the token exchanger/refresher client (FTS) has a different token duration/lifetime (1 hour in this example) than the end-user client (6 hours in this example). Please note that FTS must exchange end-user tokens for a refresh-tokens as soon as it can. The example shows that in this particular case the end-user does not get the token lifetime they expect.
[c7] fts_tokens (master) > ./full_token_lifecyle.py config.ini
issuer = https://dteam-auth.cern.ch
user_client_id = 45541bfd-09d2-4fec-9328-ca5528f140f8
fts_client_id = af32d0c4-f51f-4b08-9aec-a423a3e0c3cb
src_token jti = 8b7b953a-f60c-4035-a5e8-3b5814f50489
src_token duration = 21600
src_token_from_exchange jti = f892aee3-f8b0-4d7b-b5ce-7e6535cc3cdd
src_token_from_exchange duration = 3600
src_refresh_token jti = a7c61162-7741-48d1-90d4-03853199266b
src_token_from_refresh jti = daca2ebe-da75-4788-9aa6-d9df07697126
src_token_from_refresh duration = 3600
dst_token jti = 02d17c12-fa20-4d83-a723-4dd93a010abe
dst_token duration = 21600
dst_token_from_exchange jti = 358f95b6-8b1d-4163-ab0c-2fb776f66cfe
dst_token_from_exchange duration = 3600
dst_refresh_token jti = da730b25-e5ee-4418-bbd2-536035d8e187
dst_refresh_token_from_refresh jti = da730b25-e5ee-4418-bbd2-536035d8e187
dst_token_from_refresh jti = 13dfd341-9bac-4b12-a617-59179883f9e0
dst_token_from_refresh duration = 3600
[c7] fts_tokens (master) >
Hi,
the AT/RT duration is already configurable by changing the following environment variables:
DEFAULT_ACCESS_TOKEN_VALIDITY_SECONDS(default is 3600)DEFAULT_REFRESH_TOKEN_VALIDITY_SECONDS(default is 2592000)
These are applied to ALL clients.
However, the AT/RT duration can also be changed PER client, so if you want the same values for FTS client and end-user client you should change them through the IAM dashboard ("Tokens" section of the client configuration).
Hi,
Unfortunately what you suggest does not solve the problem for the end-users of FTS.
Clients of the FTS service will not know, or probably care about the timeout settings of the FTS service itself. Different experiments and end-users will expect the time of validity of their access-tokens to be preserved (give or take a few seconds) during the FTS exchange protocol.
Now that I see that you have the DEFAULT_ACCESS_TOKEN_VALIDITY_SECONDS and DEFAULT_REFRESH_TOKEN_VALIDITY_SECONDS variables maybe you can enforce them? What I mean by this, is you could only allow these values to be used and prevent individual client configurations from overiding them.
Regards,
Steven Murray - CERN FTS service manager
From: Roberta Miccoli @.> Sent: Tuesday, April 2, 2024 12:08 To: indigo-iam/iam @.> Cc: Steven Murray @.>; Author @.> Subject: Re: [indigo-iam/iam] Please allow exchanged and refreshed tokens to have the same durations/lifetimes as their originating access-tokens (Issue #738)
Hi,
the AT/RT duration is already configurable by changing the following environment variables:
- DEFAULT_ACCESS_TOKEN_VALIDITY_SECONDS (default is 3600)
- DEFAULT_REFRESH_TOKEN_VALIDITY_SECONDS (default is 2592000) These are applied to ALL clients.
However, the AT/RT duration can also be changed PER client, so if you want the same values for FTS client and end-user client you should change them through the IAM dashboard ("Tokens" section of the client configuration).
— Reply to this email directly, view it on GitHubhttps://github.com/indigo-iam/iam/issues/738#issuecomment-2031597710, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD2NLSGE36IIUFZTFLBG4A3Y3J7SXAVCNFSM6AAAAABFRJDA66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZRGU4TONZRGA. You are receiving this because you authored the thread.Message ID: @.***>
Hi, Unfortunately what you suggest does not solve the problem for the end-users of FTS. Clients of the FTS service will not know, or probably care about the timeout settings of the FTS service itself. Different experiments and end-users will expect the time of validity of their access-tokens to be preserved (give or take a few seconds) during the FTS exchange protocol.
Just to understand the use case: aren't the ATs obtained by FTS via token exchange an internal business of FTS? why should end-users see and/or care about them?
Hi Francesco,
Consider teh following simplified WLCG workflow:
An end user get three access-tokens: fts-submit, src-read and dst-write 2. All three tokens are given to fts by the end user when they submit a job to transfers a file 3. FTS uses the fts-submit token to decide if the job submission is allowed 4. Assuming FTS is happy the fts-submit token is discarded. 5. FTS tries as fast as it can to get refresh tokens for both the src-read and dst-write tokens. 6. FTS uses the refresh tokens if necessary 7. FTS finally orchestrates the transfer using the latest src-read and dst-write tokens.
The problem is step 5. IAM basically forces the validity periods of the original src-read and dst-write tokens to replaced with whatever the FTS client is configured with. One could argue that the original validity lifetimes are simply ignored.
My point is - this is confusing. However after some internal discussions it might not matter. Yes it was difficult to debug tokens last week because of a 6 hour token was effectively replaced by a 1 hour one, however our discussions concluded that the main VOs (ATLAS and CMS) are most probably going to decide on the most optimal token lifetime and this will be set once within their respective FTS client configurations within IAM. The FTS service does not set these values. We have to explain them and give advice.
So, it is true that IAM has caused confusion when debugging a token refresh problem, but end users saw nothing other than a permission failure that has since been fixed. I'm going to let the IAM project go on this one. However if I do get any questions as to why IAM is the way it is then I will kindly aske those concerned to talk with you directly.
Regards,
Steve
From: Francesco Giacomini @.> Sent: Tuesday, April 2, 2024 14:31 To: indigo-iam/iam @.> Cc: Steven Murray @.>; Author @.> Subject: Re: [indigo-iam/iam] Please allow exchanged and refreshed tokens to have the same durations/lifetimes as their originating access-tokens (Issue #738)
Hi, Unfortunately what you suggest does not solve the problem for the end-users of FTS. Clients of the FTS service will not know, or probably care about the timeout settings of the FTS service itself. Different experiments and end-users will expect the time of validity of their access-tokens to be preserved (give or take a few seconds) during the FTS exchange protocol.
Just to understand the use case: aren't the ATs obtained by FTS via token exchange an internal business of FTS? why should end-users see and/or care about them?
— Reply to this email directly, view it on GitHubhttps://github.com/indigo-iam/iam/issues/738#issuecomment-2031919250, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD2NLSEQ6VAUIJALKWFP3C3Y3KQI7AVCNFSM6AAAAABFRJDA66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZRHEYTSMRVGA. You are receiving this because you authored the thread.Message ID: @.***>
Hi Francesco,
In case I was not clear. Step 5 returns two tokens to FTS, the refresh-token and a new access-token, both of which we store in our database. It is this new access-token and any acces-token we get using the refresh-token that have the “FTS client” period of validity.
You commented - “an internal business of FTS”. Again, we never register FTS within any of the IAMs (excluding the WLCG instance used for testing). To give another example, confusion happened during DC24 when a 10 hour token was converted into a 1 hour token via the exchange protocol. Believe me when I say the experiments were confused.
Now, as I said before, let’s drop this with the caveat that any misunderstanding are passed on to the IAM team.
Regards,
Steve
On 2 Apr 2024, at 14:31, Francesco Giacomini @.***> wrote:
Hi, Unfortunately what you suggest does not solve the problem for the end-users of FTS. Clients of the FTS service will not know, or probably care about the timeout settings of the FTS service itself. Different experiments and end-users will expect the time of validity of their access-tokens to be preserved (give or take a few seconds) during the FTS exchange protocol.
Just to understand the use case: aren't the ATs obtained by FTS via token exchange an internal business of FTS? why should end-users see and/or care about them?
— Reply to this email directly, view it on GitHubhttps://github.com/indigo-iam/iam/issues/738#issuecomment-2031919250, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD2NLSEQ6VAUIJALKWFP3C3Y3KQI7AVCNFSM6AAAAABFRJDA66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZRHEYTSMRVGA. You are receiving this because you authored the thread.Message ID: @.***>
Hi Steve, you wrote:
"[...] we never register FTS within any of the IAMs [...]"
But how would an FTS then be allowed to do token exchange operations?
In fact, there are FTS clients registered for ATLAS, CMS and LHCb that have the token exchange grant configured. The one for LHCb says:
Client used by FTS3 service at CERN for token exchange and refresh
In any case, as things have been implemented thus far, "user" tokens are just to get transfer jobs accepted by the FTS, which then can and should have its own timings for trying to make those transfers actually happen.
As we will look further into the token workflows between Rucio/DIRAC and the FTS to improve the scalability and stability of all components involved in the data transfer machinery, we might end up with the client having to provide the source and destination tokens the FTS will actually use, without any exchanges (and potentially even without any refreshing)...
@murrayc3, of course I believe you and of course we are available to clear all misunderstandings that can arise. I hope we can sort most of them via the documentation. We are also available to adjust the behavior of IAM based on feedback from the field. Specifically on the token exchange, the RFC at some point says: "[...] the expiration time of the output token may be influenced by that of the input token [...]"; but before making changes, let's agree on a sensible behavior.
About the current flow of tokens, I wonder why FTS cannot simply be considered a trusted client and ask all the tokens that it needs. We could extend the policy engine in IAM to accommodate such an approach. But this discussion is OT for this issue.
Hi Maarten,
None of the FTS team can register an FTS service client within an experiment IAM provider because we are not members of the respective experiments. Instead we have to create SNOW tickets or contact experiment experts directly and ask them to register FTS on our behalf. This makes for the uncomfortable situation where we cannot see the settings of our clients, such as access-token lifetimes and whether or not refresh tokens are single-shot. This is what I mean by "we cannot register FTS". The WLCG IAM instance is the only one where we can be administrators and see everything. Even this was not enough in the case of the single-shot token configuration because the WLCG IAM was not new enough to show us the necessary configurations options when the related problems arose.
To make things even more fun, it is very difficult if not impossible for the FTS team to register an FTS job submitting client. Without this second type of client we cannot test if everything is OK. This results in the need to do interactive testing with the different experiments.
Now all of these teething problems will eventually go away, but you must understand how difficult it is to get IAM configured correctly and to debug the token lifecycle.
Regards,
Steve
From: Maarten Litmaath @.> Sent: Tuesday, April 2, 2024 19:50 To: indigo-iam/iam @.> Cc: Steven Murray @.>; Author @.> Subject: Re: [indigo-iam/iam] Please allow exchanged and refreshed tokens to have the same durations/lifetimes as their originating access-tokens (Issue #738)
Hi Steve, you wrote:
"[...] we never register FTS within any of the IAMs [...]"
But how would an FTS then be allowed to do token exchange operations?
In fact, there are FTS clients registered for ATLAS, CMS and LHCb that have the token exchange grant configured. The one for LHCb says:
Client used by FTS3 service at CERN for token exchange and refresh
In any case, as things have been implemented thus far, "user" tokens are just to get transfer jobs accepted by the FTS, which then can and should have its own timings for trying to make those transfers actually happen.
As we will look further into the token workflows between Rucio/DIRAC and the FTS to improve the scalability and stability of all components involved in the data transfer machinery, we might end up with the client having to provide the source and destination tokens the FTS will actually use, without any exchanges (and potentially even without any refreshing)...
— Reply to this email directly, view it on GitHubhttps://github.com/indigo-iam/iam/issues/738#issuecomment-2032688226, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD2NLSGUJQWZL4XBY7USBTTY3LVWDAVCNFSM6AAAAABFRJDA66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZSGY4DQMRSGY. You are receiving this because you authored the thread.Message ID: @.***>
Hi Francesco,
After our internal discussions we are currently going in the direction of an optimal token-lifetime that FTS instances will use per experiment. This means there is no work to be done on the IAM side. This direction can of course change in the future if experiments are not happy. So far so good ..
Regards,
Steve
From: Francesco Giacomini @.> Sent: Wednesday, April 3, 2024 10:34 To: indigo-iam/iam @.> Cc: Steven Murray @.>; Mention @.> Subject: Re: [indigo-iam/iam] Please allow exchanged and refreshed tokens to have the same durations/lifetimes as their originating access-tokens (Issue #738)
@murrayc3https://github.com/murrayc3, of course I believe you and of course we are available to clear all misunderstandings that can arise. I hope we can sort most of them via the documentation. We are also available to adjust the behavior of IAM based on feedback from the field. Specifically on the token exchange, the RFChttps://www.rfc-editor.org/rfc/rfc8693 at some point says: "[...] the expiration time of the output token may be influenced by that of the input token [...]"; but before making changes, let's agree on a sensible behavior.
About the current flow of tokens, I wonder why FTS cannot simply be considered a trusted client and ask all the tokens that it needs. We could extend the policy engine in IAM to accommodate such an approach. But this discussion is OT for this issue.
— Reply to this email directly, view it on GitHubhttps://github.com/indigo-iam/iam/issues/738#issuecomment-2033913120, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD2NLSCAGBSY7BXAYMHSH3DY3O5IXAVCNFSM6AAAAABFRJDA66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZTHEYTGMJSGA. You are receiving this because you were mentioned.
Closing since this issue looks like addressed in FTS. Please feel free to reopen it if this is not the case.
