Enable Loki to reload AWS credentials at certain frequency to avoid ExpiredToken errors when using S3 as storage

[ec-appsoss]

Is your feature request related to a problem? Please describe. We have implemented OSS Loki on pods but would like to use S3 as storage. This is working until the AWS credentials expire (10 hours). When the token has expired, Loki is not able to write on the S3 storage therefore making it unusable. The only way to avoid is bounce the Loki pods before it expires.

Describe the solution you'd like An ideal solution would be to let Loki reload or re-read AWS credentials which are stored in a meta-directory or environment variables.

Describe alternatives you've considered Currently, we are using local filesystem to store everything. However, this prevents us from breaking down Loki into micro services, i.e. Loki-read and Loki-write (ingester/querier) as it requires a common storage system like S3/Minio etc.

Additional context

[ec-appsoss]

@dannykopping any suggestions on this can be accomplished?

[dannykopping]

Periodically reloading the credentials from a file would be the most idiomatic approach here, I think.

[ec-appsoss]

@dannykopping I agree. I know that Loki has a reload endpoint but it is only implemented for a few parameters. If these parameters can be expanded, that will be great! With automatic reloading, Loki doesn't need to stop/start.

[ec-appsoss]

Is there any other recommendation on this can be done?

[dannykopping]

How did you create your key?

AFAIK you should be able to create keys that are permanent with aws iam create-access-key.

[ec-appsoss]

The following keys have expiration after server hours as required by our security policies aws_access_key_id aws_secret_access_key aws_session_token

I believe these are the same credentials being used to access S3 storage associated with our account.

[dannykopping]

OK.

As I mentioned before, you'll probably need to distribute a configmap with the new credentials and have those reloaded live. I'm not familiar with another way to achieve what you're hoping for, but maybe someone else in the community has another option.

[ec-appsoss]

Hi, yeah the objective is to load these credentials live for a regular period of time to not have "ExpiredToken" errors. Is there a way for Loki to reload them without restarting it? These parameters are embedded within Loki's config.yaml file.

s3: ""
s3forcepathstyle: false
bucketnames: loki-us-west-2-core002
endpoint: ""
region: us-west-2
access_key_id: ""   <-- was_access_key_id
secret_access_key: "" <-- was_secret_access_key

Btw, we have Loki OSS 2.4.1

[dannykopping]

Is there a way for Loki to reload them without restarting it?

Not currently; this feature would need to be added

[ec-appsoss]

I hope it can be considered soon and added as option to the runtime_config module

[itspngu]

I'd like to chime in and mention that this is a problem with GCS and JSON keys for GCP service accounts as well.

Additionally, it would imho be very preferable to have this sort of functionality built into Loki itself instead of having to rely on externally triggered calls of a reload API, process signals or deployment rollouts to enable distribution of credentials via the Secrets Store CSI Driver.

I've also asked about this on Slack before finding this open issue.

[notanaverageman]

I have also spent a few hours trying to use runtime_config feature and server.enable-runtime-reload parameter (which is confusingly under Loki documentation, but only works for Promtail).

My use case is the same: reloading the S3 credentials.

[vlad-diachenko]

I have also spent a few hours trying to use runtime_config feature and server.enable-runtime-reload parameter (which is confusingly under Loki documentation, but only works for Promtail).

My use case is the same: reloading the S3 credentials.

@notanaverageman In general, AWS SDK must be able to automatically reload the credentials. If you do not specify the credentials(access_key and secret_access_key) in Loki config, AWS SDK will resolve the credentials from the host and it should automatically reload them when they are expired. Could you please try this solution?

[notanaverageman]

Thanks, but I use Minio as the S3 provider and the Kubernetes cluster is not on AWS.

[mrmuli]

@vlad-diachenko is there a way maybe from the docs on how to define this relationship between loki config and the AWS SDK? Additionally, does resolving credentials from the host mean via the EC2 Metadata api? (correct me if I am wrong) which is still limiting for cases like on-premise especially because of services like IAM Anywhere

[emedvesApk]

I have also spent a few hours trying to use runtime_config feature and server.enable-runtime-reload parameter (which is confusingly under Loki documentation, but only works for Promtail). My use case is the same: reloading the S3 credentials.

@notanaverageman In general, AWS SDK must be able to automatically reload the credentials. If you do not specify the credentials(access_key and secret_access_key) in Loki config, AWS SDK will resolve the credentials from the host and it should automatically reload them when they are expired. Could you please try this solution?

I configured the containers that require S3 storage, in my loki-distributed helm chart, with:

env:
- name: AWS_SHARED_CREDENTIALS_FILE
  value: /vault/secrets/aws-credentials

where credentials are dinamically refreshed by the vault-agent-injector. Unfortunatly the new credentials are not picked up by loki. If i kill the container, it restarts and uses the refreshed credentials.

I applied the same solution to Mimir and it correctly refreshes credentials without external intervention.

This behaviour should really be fixed. Having to kill and restart containers is not a feasible workaround.

[rgroothuijsen]

I applied the same solution to Mimir and it correctly refreshes credentials without external intervention.

Not sure how Mimir does it, but the AWS SDK developers have indicated that they're not going to support this use case because it's too uncommon. I suppose achieving this in Loki would require regularly creating a new S3 client.

[wdonne]

I'm using pod identity association with Loki and that works fine. This is a variant of the OIDC provider approach, but it is a lot easier to set up and you don't have to put a role annotation in the service account.