cube icon indicating copy to clipboard operation
cube copied to clipboard

Bump dependency of aws-creds to fix instance metadata

Open bartlomiejolma opened this issue 3 years ago • 5 comments

Check List

  • [ ] Tests has been run in packages where changes made if available
  • [ ] Linter has been run for changed code
  • [ ] Tests for the changes have been added if not covered yet
  • [ ] Docs have been added / updated if required

Issue Reference this PR resolves

None

Description of Changes Made (if issue reference is not provided)

I want to bring this update of the aws-creds into cube: https://github.com/durch/rust-s3/pull/139/files

I was trying to use the instance metadata to mount the remote storage from s3 using the instance metadata of the pod running in the eks cluster.

I was facing the error:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: CubeError { message: "No credentials provided as arguments, in the environment or in the profile file. \n Json Error: expected value at line 1 column 1", backtrace: "", cause: User }', /build/cubestore/cubestore/src/config/mod.rs:848:86
stack backtrace:
   0: rust_begin_unwind
             at /rustc/71b8742bbcbed2cd908dbc031d6552d8b528c037/library/std/src/panicking.rs:515:5
   1: core::panicking::panic_fmt
             at /rustc/71b8742bbcbed2cd908dbc031d6552d8b528c037/library/core/src/panicking.rs:92:14
   2: core::result::unwrap_failed
             at /rustc/71b8742bbcbed2cd908dbc031d6552d8b528c037/library/core/src/result.rs:1355:5
   3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   4: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   6: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   8: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: cubestored::main::{{closure}}
  12: tokio::park::thread::CachedParkThread::block_on
  13: tokio::runtime::thread_pool::ThreadPool::block_on
  14: tokio::runtime::Runtime::block_on
  15: cubestored::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I was using the kube2iam for assuming roles.

Running the command:

curl http://169.254.169.254/latest/meta-data/iam/info

I got the wrong profile name (not the one assigned by the pod annotations). The subsequent call for the details returns a non JSON response which causes the above error.

When I use the endpoint provided in a fix the I get the correct profile name

curl http://169.254.169.254/latest/meta-data/iam/security-credentials

bartlomiejolma avatar Apr 19 '22 12:04 bartlomiejolma

@bartlomiejolma Thanks for contributing it! However, this one won't be that easy. We'd need to update Rust in Cube Store first which will be quite a change. cc @waralex @cristipp

paveltiunov avatar Apr 20 '22 03:04 paveltiunov

@bartlomiejolma Rust version has been updated. Could you please rebase your branch so we can test your changes on CI? Thanks!

paveltiunov avatar Aug 15 '22 07:08 paveltiunov

Hello, is 0.28.0 latest that we can use with cubestore? Will this PR be merged? @bartlomiejolma could you rerun tests and merge that?

Hankier avatar Nov 24 '22 15:11 Hankier

It seems that without the deps upgrade, this doesn't work?

Cube Store can retrieve security credentials from instance metadata automatically. This means you can skip defining the CUBESTORE_AWS_ACCESS_KEY_ID and CUBESTORE_AWS_SECRET_ACCESS_KEY environment variables.

https://cube.dev/docs/caching/running-in-production#storage-aws

vladvasyuk avatar Feb 17 '23 09:02 vladvasyuk

Are there any updates on this? Is it just a matter of updating the library, or do we need to do something else?

I could try to solve this with some directions.

ndrluis avatar Feb 07 '24 18:02 ndrluis

I am going to close this PR in favor of https://github.com/cube-js/cube/pull/6019, which was merged.

Thanks

ovr avatar Apr 16 '24 12:04 ovr