[BUG] New flytectl version ignores storage settings from config file

[fg91]

Describe the bug

flytectl v0.6.12 ignores the storage settings, e.g. endpoint url, in the config file while v0.5.8 does not.

admin:
  ...
logger:
   ...
storage:
  connection:
    endpoint: http://minio.flyte.svc.cluster.local:9000
    ...
  type: minio
  container: "my-s3-bucket"
  enable-multicontainer: true

(I have not tested whether other settings are ignored as well.)

Expected behavior

The new version should not ignore the storage settings.

Additional context to reproduce

I'm executing the following script in a container with either v0.6.12 (latest) or v0.5.8:

pyflyte --pkgs flytesnacks.workflows package --fast --source . --image k3d-registry.localhost:5000/workflow:latest -f

cat > config.yaml <<'_EOF'
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyteadmin.flyte.svc.cluster.local:81
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
storage:
  connection:
    access-key: minio
    auth-type: accesskey
    disable-ssl: true
    endpoint: http://minio.flyte.svc.cluster.local:9000
    region: us-east-1
    secret-key: miniostorage
  type: minio
  container: "my-s3-bucket"
  enable-multicontainer: true
_EOF

flytectl version
flytectl --config config.yaml register files --project flytesnacks --domain development --archive flyte-package.tgz --version {{inputs.parameters.version}}

I'm getting the following for v0.6.12:

{
  "App": "flytectl",
  "Build": "0a68a57",
  "Version": "0.6.12",
  "BuildTime": "2022-09-12 15:20:25.6728525 +0000 UTC m=+0.071785901"
}{
  "App": "controlPlane",
  "Build": "f10a7f2",
  "Version": "v1.1.28-2-gf10a7f2",
  "BuildTime": "2022-09-12 11:58:02.8187065 +0000 UTC m=+0.044260701"
Error: failed to upload source code from [/tmp/register4188038613/fastfc94cfb95c9d5602278cd12d2c5a08c3.tar.gz]. Error: Put "http://l
ocalhost:30084/my-s3-bucket/91/flytesnacks/development/3OHD3Z5W76BDAC2Q5XFRGTPLCY%3D%3D%3D%3D%3D%3D/fastfc94cfb95c9d5602278cd12d2c5a
08c3.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20220912%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220912T1
52031Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=e646c17183a38a5a4aa0c51d59a6ae7ba19977234399c761181
541758c8667c2": dial tcp [::1]:30084: connect: connection refused

Notice the Error: Put "http://localhost:30084/my-s3-bucket which is not the endpoint url specified in the config file.

For v0.5.8 it works as expected:

{
  "App": "flytectl",
  "Build": "ebda958",
  "Version": "0.5.8",
  "BuildTime": "2022-09-12 15:26:51.7031919 +0000 UTC m=+0.064945701"
}{
  "App": "controlPlane",
  "Build": "f10a7f2",
  "Version": "v1.1.28-2-gf10a7f2",
  "BuildTime": "2022-09-12 11:58:02.8187065 +0000 UTC m=+0.044260701"
} ---------------------------------------------------------------------------- --------- ------------------------------
| NAME (5)                                                                   | STATUS  | ADDITIONAL INFO              |
 ---------------------------------------------------------------------------- --------- ------------------------------
| /tmp/register1258794327/0_flytesnacks.workflows.workflow.prepare_data_1.pb | Success | Successfully registered file |
...

Screenshots

No response

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[welcome[bot]]

Thank you for opening your first issue here! 🛠

[kumare3]

@fg91 we have gone away from these settings and use FlyteAdmin to generate the storage request using signed urls

[fg91]

@fg91 we have gone away from these settings and use FlyteAdmin to generate the storage request using signed urls

Makes sense, I recently assigned the iam.serviceAccounts.signBlob permission to the flyte admin service accounts in our GKE clusters and fast registration works there without a problem.

In the current scenario I use the flyte-deps and flyte-core helm charts with the sandbox values for an MLOps tutorial. The underlying bucket, thus, is minio. In the tutorial I'm trying to demonstrate how flyte entities can be registered from CICD at the example of an argo workflow. Is minio able to issue signed urls? Because the error message shown above suggests that writing to Error: Put "http://localhost:30084/my-s3-bucket is not working.

---

The --storage.* args should probably be removed from these docs.

[eapolinario]

@fg91 , as @kumare3 mentioned, we now rely on flyteadmin to produce signed urls which are then used by the clients to upload the artifacts. In this particular case, you mentioned you're using the sandbox helm charts, in there you'll find this override. Can you remove that block and redeploy them again?

I opened https://github.com/flyteorg/flyte/issues/2890 to track the removal of those options from the docs.

[github-actions[bot]]

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

[github-actions[bot]]

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

[fg91]

@eapolinario from my side this issue could be closed but I saw that you reopened it on Nov 2, 23. Ok to close?