pipelines icon indicating copy to clipboard operation
pipelines copied to clipboard

[sdk] gcp secret depreciated

Open N214 opened this issue 1 year ago • 3 comments

Environment

  • KFP version: 1.7
  • KFP SDK version: 1.8.22
  • All dependencies version:

Steps to reproduce

In, kfp v1, I use the line below to pass the service account to the pipeline pod

dsl.get_pipeline_conf().add_op_transformer(gcp.use_gcp_secret('user-gcp-sa'))

However, by using kfp.v2.dsl, I get

module 'kfp.v2.dsl' has no attribute 'get_pipeline_conf'

Using the new method below works

@dsl.pipeline(
    name='GCP access in pipeline',
)
def cred_pipeline():
    from kfp.gcp import use_gcp_secret
    container_op = read_gcs().apply(use_gcp_secret('user-gcp-sa'))

But I see that the gcp file has been moved to kfp.depreciated in v2, do we have any alternative to use a service account key in v2?

Expected result

Materials and Reference


Impacted by this bug? Give it a 👍.

N214 avatar Nov 01 '23 05:11 N214

@N214 For running Pipelines on GCP, we are recommending to use Vertex SDK, where you can specify the service account there.

cc @connor-mccarthy

zijianjoy avatar Nov 02 '23 22:11 zijianjoy

@N214 For running Pipelines on GCP, we are recommending to use Vertex SDK, where you can specify the service account there.

Ah interesting ! Thank you

Just to explain a little bit more my use case here, I have on-prem hardware that I wish to use to train my models, but some datasets are store in GCS or Bigquery, so I just need a service account to be passed to the pipeline pod to have access to GCP. The pipelines should run on prem, not on GCP.

Thank you

N214 avatar Nov 03 '23 00:11 N214

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Feb 01 '24 07:02 github-actions[bot]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

github-actions[bot] avatar Mar 14 '24 07:03 github-actions[bot]