[backend] unknown method ListTasks for service api.TaskService

[vkatrychenko]

Environment

• How did you deploy Kubeflow Pipelines (KFP)? part of a full Kubeflow deployment

• KFP version: 2.2.0

• KFP SDK version: 1.8.22

Steps to reproduce

Run the following pipeline:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    pipelines.kubeflow.org/kfp_sdk_version: 1.8.22
    pipelines.kubeflow.org/pipeline_compilation_time: '2024-07-09T12:58:11.181755'
    pipelines.kubeflow.org/pipeline_spec: '{"description": "env variable", "inputs":
      [{"default": "", "name": "pipeline-root"}, {"default": "pipeline/env variable",
      "name": "pipeline-name"}], "name": "env variable"}'
    pipelines.kubeflow.org/v2_pipeline: 'true'
  generateName: env-variable-
  labels:
    pipelines.kubeflow.org/kfp_sdk_version: 1.8.22
    pipelines.kubeflow.org/v2_pipeline: 'true'
spec:
  arguments:
    parameters:
    - name: pipeline-root
      value: ''
    - name: pipeline-name
      value: pipeline-env-variable
  entrypoint: env-variable
  serviceAccountName: azure-sa
  templates:
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: pipeline-name
            value: '{{inputs.parameters.pipeline-name}}'
          - name: pipeline-root
            value: '{{inputs.parameters.pipeline-root}}'
        name: print-env-variable
        template: print-env-variable
    inputs:
      parameters:
      - name: pipeline-name
      - name: pipeline-root
    name: env-variable
  - container:
      args:
      - sh
      - -c
      - "\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3\
        \ -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1\
        \ python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.22'\
        \ && \"$0\" \"$@\"\n"
      - sh
      - -ec
      - 'program_path=$(mktemp -d)

        printf "%s" "$0" > "$program_path/ephemeral_component.py"

        python3 -m kfp.v2.components.executor_main                         --component_module_path                         "$program_path/ephemeral_component.py"                         "$@"

        '
      - "\nimport kfp\nfrom kfp.v2 import dsl\nfrom kfp.v2.dsl import *\nfrom typing\
        \ import *\n\ndef print_env_variable():\n    import os\n\n    print(f\"ENV_VARIABLE\
        \ value is: {os.getenv('ENV_VARIABLE')}\")\n\n"
      - --executor_input
      - '{{$}}'
      - --function_to_execute
      - print_env_variable
      command:
      - /kfp-launcher/launch
      - --mlmd_server_address
      - $(METADATA_GRPC_SERVICE_HOST)
      - --mlmd_server_port
      - $(METADATA_GRPC_SERVICE_PORT)
      - --runtime_info_json
      - $(KFP_V2_RUNTIME_INFO)
      - --container_image
      - $(KFP_V2_IMAGE)
      - --task_name
      - print-env-variable
      - --pipeline_name
      - '{{inputs.parameters.pipeline-name}}'
      - --run_id
      - $(KFP_RUN_ID)
      - --run_resource
      - workflows.argoproj.io/$(WORKFLOW_ID)
      - --namespace
      - $(KFP_NAMESPACE)
      - --pod_name
      - $(KFP_POD_NAME)
      - --pod_uid
      - $(KFP_POD_UID)
      - --pipeline_root
      - '{{inputs.parameters.pipeline-root}}'
      - --enable_caching
      - $(ENABLE_CACHING)
      - --
      - --
      env:
      - name: ENV_VARIABLE
        value: test_value
      - name: KFP_POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
      - name: KFP_POD_UID
        valueFrom:
          fieldRef:
            fieldPath: metadata.uid
      - name: KFP_NAMESPACE
        valueFrom:
          fieldRef:
            fieldPath: metadata.namespace
      - name: WORKFLOW_ID
        valueFrom:
          fieldRef:
            fieldPath: metadata.labels['workflows.argoproj.io/workflow']
      - name: KFP_RUN_ID
        valueFrom:
          fieldRef:
            fieldPath: metadata.labels['pipeline/runid']
      - name: ENABLE_CACHING
        valueFrom:
          fieldRef:
            fieldPath: metadata.labels['pipelines.kubeflow.org/enable_caching']
      - name: KFP_V2_IMAGE
        value: python:3.9
      - name: KFP_V2_RUNTIME_INFO
        value: '{"inputParameters": {}, "inputArtifacts": {}, "outputParameters":
          {}, "outputArtifacts": {}}'
      envFrom:
      - configMapRef:
          name: metadata-grpc-configmap
          optional: true
      image: python:3.9
      volumeMounts:
      - mountPath: /kfp-launcher
        name: kfp-launcher
    initContainers:
    - command:
      - launcher
      - --copy
      - /kfp-launcher/launch
      image: gcr.io/ml-pipeline/kfp-launcher:1.8.7
      mirrorVolumeMounts: true
      name: kfp-launcher
    inputs:
      parameters:
      - name: pipeline-name
      - name: pipeline-root
    metadata:
      annotations:
        pipelines.kubeflow.org/component_ref: '{}'
        pipelines.kubeflow.org/v2_component: 'true'
      labels:
        owner: milosjava
        pipelines.kubeflow.org/enable_caching: 'true'
        pipelines.kubeflow.org/kfp_sdk_version: 1.8.22
        pipelines.kubeflow.org/pipeline-sdk-type: kfp
        pipelines.kubeflow.org/v2_component: 'true'
    name: print-env-variable
    retryStrategy:
      backoff:
        duration: '1'
        factor: 2
        maxDuration: 10m
      limit: 1
    volumes:
    - name: kfp-launcher

In our case, we receive this error: F0709 10:41:55.338276 29 main.go:50] Failed to execute component: failure while getting executionCache: failed to list tasks: rpc error: code = Unimplemented desc = unknown method ListTasks for service api.TaskService time="2024-07-09T10:41:56.096Z" level=info msg="sub-process exited" argo=true error="<nil>" Error: exit status 1

Expected result

Pipeline should complete successfully without errors.

Materials and Reference

---

Impacted by this bug? Give it a 👍.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.