kfp-tekton icon indicating copy to clipboard operation
kfp-tekton copied to clipboard

Volume claim is generated with access mode ReadWriteMany for default storage class

Open blublinsky opened this issue 4 years ago • 3 comments

/kind bug

Here is a simple KFP application:

from kfp.components import InputPath, InputTextFile, OutputPath, OutputTextFile
from kfp.components import func_to_container_op
from kfp_tekton.compiler import TektonCompiler

# Writing many numbers
@func_to_container_op
def write_numbers(numbers_path: OutputPath(str), start: int = 0, count: int = 10):
    with open(numbers_path, 'w') as writer:
        for i in range(start, count):
            writer.write(str(i) + '\n')


# Reading and summing many numbers
@func_to_container_op
def sum_numbers(numbers_path: InputPath(str)) -> int:
    sum = 0
    with open(numbers_path, 'r') as reader:
        for line in reader:
            sum = sum + int(line)
    return sum

# Printing
@func_to_container_op
def print_text(text_path: InputPath()): # The "text" input is untyped so that any data can be printed
    '''Print text'''
    with open(text_path, 'r') as reader:
        for line in reader:
            print(line, end = '')

# Pipeline to sum 100000 numbers
def sum_pipeline(count: 'Integer' = 100000):
    numbers_task = write_numbers(count=count)
    print_text(numbers_task.output)

    sum_task = sum_numbers(numbers_task.outputs['numbers'])
    print_text(sum_task.output)

yamlFile = 'sum_pipeline.yaml'
TektonCompiler().compile(sum_pipeline, yamlFile)

The execution requires a PVC for execution.

But if you look at generated yaml file, you will see

..................
  workspaces:
  - name: sum-pipeline
    volumeClaimTemplate:
      spec:
        accessModes: [ReadWriteMany]
        resources:
          requests: {storage: 2Gi}

Which requires read write many access mode for a default storage class, which fails on PVC binding

What do I expext

..................
  workspaces:
  - name: sum-pipeline
    volumeClaimTemplate:
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests: {storage: 2Gi}

Which actually works

Environment:

  • Python Version 3.7
  • SDK Version: 0.8.1
  • Tekton Version (use tkn version): v0.24
  • Kubernetes Version (use kubectl version): v1.19.0+d856161
  • OS (e.g. from /etc/os-release):

blublinsky avatar Jun 27 '21 21:06 blublinsky

You can configure the workspace mounting strategy using these env variables https://github.com/kubeflow/kfp-tekton/tree/master/sdk#big-data-passing-workspace-configuration

Tomcli avatar Jun 29 '21 00:06 Tomcli

Thanks @Tomcli, it works great

blublinsky avatar Jun 29 '21 15:06 blublinsky

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Mar 02 '22 08:03 stale[bot]

/close fixed

Tomcli avatar Mar 28 '23 19:03 Tomcli

@Tomcli: Closing this issue.

In response to this:

/close fixed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

google-oss-prow[bot] avatar Mar 28 '23 19:03 google-oss-prow[bot]