Sagemaker PipelineExecution

[ryansteakley]

Is your feature request related to a problem? Pipeline Execution Operations

• StartPipelineExecution - > Create (Creates a pipelineExeuctionArn unique identifier)

• StopPipelineExecution → Delete

• UpdatePipelineExecution → Update

• DescribePipelineExecution → Describe

• RetryPipelineExecution →(Reuse existing pipelineExecutionArn to rerun may not support this) Does not appear to create a new PipelineExecutionArn instead re-using the existing one. Can be called when PipelineExecution has failed

• Other methods are outside the scope of Control Plane operations and will not be implemented i.e

  • SendPipelineExecutionStepSuccess

The api describes pipeline execution create as a “start” command. However we can equate this to a create operation since this generates a PipelinExecutionArn similar to any other Sagemaker Job i.e (Trainingjob,ProcessingJob, etc...) The workflow would simply be to Start the Pipeline and have it execute, until user decides to delete it.

 StartPipelineExecution:
    operation_type: Create
    resource_name: PipelineExecution
 StopPipelineExecution:
    operation_type: Delete
    resource_name: PipelineExecution

Does the community have any issues with such a resource existing in ACK framework?

Describe the solution you'd like A description of what you want to happen.

Describe alternatives you've considered A description of any alternative solutions or features you've considered.

[surajkota]

Some more information,

Each pipeline execution is a 1-1 mapping which has a resource identifier(ARN) in SageMaker. The resource once created has a states like 'Executing'|'Stopping'|'Stopped'|'Failed'|'Succeeded' which is similar to a TrainingJob resource in SageMaker.

We are not planning to support retry, only plan to support CRUD operations. (edit) To confirm with customer if this is acceptable

@ryansteakley SendPipelineExecutionStepSuccess how is it used? Can you provide an example. I want to understand if this should be part of discussion at all. It could be used while defining the pipeline

Customer usecase - Data scientists are responsible for creating models and building a repeatable workflow and platform team wants to control the pipeline execution and deployment of the model to improve the time to market/prod. They do not want to re-invent creating a workflow using K8s tools when SageMaker already has one.

[jaypipes]

@ryansteakley @surajkota seems fairly straightforward and similar to TrainingJob to me... Ryan, did you need assistance generating the resource manager for PipelineExecution resources?

[ryansteakley]

@jaypipes Thanks, don't currently need help on it.

[RedbackThomson]

RetryPipelineExecution ... Can be called when PipelineExecution has failed

You might want to think about what customers expect here. I imagine that creating a single pipeline execution, and then it fails, they would delete the resource (kubectl delete) and then recreate it, in which case you wouldn't restart it?

(I see @surajkota has already spoken about this, but that's my $0.02)