community
community copied to clipboard
aws-controllers-k8s
Sagemaker PipelineExecution
Is your feature request related to a problem? Pipeline Execution Operations
-
StartPipelineExecution - > Create (Creates a pipelineExeuctionArn unique identifier)
-
StopPipelineExecution→ Delete -
UpdatePipelineExecution→ Update -
DescribePipelineExecution→ Describe -
RetryPipelineExecution→(Reuse existing pipelineExecutionArn to rerun may not support this) Does not appear to create a new PipelineExecutionArn instead re-using the existing one. Can be called when PipelineExecution has failed -
Other methods are outside the scope of Control Plane operations and will not be implemented i.e
-
SendPipelineExecutionStepSuccess
-
The api describes pipeline execution create as a “start” command. However we can equate this to a create operation since this generates a PipelinExecutionArn similar to any other Sagemaker Job i.e (Trainingjob,ProcessingJob, etc...) The workflow would simply be to Start the Pipeline and have it execute, until user decides to delete it.
StartPipelineExecution:
operation_type: Create
resource_name: PipelineExecution
StopPipelineExecution:
operation_type: Delete
resource_name: PipelineExecution
Does the community have any issues with such a resource existing in ACK framework?
Describe the solution you'd like A description of what you want to happen.
Describe alternatives you've considered A description of any alternative solutions or features you've considered.
Some more information,
Each pipeline execution is a 1-1 mapping which has a resource identifier(ARN) in SageMaker. The resource once created has a states like 'Executing'|'Stopping'|'Stopped'|'Failed'|'Succeeded' which is similar to a TrainingJob resource in SageMaker.
We are not planning to support retry, only plan to support CRUD operations. (edit) To confirm with customer if this is acceptable
@ryansteakley SendPipelineExecutionStepSuccess how is it used? Can you provide an example. I want to understand if this should be part of discussion at all. It could be used while defining the pipeline
Customer usecase - Data scientists are responsible for creating models and building a repeatable workflow and platform team wants to control the pipeline execution and deployment of the model to improve the time to market/prod. They do not want to re-invent creating a workflow using K8s tools when SageMaker already has one.
@ryansteakley @surajkota seems fairly straightforward and similar to TrainingJob to me... Ryan, did you need assistance generating the resource manager for PipelineExecution resources?
RetryPipelineExecution... Can be called when PipelineExecution has failed
You might want to think about what customers expect here. I imagine that creating a single pipeline execution, and then it fails, they would delete the resource (kubectl delete) and then recreate it, in which case you wouldn't restart it?
(I see @surajkota has already spoken about this, but that's my $0.02)
