Proposal: Pipeline Management APIs

[StewartW]

What and Why

Expose the API management state machine behind an API Gateway.

This will allow for customers to interact with ADF without the need for configuring deployment maps in a central location, and allow for extensions (Such as CloudFormation Resource Providers, UIs etc) to be created. Bringing a new level of control for ADF Pipelines.

How?

In the global.yml create a new nested application with a condition attached. (Allowing for customers to enable/disable the API provisioning depending on their own requirements)

The API Gateway will be created and use IAM authorisation to restrict access as required. (Requiring any consumers of the API to go via SIGv4 signing for Authentication)

The API itself will be straight forward to start with. Exposing CRUDL operations for pipelines.

When it comes to the identification of stale pipelines, historically this would have been a problem, because any pipelines that are defined via the API would not be represented in S3. However recent changes to how this work has created a logical segregation between pipelines defined in different locations. (S3, API etc).

Proposed API methods

/pipelines

POST : Creates a new pipeline, based on the same JSON schema as a deployment map. GET: Returns the pipeline definitions on a page-by-page basis

/pipelines{pipeline-name}

PUT: Updates the pipeline, based on the deployment map provided DELETE: Deletes the pipeline, but will not delete the code commit repository unless specified. GET: Returns the pipeline definition and a summary of the last 10 executions. HEAD: Validate pipeline exists

TL;DR

APIs for executing pipeline management actions.