Paul Andrew

Extend Monitoring and Troubleshooting

1

Add more monitoring and troubleshooting capabilitites using Azure Workbooks.

Deal with database connection failures ensure metadata is left in good state and ready for a restart.

Improve and harden interactions with Synapse pipelines. Including: - NuGet library updates to latest GA releases. Not running on beta versions. - Create NUnit tests for Synapse. - Understand and...

Finish first pass GitHub Pages Documentation

2

Complete first pass documentation via GitHub pages. Include all current framework features and functionality as a base line and complete set of details.

Using existing check pipeline status calls to evaluate if the parent pipeline is already running. Replace management API calls to support Synapse and ADF pipelines.

Switch Email Alerting Call to Use Utils Pipeline

1

Update the infant email sender to call the utilities pipeline instead of calling the email sender function directly. Update the pipeline deployment processes as part of this change.

Extend the pipeline parameters table to include a new concept 'Parameter Sets'. A parameter set is linked to a Batch ID allowing worker pipelines to be re-used with different values...

Add Support for Framework Controlled Pipeline Cancellations of Workers

1

As part of the infant pipeline iterations. If the metadata current execution table is updated with a special cancellation state. Perform a controlled cancellation of worker pipelines as part of...

Use global params to reduce number of Lookup activity calls in Child and Infant pipelines.

1

For a given execution run use global variables to set parameters that don't change and reduce the number of lookup activity calls without the processing.

When reusing worker pipeline for different reasons provide an extra field to include details/values that support understanding and debugging.