cds

cds copied to clipboard

Expected Behavior

As a developer using CDS, When I execute a workflow which interact with 2 (or more) shared resources Then I want to be able to lock these different resources using named mutexes And I expect my workflows to only wait for a given named mutex

As a developer using CDS to run parallel tasks against an (unstable or rate-limited) API, When I execute a workflow which interact with this API Then I want to limit the amount of requests to this API by using multi-valued mutexes And I expect all the workflows and pipelines of my project to wait when all all the values of my mutexes are used

Current Behavior

There is a concept of a mutex (directive one_at_a_time) defined at the workflow level.

But:

• A mutex is a boolean, which does not allow using named resources
• The scope of a mutex is a workflow (and its application) allowing concurent workflow executions on different branches (or the same branch). But it is not defined at pipeline, stage or step level.

** Ideas**

1. Scope Definition Change: Being able to define a mutex (one_at_a_time) at the project and pipeline levels

2. Being able to define multiple mutexes, preferably named:

# Workflow YML
# ...
workflow:
  my-super-workflow:
    pipeline: my-super-pipeline
    application: my-super-application
    mutexes: # or `locks`, or `one_at_a_time`
      - "GRA7-Openstack-API"
      - "SBG5-Openstack-API"

3. Add new action(s) which would take or release a "token" in the mutex, and would allow using pipeline parameters:

# Pipeline YML
# ...
    steps:
      - checkout: '{{.cds.workspace}}'
      - mutex_take: "GRA7-Openstack-API"
      - script: echo "doing something at GRA7"
      - mutex_release: "GRA7-Openstack-API"
      - mutex_take: "{{.cds.pip.os_region}}"
      - script: echo "doing something in the region {{.cds.pip.os_region}}"
      - mutex_release: "{{.cds.pip.os_region}}"

5. Allow multi-valued mutexes:

---
# Workflow YML
# ...
workflow:
  my-super-workflow:
    pipeline: my-super-pipeline
    application: my-super-application
    mutexes: # or `locks`, or `one_at_a_time`
      - name: "GRA7-Openstack-API"
        amount_available: 5
      - name: "SBG5-Openstack-API"
        amount_available: 10 # smaller region means less API overload
...
---
# Pipeline YML
# ...
    steps:
      - checkout: '{{.cds.workspace}}'
      - mutex_take: 
          name: "GRA7-Openstack-API"
          amount: 2 # The next step executes 2 parallel requests to the API
      - script: echo "doing something at GRA7"
      - mutex_release: 
          name: "GRA7-Openstack-API"
          amount: 2 # Might be inferred from the previous `mutex_take`
...