Simplification of run:* tasks

[hirenr]

trafficstars

What would you like to be added?

Extending the discussion from Slack Discussion

Update the run:* syntax to have a unified structure.

• Env Variables
• Arguments (as string[]) passed as CLI args to run:shell / run:container / run:script underlying code command
• Transformed Task Input passed as STDIN via a new stdin property to the run:shell / run:container / run:script underlying code!
• Transformed Task input passed as Input Object to run:workflow

Proposal(s):

The advantages to the following are:

• No injection of code into scripts (for input args etc) and other underlying code!
• Makes the inline/referenced scripts fully portable without relying on the runtime
• Execution and passing of data to scripts similiar, with how it would work outside of a workflow environment. For eg. when running the script directly via node command.
• Input (via STDIN) and Output (via STDOUT, STDERR, CODE) more inline with each other.

Below is an example run:script, but other run:* would be similar!

e.g run:script

document:
  dsl: 1.0.0-alpha1
  namespace: examples
  name: call-script-input-type-example
  version: 1.0.0-alpha1
do:
  - setInput:
      set:
        url: https://petstore.swagger.io/v2/pet/2
  - runScript:
      output: 
        as: "${ fromjson }"
      run:
        script:
          language: javascript
          stdin: ${ . }
          environment:
           url: https://petstore.swagger.io/v2/pet/2
          arguments:
          - https://petstore.swagger.io/v2/pet/2
          source:
            endpoint: file:///scripts/hello_world.js

/scripts/hello_world.js

import axios from 'axios';

// Reading Input from STDIN
const input = await new Promise((resolve) => {
  let data = '';
  process.stdin.setEncoding('utf8');
  process.stdin.on('data', chunk => data += chunk);
  process.stdin.on('end', () => {
    try {
      resolve(JSON.parse(data));
    } catch (error) {
      resolve({})
    }
  });
});
const responseStdin = await axios(input.url);

// Reading from argv
const [_, __, url] = process.argv;
const responseArgv = await axios(url);

// Reading from env
const envUrl = process.env.url;
const responseEnv = await axios(envUrl);

console.log(JSON.stringify({responseStdin: responseStdin.data, responseArgv: responseArgv.data, responseEnv: responseEnv.data }))

Alternative(s):

No response

Additional info:

Why?

• Current argv/env variables are limited by the underlying OS
• Cannot pass large data via args/env
• STDIN has no such limits
• Inserting named parameters needs code injection. This goes against the ethos of a declarative workflow. Ideally, runtime should not touch the code.

Community Notes

• Please vote by adding a 👍 reaction to the feature to help us prioritize.
• If you are interested to work on this feature, please leave a comment.

[hirenr]

Eg for containers:

document:
  dsl: 1.0.0-alpha1
  namespace: examples
  name: call-script-input-type-example
  version: 1.0.0-alpha1
do:
  - setInput:
      set:
        message: Hello World
  - runContainer:
      output: 
        as: "${ fromjson }"
      input:
        as: ${ .message }
      stdin: ${ . }
      run:
        container:
          image: alpine
          command: 'cat'

Output would be Hello World

Eg for containers with args

document:
  dsl: 1.0.0
  namespace: examples
  name: call-script-input-type-example
  version: 1.0.0-alpha1
do:
  - setInput:
      set:
        message: Hello World
  - runContainer:
      input:
        from: ${ .message }
      run:
        container:
          image: alpine
          command: 'echo "Arg1: $1, Arg2: $2"'
          arguments:
          - Hello
          - World

[hirenr]

Shell Example

document:
  dsl: 1.0.0
  namespace: examples
  name: call-script-input-type-example
  version: 1.0.0-alpha1
do:
  - setInput:
      set:
        message: Hello World
  - runShell:
      input:
        from: ${ .message }
      run:
        shell:
          stdin: ${ . }
          command: |
            input=$(cat)
            echo "STDIN was: $input"
            echo "ARGS are $1 $2"
          arguments:
          - Hello
          - World

[hirenr]

Updated the recommendation to add an stdin property to the run:* schema! Using transformed input may not be the best idea. I may want to use a property from transformed input in my args.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ricardozanini]

@cdavernas what do you think about this feature? Can we move on? I can open the PR if @hirenr doesn't mind (or they can open it too - more contribs are welcome).

[cdavernas]

@ricardozanini It's a great proposal afaik. The args, however, cannot be an array of strings for subflows, which require a map. Other than that, I think we can proceed

[hirenr]

@ricardozanini id love to contribute! i will create a PR tomorrow.

@cdavernas run:workflow should not have the arg structure. I feel for that the current structure should suffice. wdyt!