if [WIP] Build `if-check`

Sub of: #629 -> #650

What

Build a command line tool called if-check that wraps calls to if-env and if-diff. On the command line, a user can call if-check, passing an output file, and the tool will install the necessary dependencies as defined by if-env and then if-diff the provided output file against a newly generated version obtained by executing the context and tree sections from the given output file as if it were a fresh manifest.

Why

As a user I want to be able to re-execute a manifest and verify that the outputs are correctly reported. if-check is a convenience tool that wraps the individual steps required to verify a manifest into a single command.

The point of this is to verify that a given output file was correctly reported

Say someone gives you a manifest file to demonstrate the environmental impact of their application. Maybe you don't trust them completely, or are suspicious they've made a mistake. You can just pass their manifest to if-check and it will run it back through IF and report whether the given data is accurate or not - if not, if-check will report where there are differences.

In terms of the implementation, if-check should do the following under the hood:

run if-env on the given manifest (which we can refer to as manifest.yml which will recreate the original runtime environment reported in the manifest locally
run the manifest and save the output to a temporary file (here we can refer to it as re-executed.yml)
run if-diff with if-diff --source manifest --target re-executed.yml
report the output from if-diff to the console

This means the if-check CLI tool can be an alias to:

if-env && ie given-file.yml --stdout | if-diff --target given-file.yml

The result should be an if-diff output, for example:

Files match!

or

Files do not match!
tree.children.child.inputs[0]['cpu/energy']
source: 50
target: 30

Acceptance Criteria

A command line tool, if-check exists that takes an output file as input, and returns the output from if-diff
- The command line tool if-check should take a single argument, --file whose value should be a path to an output file.
- IThe if-env tool can be run on the output file, given information in it's execution block. The output file can then be used as an input to ie, to re-run the given it in precisely the same environment (at least at the level of package.json, not necessarily at the Node or OS level).
- The given output file and the file generated by rerunning the manifest should be compared. The if-check command should yield precisely the same output as running if-env and if-diff separately on a file.
GIVEN a user has downloaded and installed if
WHEN they run if-check passing a valid output file
THEN they receive the if-diff output comparing their given output file with a new one generated by re-executing the given one

e.g. the following output file (example-file.yml) is provided:


name: sum
description: successful path
tags: null
initialize:
  plugins:
    sum:
      path: '@grnsft/if-plugins'
      method: Sum
      global-config:
        input-parameters:
          - cpu/energy
          - network/energy
        output-parameter: energy
  outputs:
    - yaml
execution:
  command: >-
    /home/user/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/user/Code/if/src/index.ts -m manifests/outputs/test.yaml -o
    manifests/outputs/test2.yml
  environment:
    if-version: v0.3.2
    os: linux
    os-version: 5.15.0-105-generic
    node-version: 21.4.0
    date-time: 2024-05-15T10:28:36.414Z (UTC)
    dependencies:
      - '@babel/[email protected]'
      - '@babel/[email protected]'
      - '@commitlint/[email protected]'
      - '@commitlint/[email protected]'
      - '@grnsft/[email protected]'
      - '@grnsft/[email protected]'
      - '@jest/[email protected]'
      - '@types/[email protected]'
      - '@types/[email protected]'
      - '@types/[email protected]'
      - '@types/[email protected]'
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum: null
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
          energy: 0.002

The following command is executed:

if-check example-file.yml

The following response is expected:

Files match!

The following output file has a deliberate error introduced in the outputs/energy field.


name: sum
description: successful path
tags: null
initialize:
  plugins:
    sum:
      path: '@grnsft/if-plugins'
      method: Sum
      global-config:
        input-parameters:
          - cpu/energy
          - network/energy
        output-parameter: energy
  outputs:
    - yaml
execution:
  command: >-
    /home/user/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/user/Code/if/src/index.ts -m manifests/outputs/test.yaml -o
    manifests/outputs/test2.yml
  environment:
    if-version: v0.3.2
    os: linux
    os-version: 5.15.0-105-generic
    node-version: 21.4.0
    date-time: 2024-05-15T10:28:36.414Z (UTC)
    dependencies:
      - '@babel/[email protected]'
      - '@babel/[email protected]'
      - '@commitlint/[email protected]'
      - '@commitlint/[email protected]'
      - '@grnsft/[email protected]'
      - '@grnsft/[email protected]'
      - '@jest/[email protected]'
      - '@types/[email protected]'
      - '@types/[email protected]'
      - '@types/[email protected]'
      - '@types/[email protected]'
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum: null
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
          energy: 0.0005

Running if-check example-file-2.yml should yield:

File do not match!
tree.children.child.intputs[0]['energy']
source: 0.002
target: 0.0005

A set of identical and non-identical manifests exist as a test set, whose if-check outcome we know a priori and can be used to verify that if-check is operating correctly.
- This might be satisfied naturally as part of Build if-diff core feature #637
GIVEN the test set exists
WHEN a user runs if-check on those test manifests
THEN the if-check output matches the expected outputs
Documentation exists in the form of a README in the IF repository and a section on if.greensoftware.foundation/docs/reference/cli
GIVEN the documentation has been added to if-greensoftware.foundation
WHEN a user visits the site and searches for if-check
THEN they will find a dedicated page explaining the tool, with usage examples.

Prerequisites

Apr 11 '24 08:04 jmcook1186

@jmcook1186 I saw this mentioned in the #615 and got the impression that we expect the dev work to start on this fairly soon, however, there is no acceptance criteria, is this something that should be brought into this week for designing / refining? 🙏

Apr 23 '24 20:04 zanete

Yes this needs design work! @zanete

Apr 24 '24 09:04 jmcook1186

Updated Acceptance criteria and description following discussion in IF weekly call

Apr 25 '24 12:04 jmcook1186

@jawache please take a look at the latest AC to move it along

Apr 25 '24 15:04 zanete

@jawache please check over the AC and then we can move it to post #636 and #637 as those are blocking

Apr 29 '24 10:04 zanete

@jawache please check over the AC and then we can move it to post https://github.com/Green-Software-Foundation/if/issues/636 and https://github.com/Green-Software-Foundation/if/issues/637 as those are blocking

May 09 '24 15:05 zanete

@zanete and @jmcook1186 same feedback as the other tickets, can you make the AC with actual manifest files and command lines and actual output/console messages!

Roughly reading then it's on the right lines but there is so much that is left to assumptions at this point. It's a cli tool so the ACs should run this command line, with these exact params and files, and this should be the exact output.

May 14 '24 20:05 jawache

@jawache issue is updated with manifests and more specific acceptance criteria - appreciate your sign-off before this gets worked on next week

May 20 '24 10:05 jmcook1186

Thanks @jmcook1186 looks good however I think we need to think about the context of (a) running a folder of files (b) how this integrates into ci/cd since one ultimate goal is to use this as our test infrastructure for changes to code.

PR Runs if-check on folder of manifests If it passes all good If one fails then should return fail and the logs show which one failed.

Not sure if to meet the above if-check needs functionality OR we can just create a bash script that runs if-check for us in a loop.

If it's bash script I think if-check needs to return, pass through, the return error code from if diff.

May 23 '24 21:05 jawache

I think it would be nicer to handle multiple files inside if-check rather than relying on bash scripts or other 'external' solutions. I added some description to the ticket related to this.

May 29 '24 13:05 jmcook1186

@jmcook1186, can we change the flags? The -m flag would refer to the manifest file, and the -f ( or -d) to the folder (either --directory or --dir). I think this will be more relevant to our CLI commands. What do you think?

Jun 20 '24 06:06 manushak

Hi @manushak - yes, fine!

Jun 20 '24 08:06 jmcook1186

there could be a PR for this tomorrow 💪

Jun 20 '24 15:06 zanete

if if copied to clipboard

[WIP] Build `if-check`

if
if copied to clipboard