hide injecting several input attributes into behind profile

[bernt-matthias]

trafficstars

There seem to be better ways to access metadata like dbkey. Also seems error prone since only the first data set is used.

As far as I seen these features are also undocumented.

Wondering if at IUC we should "enforce" more recent profiles.

[jmchilton]

I've done so much work to preserve those buggy things. I like this idea a lot. I wonder what the deal is is with the |__identifier attributes - any chance you blamed it back to a source commit where it was added?

[bernt-matthias]

I wonder what the deal is is with the |__identifier attributes

I was wondering as well

any chance you blamed it back to a source commit where it was added?

I tried and stopped 14years in the past .. let my try harder :)

[bernt-matthias]

I guess over here https://github.com/galaxyproject/galaxy/commit/7bfc238304ac5e46abac83ff8eb222011ffe9dbe .. https://github.com/galaxyproject/galaxy/blob/7bfc238304ac5e46abac83ff8eb222011ffe9dbe/lib/galaxy/tools/actions/init.py#L44

The |__identifier stuff seems to stem from this commit https://github.com/galaxyproject/galaxy/commit/2f15eb0d788eccf1406e548f367a72a0c9d20fb1

[bernt-matthias]

Wondering if the code that I left in the else branch can be "removed"...

[bernt-matthias]

So tests are failing... Surprise! :)

First I realized that input_ext and input_dbkey are used later in the same function (one of them as global variable to a local function ... urgs). Nevertheless I did a bit of research (aka guessing) on how they are used and if they need to be stored in incoming.

input_ext is determined in DefaultToolAction.execute() and then passed passed to determine_output_format() where a variable ext is initialized depending on the format attribute of the output (output.format), i.e. ext=input_ext if output.format is "input" and ext=output.format otherwise

The profile versions differ in the initialization of input_ext

• <16.04 input_ext is initialized as 'data' and overwritten by the extension of the last input (which might be a random input .. nor sure if there is a guaranteed order, source code comments indicate that its random)

• = 16.04 input_ext is initialised as 'input'

So, if output.format is "input" in

• <16.04 ext becomes the extension of an input

• =16.04 ext becomes "input"

So I tried to find where the string "input" / 'input' also appears and might be used.

This seems to be the case in collect_primary_datasets in lib/galaxy/job/execution/output_collect.py which has a parameter "input_ext" and which has a very similar code block

          if ext == "input":
              ext = input_ext

This function is called by

• set_metadata_portable in lib/galaxy/metadata/set_metadata.py

  input_ext = json.loads(metadata_params["job_params"].get("__input_ext", '"data"'))

• and essentially discover_outputs in lib/galaxy/jobs/init.py

  input_ext = input_params.get("__input_ext") if input_ext is not None input_ext = json.loads(input_ext) else input ext init with "data" and overwritten with ext of last input

At adjacent code positions one also finds references to "dbkey". So it seems that we cant simply remove storing "dbkey" and alike in the incoming dict(?) that seems to be produced in DefaultToolAction.execute().

In summary it appears to me that:

• the variables seem to be computed on execution and stored in a way that I do not understand yet
• the values are reused when processing outputs

I guess the two variables are just the beginning... So I'm considering to not follow this path .. :(

Ideas? .. Do you see small step(s) how we could improve this?

[mvdbeek]

This looks like it'd be a good bit of work to wrap up, OK to move to 21.05 ?

[bernt-matthias]

OK to move to 21.05 ?

At least :) ..

[jmchilton]

Moving to 21.09 also 😅, please ping again though if it wrapped up sooner. This is good and important work.