nextflow icon indicating copy to clipboard operation
nextflow copied to clipboard

Refactor lineage JSON serialization with a root element including version, kind and spec properties

Open jorgee opened this issue 6 months ago • 4 comments

This PR refactors the Lineage data encoder to serialize lineage records using the following structure:

{
  "version": <lineage version>,
  "kind": <lineage record class>,
  "spec": <record description>
}

For instance a FileOutput type is serialized as:

{
  "version": "lineage/v1beta1",
  "kind": "FileOutput",
  "spec": {
    "path": "/home/jorgee/nextflow_tests/provenance-test/results/samples.json",
    "checksum": {
      "value": "30a926dd315d4f4b9d6389803a1324ff",
      "algorithm": "nextflow",
      "mode": "standard"
    },
    "source": "lid://31e092e42c71b9545a9c5fa91689340c",
    "workflowRun": "lid://31e092e42c71b9545a9c5fa91689340c",
    "taskRun": null,
    "size": 201,
    "createdAt": "2025-05-14T10:25:51.563182873+02:00",
    "modifiedAt": "2025-05-14T10:25:51.563182873+02:00",
    "labels": [
      "foo"
    ]
  }
}

jorgee avatar May 14 '25 08:05 jorgee

Deploy Preview for nextflow-docs-staging canceled.

Name Link
Latest commit 7584275932ef4f13740fc24257a3eecd0f001ab5
Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/686d69e148599b00085dc74e

netlify[bot] avatar May 14 '25 08:05 netlify[bot]

This looks nice, likely we needed to update the version to v1beta2 if we go ahead with this. Also in a ideal world, it should be possible to keep backward compatibility with previous spec version, but I can understand that cannot be that simple

pditommaso avatar May 14 '25 18:05 pditommaso

For these PR, as there aren't changes in the model I think there is no need to update the version. I can make able to read both with spec and without with a small change. However, with changes such as in #6069 a mechanism for version compatibility is required.

jorgee avatar May 15 '25 08:05 jorgee

It might be enough to simply ignore records that have an incompatible version and warn about them so that the user is aware. At least while the version is in beta. Maybe in the future we could add backwards compatibility, e.g. v1 / v2 data models, but hopefully we can iron out any changes in the beta period

bentsherman avatar May 15 '25 11:05 bentsherman

I've added the backward compatibility with the previous lineage format and test. Please review and merge if OK.

Claude prompt for the posterity:

This branch changes the JSON structure of the lineage model, see the changes of LinTypeAdapterFactory comparing with the master branch. Howevever I'd to keep the read compatibility with the previous version. Can you make a patch for it? When done write a unit test to validate it.

pditommaso avatar Jul 08 '25 18:07 pditommaso

Changes are fine with me.

jorgee avatar Jul 09 '25 05:07 jorgee