nextflow
nextflow copied to clipboard
Refactor lineage JSON serialization with a root element including version, kind and spec properties
This PR refactors the Lineage data encoder to serialize lineage records using the following structure:
{
"version": <lineage version>,
"kind": <lineage record class>,
"spec": <record description>
}
For instance a FileOutput type is serialized as:
{
"version": "lineage/v1beta1",
"kind": "FileOutput",
"spec": {
"path": "/home/jorgee/nextflow_tests/provenance-test/results/samples.json",
"checksum": {
"value": "30a926dd315d4f4b9d6389803a1324ff",
"algorithm": "nextflow",
"mode": "standard"
},
"source": "lid://31e092e42c71b9545a9c5fa91689340c",
"workflowRun": "lid://31e092e42c71b9545a9c5fa91689340c",
"taskRun": null,
"size": 201,
"createdAt": "2025-05-14T10:25:51.563182873+02:00",
"modifiedAt": "2025-05-14T10:25:51.563182873+02:00",
"labels": [
"foo"
]
}
}
Deploy Preview for nextflow-docs-staging canceled.
| Name | Link |
|---|---|
| Latest commit | 7584275932ef4f13740fc24257a3eecd0f001ab5 |
| Latest deploy log | https://app.netlify.com/projects/nextflow-docs-staging/deploys/686d69e148599b00085dc74e |
This looks nice, likely we needed to update the version to v1beta2 if we go ahead with this. Also in a ideal world, it should be possible to keep backward compatibility with previous spec version, but I can understand that cannot be that simple
For these PR, as there aren't changes in the model I think there is no need to update the version. I can make able to read both with spec and without with a small change. However, with changes such as in #6069 a mechanism for version compatibility is required.
It might be enough to simply ignore records that have an incompatible version and warn about them so that the user is aware. At least while the version is in beta. Maybe in the future we could add backwards compatibility, e.g. v1 / v2 data models, but hopefully we can iron out any changes in the beta period
I've added the backward compatibility with the previous lineage format and test. Please review and merge if OK.
Claude prompt for the posterity:
This branch changes the JSON structure of the lineage model, see the changes of
LinTypeAdapterFactorycomparing with the master branch. Howevever I'd to keep the read compatibility with the previous version. Can you make a patch for it? When done write a unit test to validate it.
Changes are fine with me.