metadata metadata

[cshamis]

Open for discussion: We may have a problem with how we're handling file metadata in SCALE/SEED. The input_file_manifest.json includes all the file metatdata scale knows about the input files, but ... in the case of a mounted S3 workspace, it cannot be written and will cause an error. Moreover ALL input file metadata records are stored in that ONE manifest file. Pushing parse-results back into the scale file records requires writing a metadata.json in the output directory using a different JSON format.

What we really want:

• We need a way for a job to READ input file metadata.
• We need a way for a job to UPDATE input file metadata.
• We need a way for a job to CREATE output file metadata when it creates a new output file.
• It would be desirable for this mechanism to be the SAME for all three.

The proposal:

• All metadata is provided via file objects and follows the {filename}.metadata.json naming convention.
• json format will include everything in the scale_file record, but only certain fields can be modified. (certain scale-internals will not be changeable: i.e. filesize, filename, creation time, transfer time, permanlink, strike_id, workspace_id, etc.)
• Input files go into top level container directory /scale/input as they do now.
• Input metadata files will go into a newly created toplevel directory in the container called "/scale/metadata" (solves the unreliable write ability of the workspace problem)
• Output files go into top level container directory /scale/output as they do now.
• Output metatdata files will go into the /scale/output directory as required by SEED spec.

Right now, I don't think we have a way for a parse job to update the scale file record at all.

I'm leaning towards this being necessary for 7.0.x ; god help me.

[gisjedi]

This is related to #1289, but is a superset of that issue. This is something we should keep as context for future designs, but is out of scope for the immediate need. #1289 will address the immediate need.