output spans multiple directories

[sterlingbaldwin]

When performing a CMOR conversion late at night, if the processing starts before midnight, and finishes after midnight, the output will be placed in multiple directories. This is somewhat annoying, as additional manual steps must be performed to put all the files into one directory. This can be especially burdensome when processing a large number of datasets and adds a non-trivial amount of manual work while accomplishing nothing of value. It would be better if CMOR put all the output files into the same directory, even when the processing took place over >1 days.

[durack1]

FYI @sterlingbaldwin https://github.com/PCMDI/cmor/issues/210 (also https://github.com/PCMDI/cmor/issues/267), and specifically https://github.com/PCMDI/cmor/issues/210#issuecomment-317893707

[sterlingbaldwin]

Thanks Paul, using "_myversion":"v20170725" in the metadata file is an acceptable workaround, however I dont see the benefit of having the default behavior to be placing the output in multiple directories. It would be nice to see this functionality documented somewhere, since comments on years old closed issues perhaps arent the easiest way to disseminate information. Are there more hidden _ prefixed config values that change CMORs behavior?

[durack1]

@sterlingbaldwin the benefit is mostly for the archive itself, as if we exposed these hard coding options to most users, they would set their version once and all subsequent files would be written with a version of v20180701 or equivalent. By having the default pull the current date, such an issue is avoided for the most part.

I also note that quite a number of the larger centres were using CMOR in parallel, and so trying to manage all of this becomes infinitely harder as your use cases start to multiply

[durack1]

Closing this issue, as it's stale - but we should consider this use case in the CMOR4 planning