Spine-Toolbox
Spine-Toolbox copied to clipboard
spine-tools
Define result directory
It would be great to be able to define one's own result-directory. The idea to have folders by scenario and date of model run is great, but since the date and time are automatically determined, it is impossible to automatize data processing (I am using my own scripts to analyze results data and currently have to add any result-file path manually).
Would it be possible to add your result data analysis scripts as a Tool to the workflow? You can pass the correct file paths to the scripts as command line arguments that can be set from the Tool properties tab. Or you could just add a Tool that copies the results to a known final directory.
I can try to add my scripts. We also have a loop pushing the data around integrated in our workflow anyway, so I can also use that as a first step. It is just that this would create the double amount of data with each iteration.
It is just that this would create the double amount of data with each iteration.
I'm not sure how you've setup the workflow, but perhaps a cleanup step could be added somewhere if disk space is of concern?
A heads-up: I'm currently working on a new project item that will let one to copy the output files of previous item to a fixed destination directory.
Hi @soininen interesting. You know I'm not so fond of new project items but do we really need this new one?
You know I'm not so fond of new project items but do we really need this new one?
I see this type of project item as a nice "terminus" for a workflow. Otherwise, we need to implement an output directory setting to other project items (Tool and Exporter, at least) which I'm not so fond of. I think a separate project item is more general solution.
I'd like to have only necessary tools. I don't quite understand the use case here. When @SilkeJo runs the workflow, it could presumably have multiple scenarios. If you save them in the same folder, then you'll overwrite (unless the filename gets a timestamp or something). Would that be the idea? To have multiple files in the folder that are then all processed by the same post-processing tool?
How does the 'terminus' tool work with an exporter? Catches all scenarios and writes them into separate files in one directory instead of separate directories?
Thanks for the explanation @soininen - it sounds actually pretty reasonable. I agree with @jkiviluo too that we need to make sure that we need it...
If you save them in the same folder, then you'll overwrite (unless the filename gets a timestamp or something). Would that be the idea? To have multiple files in the folder that are then all processed by the same post-processing tool?
The idea was to keep it simple: copy all files to the hard-coded folder overwriting the previous ones which would always give the latest output only. The historical files would still be available in Tool's or Exporter's output directory (in Exporter you'd need to check the "add time stamps" option which you'd usually do for debugging purposes). This functionality has been requested in the past a few times.
How does the 'terminus' tool work with an exporter? Catches all scenarios and writes them into separate files in one directory instead of separate directories?
This is actually not only an issue with Exporter, but also with Tools that produce output files. I really don't see any other solution here than to create separate directories within the fixed directory as you cannot write the same files in parallel - you would never know which filter created which file! However, the more I think about it, the more I feel it beats the purpose of the item.
I think I'll put this item on hold for now since I feel it needs more thinking.
I suppose you could timestamp/scenario-name output files, but it can become unwieldy with exporters/tools that produce many files per scenario.