cubed
cubed copied to clipboard
Calculating the cost of a computation
It would be useful to provide numbers for actual resources used after a computation is complete so its cost can be calculated.
We probably need:
- total worker seconds
- total storage
We can calculate 1 from the events dataframe created by HistoryCallback, since we know how long each task ran for.
For 2, we can measure the storage used by all the intermediate Zarr arrays in the working directory. This doesn't have a time component, but in computing the cost we could assume that all intermediate storage is deleted at the end of the computation (say).
cc @TomNicholas
See my email to you just now!