cubed icon indicating copy to clipboard operation
cubed copied to clipboard

Calculating the cost of a computation

Open tomwhite opened this issue 2 years ago • 2 comments

It would be useful to provide numbers for actual resources used after a computation is complete so its cost can be calculated.

We probably need:

  1. total worker seconds
  2. total storage

We can calculate 1 from the events dataframe created by HistoryCallback, since we know how long each task ran for.

For 2, we can measure the storage used by all the intermediate Zarr arrays in the working directory. This doesn't have a time component, but in computing the cost we could assume that all intermediate storage is deleted at the end of the computation (say).

cc @TomNicholas

tomwhite avatar Jun 21 '23 14:06 tomwhite

See my email to you just now!

TomNicholas avatar Jun 21 '23 14:06 TomNicholas

There's more detail in the blog post and notebook.

tomwhite avatar Jul 19 '23 16:07 tomwhite