xarray-sql 100 Trillion Row Challenge

100 Trillion Row Challenge

Open alxmrs opened this issue 1 year ago • 3 comments

How fast could we calculate the min, max, avg temperature for every atmospheric level of ARCO-ERA5?

https://www.morling.dev/blog/one-billion-row-challenge/

https://medium.com/coiled-hq/one-trillion-row-challenge-5bfd4c3b8aef

Mar 04 '24 11:03 alxmrs

AKA what's the fastest way to calculate a climatology?

Mar 05 '24 05:03 alxmrs

I easily get confused by order of magnitude. The number of rows is the product of the dimensions of ARCO-ERA5, which I calculate as:

= 732,072 hours * 720 lat * 1440 lng * 37 lvl = 2.8083453e+13 = 28 Trillion

Mar 05 '24 06:03 alxmrs

The latest version of ARCO-ERA5 supports all 137 model levels! Thus, the calculation would be:

732,072 hours * 720 lat * 1440 lng * 137 lvl

= 1.0398468e+14 = ~100 Trillion

Oct 10 '25 10:10 alxmrs