linopy icon indicating copy to clipboard operation
linopy copied to clipboard

Support for sparse xarrays

Open staadecker opened this issue 4 months ago • 23 comments

First of, great work with the library so far!! This is significantly better than what currently exists out there.

Many energy systems models can be extremely sparse. For example, in one recent case, a linear expression is composed of the sum of only 0.1% of the coordinates of an xarray. Not supporting sparsity hence would increase our model size by ~900x. This effectively prohibits us from using Linopy as is.

Suggested solution

Ideally linopy could support xarrays based on the sparse library. I've tried the library but it doesn't work with linopy particularly when trying to combine sparse and not sparse xarrays.

Rejected solutions

  1. I tried using MultiIndices which are natively supported in xarray however broadcasting doesn't work when combining a sub-dimension of a multi-index. In general, my gut tells me this is the wrong direction to take for the library.

  2. I'm aware of the mask parameter for creating constraints and variables however this a) still requires a large memory allocation and b) doesn't help during the computation of linear expressions (it only helps at the variable/constraint creation stage).

staadecker avatar Mar 07 '24 05:03 staadecker