rateslib icon indicating copy to clipboard operation
rateslib copied to clipboard

DEP: change `pandas` to `polars` for package size

Open attack68 opened this issue 1 year ago • 4 comments
trafficstars

pandas is a more fully featured library, but 99% of those features are not used by rateslib. polars is more efficient for the use cases but suffers from portability. Until polars has the flexibility of being installed in pyodide this can remain onhold.

attack68 avatar Nov 11 '24 12:11 attack68

0.27 Pyodide has added Polars.

attack68 avatar Jan 07 '25 12:01 attack68

@attack68 Activating this issue. Now that there is a Polars build for Pyodide, would you see a prospect to move onwards with this matter? I'd gladly offer my help.

ToukoH avatar May 24 '25 22:05 ToukoH

The hardest part of this are two things:

  • maintaining the ability to handle pandas Series inputs (does narwhals allow this?)
  • replacing the code that sequentially allows the construction of a risk dataframe using keys for a pandas.MultiIndex. I am not familiar enough with polars to know if this is easily doable.

attack68 avatar May 26 '25 21:05 attack68

  • Narwhals is a pretty unknown framework for me, but from a quick look, I wouldn’t be surprised if it did. Also, building custom adapters should not be that hard of a task.

  • Assuming you are referring to the Solver module, this migration could get pretty hairy. Polars does not natively support multi-indexing (see https://docs.pola.rs/user-guide/migration/pandas) so that option is off the table. Correct me if I'm wrong, but it looks like multi-indices are predominantly used for formatting results in a readable way and not in any performance critical operations, although it enables quite straight forward retrieval as well. I really don’t have a clear proposal for a more efficient approach that leverages Polars in this regard.

That being said, any performance critical sections currently relying on Pandas (if those exist) could serve as a good starting point for experimenting with how Polars might integrate to Rateslib.

Edit: If the sole purpose of the migration is to shrink the package size, it may not justify the effort.

ToukoH avatar May 27 '25 09:05 ToukoH