How do nodejs-polars track/maintain feature-parity with py-polars?

[sorhawell]

Dear @universalmind303

I work on implementing polars for R and very much see nodejs-polars as the example to follow. Especially when using rust-polars core features, not in the public polars crate, I found the solutions in nodejs-polars.

My unit tests likely will catch implementation bugs, but whenever py-polars makes a behavior change, it requires I manually notice this and update/add such behavior too. I was asked what is lifecycle policy of rpolars, and I'm very interested to learn of your thoughts and experiences of maintaining nodejs-polars along the main projects rust-polars and py-polars.

best

[sorhawell]

related issue "Add information in docs regarding status/maturity/future?" #6

[ritchie46]

@sorhawell if you plan to keep API and behavioral parity as close as possible to python polars would you consider moving the repo under the pola-rs namespace?

This likely will help discovery and attraction help by the community.

[universalmind303]

As far as feature parity goes. Ideally i'd like to write some scripts that generate a coverage map between node & python features. We know some features will be much more difficult to support in JS (if not impossible). (lambdas, and interop methods [pandas, arrow]). But until we have some automated way of tracking this, it is a manual task to update the features to match py-polars.

[sorhawell]

@ritchie46 Thanks, that would be super cool and potentially very helpful :) !

[sorhawell]

@universalmind303 Yeah maybe such a script would be a good start. I could track what python/rust functions in py-polars directory has been altered between releases. That might be a good checklist functions I should eyeball for changed behavior before bumping the polars dependency version. I try also moving towards similar file structure and file content order as py-polars for the sake of sanity.

[sorhawell]

Lambdas thread-safety was difficult for me to implement in r-polars as R is very thread unsafe and has no GIL concept. I keep the main thread to communicate with R and handle all incomming lambdas-requests from the polars threads(pool). If lambda-request buffer is empty the R thread just sleeps or occasionally checks for user-interrupts. That way all R lambdas can happily stay single-threaded and co-exists with multi-threaded polars. Maybe the nodejs lambdas challange is something completely different :)