Upgrade PyO3 to 0.21 and switch to new API

[itamarst]

Description

In PyO3 0.21, &'py PyAny will be replaced by Bound<'py, PyAny>. The benefits for upgrading are increased performance, the costs of not upgrading are that the old API is deprecated and will, eventually, go away: https://polar.sh/davidhewitt/posts/replacing-pyo3-api-pt1

https://pyo3.rs/latest/migration documents the upgrade process (at time of writing 0.21 is still beta, but will presumably be released soon). Based on their suggestions, I think this would look like this for Polars:

1. First PR: Switch to 0.21, enable gil-refs feature, fix other migration issues.
2. A series of PRs that migrate over to Bound. The development process would involve disabling gil-refs temporarily, locally only, to spot APIs that need migration, but given this would probably be a series of PRs, the feature should continue to be on in main.
3. Final PR: When no deprecation warnings are left, disable gil-refs.

• [x] #15621
• [x] #15666
• [x] #15669
• [x] #15785
• [x] #15864
• [x] #15890
• [ ] More upgrading to new PyO3 0.21 APIs (may involve multiple tickets)
• [ ] Switch to native abi3 datetime support, see below explanation from @stinodego

[itamarst]

This is something I can help work on.

[stinodego]

Yes we will definitely upgrade to the new PyO3 version when it's released. I had planned to work on that myself but if it's something you're interested in working on I think it's fine if you want to do the honors here.

[itamarst]

0.21.1 is out, so I will start on this either this week or next.

[stinodego]

Something about support for chrono? Need to get reminder from @stinodego

PyO3 has support for the Python datetime module: https://pyo3.rs/main/doc/pyo3/types/struct.pydatetime

However, this was never available under the abi3 feature flag.

So we have a bunch of workarounds in place to materialize/read datetime objects. Specially, we have defined Python functions here: https://github.com/pola-rs/polars/blob/main/py-polars/polars/_utils/convert.py

...that are called from Rust, for example here: https://github.com/pola-rs/polars/blob/8ef2e212cf52c318e18d12d38b02c3ad0a918ab2/py-polars/src/conversion/any_value.rs#L61-L64

The 0.21 release notes mention:

Extended chrono / datetime conversions, including support for the abi3 feature and the chrono-tz crate

So I am expecting that we can now take advantage of native PyO3 datetime support. And I expect a good performance boost + code cleanup to come from this.

[itamarst]

I'm gonna say this is done, insofar as it covers the main Polars repository, and the pyo3-polars issue is already filed for that repository.

[stinodego]

Thanks a lot for the thorough work @itamarst !