feat(bindings/python): Add from_uri constructors for Operators
Which issue does this PR close?
Closes #6984 .
Rationale for this change
Instantiating Operators from URLs greatly increases flexibility for consumers of the openDAL API as an abstract backend.
What changes are included in this PR?
Add from_uri static methods to the Operator and AsyncOperator classes in the python bindings.
Are there any user-facing changes?
Two new methods are available to users of the python library.
AI Usage Statement
None
Other notes
I am having issues with my development environment which the devcontainer is not making any easier, so I'm flying a bit blind and relying on CI.
Tests pass but I'm not clear how best to test the new functionality as it would mean parsing the config provided by the setup_config fixture and then constructing a URI out of it, which is going to be different for each service.
Thank you for working on this! I will invite @chitralverma to take a review on this.
@clbarnes can you run stubgen with just to get the python stubs as well?
I found that running stubgen locally drops a bunch of services and deletes a lot of generated code as well - possibly all the services which have been split out of core into their own crates?
--- a/bindings/python/src/services.rs
+++ b/bindings/python/src/services.rs
@@ -62,16 +62,6 @@ submit! {
#[pyo3(rename_all = "PascalCase")]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum PyScheme {
- #[cfg(feature = "services-aliyun-drive")]
- AliyunDrive,
- #[cfg(feature = "services-alluxio")]
- Alluxio,
- #[cfg(feature = "services-azblob")]
- Azblob,
- #[cfg(feature = "services-azdls")]
- Azdls,
- #[cfg(feature = "services-azfile")]
- Azfile,
#[cfg(feature = "services-b2")]
B2,
#[cfg(feature = "services-cacache")]
@@ -84,24 +74,14 @@ pub enum PyScheme {
Dropbox,
#[cfg(feature = "services-fs")]
Fs,
- #[cfg(feature = "services-ftp")]
- Ftp,
- #[cfg(feature = "services-gcs")]
- Gcs,
#[cfg(feature = "services-gdrive")]
Gdrive,
- #[cfg(feature = "services-ghac")]
- Ghac,
#[cfg(feature = "services-gridfs")]
Gridfs,
- #[cfg(feature = "services-hdfs-native")]
- HdfsNative,
#[cfg(feature = "services-http")]
Http,
#[cfg(feature = "services-huggingface")]
Huggingface,
- #[cfg(feature = "services-ipfs")]
- Ipfs,
#[cfg(feature = "services-ipmfs")]
Ipmfs,
#[cfg(feature = "services-koofr")]
etc.. That comes from the cargo run --quiet --manifest-path=../../dev/Cargo.toml -- generate -l python step.
The second step doesn't finish successfully - cargo run --quiet --bin stub_gen.
opendal/bindings/python on py-from-uri [$!⇡] is 📦 v0.47.0 via 🐍 v3.14.0 (opendal) via 🦀 v1.91.0 took 1m3s
❯ cargo run --quiet --bin stub_gen
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00000001f16b6080 [main] (most recent call first):
<no Python frame>
but python -c 'import encodings' works.
This could possibly resolved using the devcontainer, but that has issues too: https://github.com/apache/opendal/issues/6992