httpimport

httpimport copied to clipboard

Hi! First of all thank you for your library. John can you please confirm that httpimport works with remote libraries and packages only if:

• these remote libraries/packages are full (inside the folder 'master' user finds all the external required libraries) or

• all the dependencies (external required libraries) are already installed properly in the python environment?

An other question is: if I want to import a remote library/package named "A" that requires some external packages not available inside the 'master' folder of "A" (for example the one I sent you, fipy), can I create an alternative library/package named "A_full" with inside all the external required libraries/packages (pure-python or not-pure python like *.dll, *.so, *.a, *.lib, ... ) in order to be able to import all without doing 'pip install ...' before using httpimport with lib "A"?

A simpler example: if remote lib "A" requires a remote lib "B" with a different url from "A", httpimport can't import "A" because httpimport can't know where to search for "B" in internet. But if I add lib "B" inside the master folder of "A" creating a github clone of the lib "A" + lib "B", is it possible to use httpimport with the cloned lib "A"?

I do mostly scientific computation with Python and sometimes I use a remote server that offers a full python environment with a lot of useful math packages (https://sagecell.sagemath.org/). Sometimes I need some packages not installed in the remote server, so it would be very useful to import temporary these packages with a remote importer via web (user can't install with pip any library in the server, user has only RAM and CPU time for each calculation).

Thank you! Regards Matteo

Hello Mat! Let's spare some insight.

On File Types that can be imported

httpimport is a pure Python importer. That means that it only imports pure Python files (.py only for the time being, and .pyc is also possible to be implemented). Extensions compiled from C (.dll, .so, .a, .lib, .pyd) can't be supported without big changes (implementing/reusing an in-memory/fileless way to get these objects in runtime and Namespace). Yet, the compilation has to be correct (same arch and OS) for the platform that is using httpimport. That can be quirky.

So, we end up with support for packages/modules that only contain:

• .py files in directory structure
• single .py files

On Dependency Hell of packages/modules

Dependencies can be handled recursively, like in every other Finder/Loader:

Let's assume that test_package depends on test_package_2 and both are not available in current $PYTHONPATH. Specifically:

• test_package is served under http://localhost:8001
• test_package_2 is served under http://localhost:8002

The following code will import test_package:

>>> from httpimport import remote_repo
>>> with remote_repo(['test_package_2'],"http://localhost:8002") :
...     with remote_repo(['test_package'],"http://localhost:8001") :
...             import test_package
... 
[!] 'test_package.test_package_2' not found in HTTP repository. Moving to next Finder.
>>> 
>>> test_package
<module 'test_package' from 'http://localhost:8001/test_package/__init__.py'>
>>> test_package.test_package_2
<module 'test_package_2' from 'http://localhost:8002/test_package_2/__init__.py'>
>>> 

The Warning was generated by the Importer that queried the http://localhost:8001 HTTP repo (inner with statement). When it moved to the next Finder (that queried the http://localhost:8002 repo), no Warning fired, hence the import was Succesful!

Ok John, thank you very much for answer!

We know that when a program "A" is executed in a computer, the computer tries to execute the program "A" by searching for all needed libraries in some folders stored in hard disk (it depends on the executable or script main file).

Well, based on your knowledge and in general (regardless of httpimport), can we execute a main program or a main python script "A" that requires some external libraries (let's suppose these libraries are stored in a remote hard disk like GitHub, Dropbox, etc... and we know the url of each required library) by reading and storing in RAM memory of the user's computer both the main program or script "A" and all libraries as if they were on the hard disk of the user's computer? Does something like this exist?

In other words, can we temporarily put in the RAM memory the entire folder tree (with all the contents, required files/libraries in sub-folders, etc...) to run a script, instead of having this folder tree physically saved on the user's hard disk? We know that httpimport can do it with pure python scripts (in my opinion it is a great idea!).

I know that if I can read a library compiled in Linux, I can't read the same library using Windows (a program for Linux os can't be run using Windows os). But the python remote server provided by SageMathCell uses a Linux distribution, so user could compile what he/she wants with a Linux virtual machine installed on his/her computer, create a personal folder with tha main script and all the required libraries (pure python scripts or compiled files by user), save this folder in a remote hard disk like GiHub or personal Dropbox account, use the remote python server to import/load his/her own library to use it with any online device and an interface between his device and the remote python server.

What could be the advantages and disadvantages of a system that copies in the RAM of a computer the complete hierarchy of folders/subfolders saved on the hard disk in order to run any program in RAM as if it was saved on the hard disk with all its dependencies?

Thank you! Regards

In a nutshell you are requesting a native (Linux/Windows) linker-loader using HTTP/S (or networking in general).

This is a little bit out of scope for httpimport (as it is Python only thing), and I'm not aware if this can be done in general. But to get it straight, you need such a linker that using ldd will list a HTTP/S URL instead of local library files. This is interesting stuff!

I think if .zip import is supported, it will take care of dependencies a great deal