mip: Support relative source file paths in package.json.

[ned-pcs]

This PR adds the ability for mip to load files from relative source paths given in the urls section of package.json files.

Previously, the only accepted forms for source paths were http://, https://, or github:.

These changes allow a source path to be given without one of these prefixes, in which case the URL of the parent of the package.json is prepended.

So instead of someone forking a repository and having to change source lines that start with github:upstream_repository/filename into github:my_repository/filename they can now just say filename

which will survive forking or cloning without change.

This also allows doing mip.install() from a file:// location. This makes it easy to do a git clone of a repository and test installation of a package from the local filesystem, especially using the unix port.

To install this package from my fork for testing, (I think you can) use commands such as:

$ mpremote connect /dev/ttyUSB0 mip install --index https://ned-pcs.github.io/micropython-lib/mip/feature/mip-allow-relative-sources mip

Or from a networked device:

import mip
mip.install(mip, index="https://ned-pcs.github.io/micropython-lib/mip/feature/mip-allow-relative-sources")

And then restart your device, as you have now added a new version of mip. To test this, you will need to make sure that your lib directory comes before .frozen in your sys.path before importing mip.

[andrewleech]

Adding support for installation from a local folder has been on my to-do list, thanks for adding it here.

I'm also keen to support direct install from manifest rather than needing the json file but it feels right to add just the relative & local path stuff as a clean change first. I'll still try to look into inline manifest support myself on top of this eventually!

[ned-pcs]

Actually, directly creating a package.json from a manifest.py doesn't really cover the entire requirement.

As a package developer, I want to ignore .mpy files and other intermediate/implementation details, concentrating on the source files. However, the source files may well include:

• Python files, which get handled pretty well presently, both in micropython-lib distribution and in the micropython build process
• Binary resources, which need to be converted to Python using mpy_bin2res.py and loaded using pkg_resources
• Other files that need special processing (like the conversion of utemplate .tpl files into _tpl.py generators using utemplate_util.py

The problem is that the manifest.py doesn't allow for these other conversion steps (some of which may need to be specified in a package-dependent manner, depending on the nature of the source files).

And whatever scheme is developed will have to deal both with building into frozen flash as well as distribution via files or networking.

I guess this is something that needs a broader discussion with the community.

[andrewleech]

I know support for freezing non-python resources is included in this PR: https://github.com/micropython/micropython/pull/8381

This will presumably include the manifest support for describing said resources, however I don't know if conversion across to json or mip handling has been considered.

The trick of converting/wrapping them in a python file is a neat workaround for now, but ultimately inefficient and fiddly.

That being said it can be quite handy, I've actually just used the same binary 2 python concept to wrap up libraries and files to build a cpython app as a single static exe on desktop with pyoxidizer :-)

[ned-pcs]

OK, I tested some additional cases and found that I was getting OSError -113 (and/or 118?) from time to time. I added retries around these errors after a gc.collect() and a sleep, which seems to have increased reliability on my ESP32.

[ned-pcs]

I would like to add an optional third field for the contents of urls that would allow for secondary github: URLs (those not from the same repo as the package.json) to specify a non-HEAD branch. My current PR just sets the branch for secondary github: URLs to HEAD.