top: Replace setup.py/metadata.txt with manifest.py.

[jimmo]

This is the first step towards updating MicroPython's package management and deployment system in a way that combines freezing, on-device-install and host-based-install. This uses the new manifest.py features: require(), package(), and module().

More details at the corresponding PR in the main: repo https://github.com/micropython/micropython/pull/8914.

[andrewleech]

I don't think split packages work in freezing currently, trying to freeze with a manifest that includes os.path results in

CC build-dev/frozen_content.c
build-dev/frozen_content.c:131541:27: error: redefinition of ‘const_qstr_table_data_os___init__’
131541 | static const qstr_short_t const_qstr_table_data_os___init__[4] = {
       |                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-dev/frozen_content.c:70201:27: note: previous definition of ‘const_qstr_table_data_os___init__’ was here
70201 | static const qstr_short_t const_qstr_table_data_os___init__[4] = {
      |                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-dev/frozen_content.c:131548:33: error: redefinition of ‘frozen_module_os___init__’
131548 | static const mp_frozen_module_t frozen_module_os___init__ = {
       |                                 ^~~~~~~~~~~~~~~~~~~~~~~~~
build-dev/frozen_content.c:70208:33: note: previous definition of ‘frozen_module_os___init__’ was here
70208 | static const mp_frozen_module_t frozen_module_os___init__ = {
      |                                 ^~~~~~~~~~~~~~~~~~~~~~~~~
build-dev/frozen_content.c:70208:33: error: ‘frozen_module_os___init__’ defined but not used [-Werror=unused-const-variable=]
cc1: all warnings being treated as errors

edit: Ignore this. the problem here was caused by my (scripted) manifest including both unix-ffi/os and python-stdlib/os

[jimmo]

Rebased and updated for recent changes.

edit: Ignore this. the problem here was caused by my (scripted) manifest including both unix-ffi/os and python-stdlib/os

@andrewleech Yeah this was on the TODO list... basically the problem is we need the whole thing to be one namespace. Initially I considered renaming unix-ffi/os to unix-ffi/os-unix, but this ends up leading to other problems, and I don't think it really makes sense.

I have tried to address this with an extra feature in https://github.com/micropython/micropython/pull/8914 and by an update that I've just pushed to this branch that essentially silos out the unix-ffi directory so that:

• By default, require() will not get you a unix-ffi library.
• If you specify unix=True in requre(), then the unix-ffi version will take precedence.

[andrewleech]

• If you specify unix=True in requre(), then the unix-ffi version will take precedence.

Personally I'd prefer it to be ffi=True as calling the arg Unix might make people think they need that set to use at all on Unix port.

[jimmo]

Personally I'd prefer it to be ffi=True as calling the arg Unix might make people think they need that set to use at all on Unix port.

@andrewleech I considered that, but the unix-ffi modules are more than just ffi. Some of them just use features that only work or make sense on unix. I agree "unix" isn't the best name for this arg though.

Can we solve this with documentation instead? The rule is pretty simple, especially given that the default behaviour (don't pass the arg) is what you want almost all of the time. The primary goal is to make impossible to accidentally require() a unix module from a microcontroller build.

There is nothing stopping you doing this

require("os")
require ("os, unix=True)

either directly or transitively.

Although if we go ahead with options 2 or 4 from https://github.com/micropython/micropython/issues/9018 then this problem goes away because the modules will be renamed to "os_ext" and "os_ext_ffi".

I also considered removing manifest.py for everything in unix-ffi (i.e. it's just an unmaintained contrib directory of files you can use however you like)... but this seems over the top.

[jimmo]

What if I renamed the arg.to unix_ffi to align it with the fact that it's literally (modulo underscore vs hyphen) the unix-ffi directory?

[andrewleech]

That's a good / unambigious argument.

[jimmo]

OK thanks! Renamed to unix_ffi

[dpgeorge]

Merged in ecef7a506ccbf31bbbdc0369d17f1d09543e0442 through f3cfc52ab076fc913dc6e266fb7370b418c8f542