Freezing the byte-code of a module and include it in the firmware doesn't work

[RobertLucian]

I'm working on reducing the RAM usage of a module I'm writing in micropython and I've hit a roadblock - after just a couple hundred of lines the microbit stops working.

---

So, I started looking into converting the python script into a .mpy bytecode and then convert it to a .c file to have it compiled within the firmware. These are the steps I have followed (with support from https://github.com/bbcmicrobit/micropython/issues/347 & https://github.com/bbcmicrobit/micropython/issues/530):

1. Cloned https://github.com/bbcmicrobit/micropython repo on an Ubuntu machine and then used the following hash https://github.com/bbcmicrobit/micropython/blob/master/inc/genhdr/mpversion.h#L3 to clone the official https://github.com/micropython/micropython repo.

2. Created perf.py module of which implementation can be seen down below.

3. Built mpy-cross program from the official version of micropython and then I placed it system-wide so I can use it everywhere.

4. Ran mpy-cross perf.py to generate the perf.mpy bytecode.

5. Placed perf.mpy in an empty folder inside the tools folder in the official repo clone and ran python make-frozen.py generic-folder-name > frozen_module.c.

6. Copied frozen_module.c file in the microbit/micropython repo clone in source/py and then added the #define MICROPY_MODULE_FROZEN (1) and #define MICROPY_MODULE_FROZEN_STR (1) instructions in inc/microbit/mpconfigport.h header on line 30.

7. cded to the root directory of the microbit/micropython repo clone and ran make all as yt build doesn't work - probably due to having had to install gcc-arm-embedded instead of gcc-arm-none-eabi package. That's because the pmiller-opensource/ppa repo is not accessible and it gives me a 404.

8. Copied the build/bbc-microbit-classic-gcc-nosd/source/microbit-micropython.hex binary over the microbit and waited for it to flash it.

---

Upon REPLing into the microbit, I tried to import the module called perf and I got an ImportError telling me it doesn't exist. Then I typed help('modules') and there was a perf module, but it was called perf., which doesn't make any sense. I took the bait and I tried to import perf. and I got a syntax error, which I think it's the expectable behavior. Here's the response I got from help('modules'):

Type "help()" for more information.
>>> help('modules')
__main__          love              os                this
antigravity       machine           perf.             time
array             math              radio             ucollections
audio             microbit          random            ustruct
builtins          micropython       speech            utime
collections       music             struct
gc                neopixel          sys
Plus any modules on the filesystem
>>>

Here's a preview of freeze_module.c:

#include <stdint.h>
const char mp_frozen_str_names[] = {
"perf.mpy\0"
"\0"};
const uint32_t mp_frozen_str_sizes[] = {
602,
};
const char mp_frozen_str_content[] = {
"M\x02\x02\x1f" "=\x02\x00\x00\x00\x00\x00\x0e\n\x00\xf7\x00" ")Hg\x85\x0e\x85\x08\x00\x00\xff\x80\x11" "h\xf8\x00" "$\xf8\x00\x80\x11" "h\xf9\x00" "$\xf9\x00\x14\x80\xce\x10" "$\xfa\x00" "`\x00" "$\xfb\x00" "`\x01" "$\x00\x01" "`\x02" "$\x08\x01\x11" "[\x08" "<module>\x07" "perf.py\x04" "time\x04" "time\x08" "microbit\x08" "microbit\x01" "N\x0b" "read_button\x07" "time_it\x07" "call_it\x00\x03" "b\x07\x00\x00\x01\x00\x00\x14\xfb\x00\xf7\x00\x81\x07" "*)$$$$$$$$$\x00\x00\xff\x1c\xf9\x00\x1d\xfe\x00\x1d\xfd\x00\xc1\xb0\x8a\xe0\x80" "5,\x80" "0\xc2\xb1" "d\x00" "2\xb1" "d\x00" "2\xb1" "d\x00" "2\xb1" "d\x00" "2\xb1" "d\x00" "2\xb1" "d\x00" "2\xb1" "d\x00" "2\xb1" "d\x00" "2\xb1" "d\x00" "2\xb1" "d\x00" "2\x81\xe9" "13\xf0" "6\xce\x7f" "22\x11" "[\x0b" "read_button\x07" "perf.py\x08" "microbit\x08" "button_a\nis_pressed\x00\x00\x01" "nU\r\x00\x00\x02\x00\x00\x0e\x00\x01\xf7\x00\x81\x15" ")%)+#\x00\x00\xff\x1c\xf8\x00\x1e\x03\x01" "f\x00\xc2\xb0\xb1" "d\x01" "2\x1c\xf8\x00\x1e\x03\x01" "f\x00\xc3\x1c\xf8\x00\x1e\x06\x01\xb3\xb2" "f\x02\xc4\x17\x02\xc5\x1c\xc7\x00\xb5\x1e\x8c\x00\xb4\x17\x03\xde\xb4\xb1\xe0\xb1\xb4\xe0\x17\x04\xde" "f\x03" "d\x01" "2\x11" "[\x07" "time_it\x07" "perf.py\x04" "time\x08" "ticks_us\x04" "time\x08" "ticks_us\x04" "time\nticks_diff\x05" "print\x06" "format\x03\x00\x01" "f\x01" "ns6{:5.3f} sec, {:6.3} usec/button : {:8.2f} kbuttons/secf\x05" "1e-06f\x06" "1000.0\x1e\x03\x00\x00\x00\x00\x00\x09\x08\x01\xf7\x00\x81\x1d\x00\x00\xff\x1c\x00\x01\x1c\xfb\x00\x1c\xfa\x00" "d\x02" "2\x11" "[\x07" "call_it\x07" "perf.py\x07" "time_it\x0b" "read_button\x01" "N\x00\x00\0"
};

And here's the source code of perf.py. And yeah, I know, I used your example program from one of your @dpgeorge conferences :) :

import time
import microbit

N = 10000

# read the button n times
def read_button(n):
        is_pressed = microbit.button_a.is_pressed
        for i in range(n/10):
                is_pressed()
                is_pressed()
                is_pressed()
                is_pressed()
                is_pressed()
                is_pressed()
                is_pressed()
                is_pressed()
                is_pressed()
                is_pressed()

def time_it(f, n):
        t0 = time.ticks_us()
        f(n)
        t1 = time.ticks_us()
        dt = time.ticks_diff(t1, t0)
        fmt = '{:5.3f} sec, {:6.3} usec/button : {:8.2f} kbuttons/sec'
        print(fmt.format(dt * 1e-6, dt / n, n / dt * 1e3))

def call_it():
        time_it(read_button, N)

# call_it()

---

Now, if I just skip the byte-compiling stuff (that is skipping mpy-cross) and just do the freezing stuff to generate the c file, the perf module appears when calling help('modules') and I can import it and even run whatever it's in there just fine. But it seems to stop working when I try to freeze the byte-code.

Does anyone know what's the problem here? Am I missing something? I truly hope it's not something obvious.

[dpgeorge]

But it seems to stop working when I try to freeze the byte-code.

The MICROPY_MODULE_FROZEN_STR option is only intended to be used to freeze .py scripts. It doesn't work if you try to freeze a .mpy.

There are a few different ways to freeze things and it can get a bit complicated how to do it a certain way. The option you want is MICROPY_MODULE_FROZEN_MPY, and then tools/mpy-tool.py -f script.mpy, and then compile that file into the firmware (there are a few extra bits that might be needed, so maybe look at how the upstream ports/stm32 builds itself for hints).

---

An alternative is to import .mpy files from the microbit's filesystem directly (ie don't freeze them into the firmware). This feature requires a few small changes to the source code. First, add this line to inc/microbit/mpconfigport.h:

#define MICROPY_PERSISTENT_CODE_LOAD (1)

Then in source/microbit/filesystem.c delete the existing function mp_lexer_new_from_file and add these 2 functions to the end of this file:

void mp_reader_new_file(mp_reader_t *reader, const char *filename) {
    file_descriptor_obj *fd = microbit_file_open(filename, strlen(filename), false, false);
    if (fd == NULL) {
        mp_raise_OSError(MP_ENOENT);
    }
    reader->data = fd;
    reader->readbyte = file_read_byte;
    reader->close = (void(*)(void*))microbit_file_close;
}

mp_lexer_t *mp_lexer_new_from_file(const char *filename) {
    mp_reader_t reader;
    mp_reader_new_file(&reader, filename);
    return mp_lexer_new(qstr_from_str(filename), reader);
}

Then you'll need to check out v1.9.2 of upstream MicroPython and build mpy-cross from that version (to match the version of MicroPython used on the microbit), and use that mpy-cross to compile your .py to .mpy. Then copy the .mpy to the microbit's filesystem (eg using tools/upload.py in this repo).

Hopefully that helps to get it working!

[RobertLucian]

The MICROPY_MODULE_FROZEN_STR option is only intended to be used to freeze .py scripts. It doesn't work if you try to freeze a .mpy. There are a few different ways to freeze things and it can get a bit complicated how to do it a certain way. The option you want is MICROPY_MODULE_FROZEN_MPY, and then tools/mpy-tool.py -f script.mpy, and then compile that file into the firmware (there are a few extra bits that might be needed, so maybe look at how the upstream ports/stm32 builds itself for hints).

Regarding the above quote, I ran tools/mpy-tool.py -f perf.mpy > frozen_module.c (after having checked out the appropriate git commit) and then I copied that source to source/py in the microbit repo. Following that, I made sure to have this 3 includes in inc/microbit/mpconfigport.h:

#define MICROPY_MODULE_FROZEN     (1) // not sure if this is needed anymore but I added it anyway
#define MICROPY_MODULE_FROZEN_STR (1) // same as above
#define MICROPY_MODULE_FROZEN_MPY (1)

And then ran make all in the microbit repo and at the end of the compilation phase I got the following errors:

Build errors on microbit firmware w/ frozen module

[219/326] Building C object source/CMakeFiles/microbit-micropython.dir/home/robert/repos/mpy-microbit/source/py/frozen_module.c.o
FAILED: source/CMakeFiles/microbit-micropython.dir/home/robert/repos/mpy-microbit/source/py/frozen_module.c.o
/usr/bin/arm-none-eabi-gcc -DYOTTA_MODULE_NAME=microbit-micropython -Dmicrobit_micropython_EXPORTS -Igenerated/include -I/home/robert/repos/mpy-microbit -I/home/robert/repos/mpy-microbit/yotta_modules/microbit-dal -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic -I/home/robert/repos/mpy-microbit/yotta_modules/ble -I/home/robert/repos/mpy-microbit/yotta_modules/ble-nrf51822 -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk -I/home/robert/repos/mpy-microbit/inc -I/home/robert/repos/mpy-microbit/inc/microbit -I/home/robert/repos/mpy-microbit/yotta_modules/microbit-dal/inc/core -I/home/robert/repos/mpy-microbit/yotta_modules/microbit-dal/inc/types -I/home/robert/repos/mpy-microbit/yotta_modules/microbit-dal/inc/drivers -I/home/robert/repos/mpy-microbit/yotta_modules/microbit-dal/inc/bluetooth -I/home/robert/repos/mpy-microbit/yotta_modules/microbit-dal/inc/platform -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/api -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/hal -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/cmsis -I/home/robert/repos/mpy-microbit/yotta_modules/ble-nrf51822/source/btle -I/home/robert/repos/mpy-microbit/yotta_modules/ble-nrf51822/source/btle/custom -I/home/robert/repos/mpy-microbit/yotta_modules/ble-nrf51822/source/common -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/ble/ble_radio_notification -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/ble/ble_services/ble_dfu -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/ble/common -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/ble/device_manager -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/ble/device_manager/config -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/ble/peer_manager -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/device -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/drivers_nrf/ble_flash -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/drivers_nrf/delay -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/drivers_nrf/hal -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/drivers_nrf/pstorage -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/drivers_nrf/pstorage/config -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/libraries/bootloader_dfu -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/libraries/bootloader_dfu/hci_transport -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/libraries/crc16 -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/libraries/hci -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/libraries/scheduler -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/libraries/timer -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/libraries/util -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/libraries/fds -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/libraries/fstorage -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/libraries/experimental_section_vars -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/softdevice/common/softdevice_handler -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/softdevice/s130/headers -I/home/robert/repos/mpy-microbit/yotta_modules/nrf51-sdk/source/nordic_sdk/components/toolchain -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/cmsis/TARGET_NORDIC -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/cmsis/TARGET_NORDIC/TARGET_MCU_NRF51822 -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/cmsis/TARGET_NORDIC/TARGET_MCU_NRF51822/TOOLCHAIN_GCC_ARM -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/cmsis/TARGET_NORDIC/TARGET_MCU_NRF51822/TOOLCHAIN_GCC_ARM/TARGET_MCU_NRF51_16K_S110 -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC/TARGET_MCU_NRF51822 -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC/TARGET_MCU_NRF51822/Lib -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC/TARGET_MCU_NRF51822/Lib/nordic_sdk -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC/TARGET_MCU_NRF51822/Lib/nordic_sdk/components -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC/TARGET_MCU_NRF51822/Lib/nordic_sdk/components/libraries -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC/TARGET_MCU_NRF51822/Lib/nordic_sdk/components/libraries/crc16 -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC/TARGET_MCU_NRF51822/Lib/nordic_sdk/components/libraries/scheduler -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC/TARGET_MCU_NRF51822/Lib/nordic_sdk/components/libraries/util -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC/TARGET_MCU_NRF51822/Lib/s110_nrf51822_8_0_0 -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC/TARGET_MCU_NRF51822/Lib/s130_nrf51822_1_0_0 -I/home/robert/repos/mpy-microbit/yotta_modules/mbed-classic/targets/hal/TARGET_NORDIC/TARGET_MCU_NRF51822/TARGET_NRF51_MICROBIT -I/home/robert/repos/mpy-microbit/source -std=c99 -fno-exceptions -fno-unwind-tables -ffunction-sections -fdata-sections -Wall -Wextra -mcpu=cortex-m0 -mthumb -D__thumb2__ -Os -g -gdwarf-3 -DNDEBUG   -DTOOLCHAIN_GCC -DTOOLCHAIN_GCC_ARM -DMBED_OPERATORS -DNRF51 -DTARGET_NORDIC -DTARGET_M0 -D__MBED__=1 -DMCU_NORDIC_16K -DTARGET_NRF51_MICROBIT -DTARGET_MCU_NORDIC_16K -DTARGET_MCU_NRF51_16K_S110  -DTARGET_NRF_LFCLK_RC -DTARGET_MCU_NORDIC_16K -D__CORTEX_M0 -DARM_MATH_CM0 -DNO_BLE -include "/home/robert/repos/mpy-microbit/build/bbc-microbit-classic-gcc-nosd/yotta_config.h" -MMD -MT source/CMakeFiles/microbit-micropython.dir/home/robert/repos/mpy-microbit/source/py/frozen_module.c.o -MF source/CMakeFiles/microbit-micropython.dir/home/robert/repos/mpy-microbit/source/py/frozen_module.c.o.d -o source/CMakeFiles/microbit-micropython.dir/home/robert/repos/mpy-microbit/source/py/frozen_module.c.o -c /home/robert/repos/mpy-microbit/source/py/frozen_module.c
/home/robert/repos/mpy-microbit/source/py/frozen_module.c:34:5: error: redeclaration of enumerator 'MP_QSTR__lt_module_gt_'
     MP_QSTR__lt_module_gt_ = MP_QSTRnumber_of,
     ^~~~~~~~~~~~~~~~~~~~~~
In file included from /home/robert/repos/mpy-microbit/inc/py/obj.h:31:0,
                 from /home/robert/repos/mpy-microbit/inc/py/objint.h:30,
                 from /home/robert/repos/mpy-microbit/source/py/frozen_module.c:2:
/home/robert/repos/mpy-microbit/inc/genhdr/qstrdefs.generated.h:13:6: note: previous definition of 'MP_QSTR__lt_module_gt_' was here
 QDEF(MP_QSTR__lt_module_gt_, (const byte*)"\xbd\x08" "<module>")
      ^
/home/robert/repos/mpy-microbit/inc/py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
/home/robert/repos/mpy-microbit/source/py/frozen_module.c:36:5: error: redeclaration of enumerator 'MP_QSTR_time'
     MP_QSTR_time,
     ^~~~~~~~~~~~
In file included from /home/robert/repos/mpy-microbit/inc/py/obj.h:31:0,
                 from /home/robert/repos/mpy-microbit/inc/py/objint.h:30,
                 from /home/robert/repos/mpy-microbit/source/py/frozen_module.c:2:
/home/robert/repos/mpy-microbit/inc/genhdr/qstrdefs.generated.h:725:6: note: previous definition of 'MP_QSTR_time' was here
 QDEF(MP_QSTR_time, (const byte*)"\xf0\x04" "time")
      ^
/home/robert/repos/mpy-microbit/inc/py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
/home/robert/repos/mpy-microbit/source/py/frozen_module.c:37:5: error: redeclaration of enumerator 'MP_QSTR_microbit'
     MP_QSTR_microbit,
     ^~~~~~~~~~~~~~~~
In file included from /home/robert/repos/mpy-microbit/inc/py/obj.h:31:0,
                 from /home/robert/repos/mpy-microbit/inc/py/objint.h:30,
                 from /home/robert/repos/mpy-microbit/source/py/frozen_module.c:2:
/home/robert/repos/mpy-microbit/inc/genhdr/qstrdefs.generated.h:26:6: note: previous definition of 'MP_QSTR_microbit' was here
 QDEF(MP_QSTR_microbit, (const byte*)"\xc0\x08" "microbit")
      ^
/home/robert/repos/mpy-microbit/inc/py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
/home/robert/repos/mpy-microbit/source/py/frozen_module.c:42:5: error: redeclaration of enumerator 'MP_QSTR_button_a'
     MP_QSTR_button_a,
     ^~~~~~~~~~~~~~~~
In file included from /home/robert/repos/mpy-microbit/inc/py/obj.h:31:0,
                 from /home/robert/repos/mpy-microbit/inc/py/objint.h:30,
                 from /home/robert/repos/mpy-microbit/source/py/frozen_module.c:2:
/home/robert/repos/mpy-microbit/inc/genhdr/qstrdefs.generated.h:181:6: note: previous definition of 'MP_QSTR_button_a' was here
 QDEF(MP_QSTR_button_a, (const byte*)"\xed\x08" "button_a")
      ^
/home/robert/repos/mpy-microbit/inc/py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
/home/robert/repos/mpy-microbit/source/py/frozen_module.c:43:5: error: redeclaration of enumerator 'MP_QSTR_is_pressed'
     MP_QSTR_is_pressed,
     ^~~~~~~~~~~~~~~~~~
In file included from /home/robert/repos/mpy-microbit/inc/py/obj.h:31:0,
                 from /home/robert/repos/mpy-microbit/inc/py/objint.h:30,
                 from /home/robert/repos/mpy-microbit/source/py/frozen_module.c:2:
/home/robert/repos/mpy-microbit/inc/genhdr/qstrdefs.generated.h:183:6: note: previous definition of 'MP_QSTR_is_pressed' was here
 QDEF(MP_QSTR_is_pressed, (const byte*)"\xe6\x0a" "is_pressed")
      ^
/home/robert/repos/mpy-microbit/inc/py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
/home/robert/repos/mpy-microbit/source/py/frozen_module.c:44:5: error: redeclaration of enumerator 'MP_QSTR_n'
     MP_QSTR_n,
     ^~~~~~~~~
In file included from /home/robert/repos/mpy-microbit/inc/py/obj.h:31:0,
                 from /home/robert/repos/mpy-microbit/inc/py/objint.h:30,
                 from /home/robert/repos/mpy-microbit/source/py/frozen_module.c:2:
/home/robert/repos/mpy-microbit/inc/genhdr/qstrdefs.generated.h:219:6: note: previous definition of 'MP_QSTR_n' was here
 QDEF(MP_QSTR_n, (const byte*)"\xcb\x01" "n")
      ^
/home/robert/repos/mpy-microbit/inc/py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
/home/robert/repos/mpy-microbit/source/py/frozen_module.c:45:5: error: redeclaration of enumerator 'MP_QSTR_ticks_us'
     MP_QSTR_ticks_us,
     ^~~~~~~~~~~~~~~~
In file included from /home/robert/repos/mpy-microbit/inc/py/obj.h:31:0,
                 from /home/robert/repos/mpy-microbit/inc/py/objint.h:30,
                 from /home/robert/repos/mpy-microbit/source/py/frozen_module.c:2:
/home/robert/repos/mpy-microbit/inc/genhdr/qstrdefs.generated.h:724:6: note: previous definition of 'MP_QSTR_ticks_us' was here
 QDEF(MP_QSTR_ticks_us, (const byte*)"\x5a\x08" "ticks_us")
      ^
/home/robert/repos/mpy-microbit/inc/py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
/home/robert/repos/mpy-microbit/source/py/frozen_module.c:46:5: error: redeclaration of enumerator 'MP_QSTR_ticks_diff'
     MP_QSTR_ticks_diff,
     ^~~~~~~~~~~~~~~~~~
In file included from /home/robert/repos/mpy-microbit/inc/py/obj.h:31:0,
                 from /home/robert/repos/mpy-microbit/inc/py/objint.h:30,
                 from /home/robert/repos/mpy-microbit/source/py/frozen_module.c:2:
/home/robert/repos/mpy-microbit/inc/genhdr/qstrdefs.generated.h:722:6: note: previous definition of 'MP_QSTR_ticks_diff' was here
 QDEF(MP_QSTR_ticks_diff, (const byte*)"\xb1\x0a" "ticks_diff")
      ^
/home/robert/repos/mpy-microbit/inc/py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
/home/robert/repos/mpy-microbit/source/py/frozen_module.c:47:5: error: redeclaration of enumerator 'MP_QSTR_print'
     MP_QSTR_print,
     ^~~~~~~~~~~~~
In file included from /home/robert/repos/mpy-microbit/inc/py/obj.h:31:0,
                 from /home/robert/repos/mpy-microbit/inc/py/objint.h:30,
                 from /home/robert/repos/mpy-microbit/source/py/frozen_module.c:2:
/home/robert/repos/mpy-microbit/inc/genhdr/qstrdefs.generated.h:672:6: note: previous definition of 'MP_QSTR_print' was here
 QDEF(MP_QSTR_print, (const byte*)"\x54\x05" "print")
      ^
/home/robert/repos/mpy-microbit/inc/py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
/home/robert/repos/mpy-microbit/source/py/frozen_module.c:48:5: error: redeclaration of enumerator 'MP_QSTR_format'
     MP_QSTR_format,
     ^~~~~~~~~~~~~~
In file included from /home/robert/repos/mpy-microbit/inc/py/obj.h:31:0,
                 from /home/robert/repos/mpy-microbit/inc/py/objint.h:30,
                 from /home/robert/repos/mpy-microbit/source/py/frozen_module.c:2:
/home/robert/repos/mpy-microbit/inc/genhdr/qstrdefs.generated.h:579:6: note: previous definition of 'MP_QSTR_format' was here
 QDEF(MP_QSTR_format, (const byte*)"\x26\x06" "format")
      ^
/home/robert/repos/mpy-microbit/inc/py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
/home/robert/repos/mpy-microbit/source/py/frozen_module.c:49:5: error: redeclaration of enumerator 'MP_QSTR_f'
     MP_QSTR_f,
     ^~~~~~~~~
In file included from /home/robert/repos/mpy-microbit/inc/py/obj.h:31:0,
                 from /home/robert/repos/mpy-microbit/inc/py/objint.h:30,
                 from /home/robert/repos/mpy-microbit/source/py/frozen_module.c:2:
/home/robert/repos/mpy-microbit/inc/genhdr/qstrdefs.generated.h:335:6: note: previous definition of 'MP_QSTR_f' was here
 QDEF(MP_QSTR_f, (const byte*)"\xc3\x01" "f")
      ^
/home/robert/repos/mpy-microbit/inc/py/qstr.h:41:23: note: in definition of macro 'QDEF'
 #define QDEF(id, str) id,
                       ^~
[224/326] Building C object source/CMakeFiles/microbit-micropython.dir/home/robert/repos/mpy-microbit/source/py/objexcept.c.o
ninja: build stopped: subcommand failed.
error: command ['ninja'] failed
Makefile:19: recipe for target 'yotta' failed
make: *** [yotta] Error 1
robert@robert-700t1c:~/repos/mpy-microbit

It all looks like this stems from having instructions being redefined inside frozen_module.c source file. And I was going through the upstream micropython and I saw in the minimal port a line where the -q option is included along with -f: https://github.com/micropython/micropython/blob/b9ec6037edf5e6ff6f8f400d70f7351d1b0af67d/ports/minimal/Makefile#L61 Am I supposed to generate a header file before doing the compilation?

[dpgeorge]

Am I supposed to generate a header file before doing the compilation?

Yes you need to use the -q option to mpy-tool.py, and pass it qstrdefs.preprocessed.h. To generate this latter file you'll need to run tools/makeqstrhdr.py (that script is in this repo).

[RobertLucian]

Thank you a lot for your support so far. That's really great!

Okay, I've done the following things in the following order:

1. make all on this repo.

2. Ran ./tools/makeqstrhdr.sh tool (on mine it's actually an sh file and not a py).

3. Copied the generated inc/genhdr/qstrdefs.preprocessed.h to the folder where the mpy bytecode for my module is.

4. Ran python mpy-tool.py -f -q qstrdefs.preprocessed.h perf.mpy > frozen_module.c.

5. Copied frozen_module.c to source/py in this repo.

6. Ran the make all command again without removing the previous build.

7. Copied the /build/bbc-microbit-classic-gcc-nosd/source/microbit-micropython.hex over my microbit and waited it to flash it.

8. Opened REPL and typed help('modules') and I got the usual list of modules that come by-default. There's no sign of perf module in there.

Do you have any thoughts on this?

[dpgeorge]

All steps in the above comment are correct. The only thing missing is an additional step:

5a. Edit inc/microbit/mpconfigport.h and add these 2 lines:

#define MICROPY_QSTR_EXTRA_POOL (mp_qstr_frozen_const_pool)
#define MICROPY_MODULE_FROZEN_MPY (1)

[RobertLucian]

Hi @dpgeorge,

That was the missing link for sure. I got it to work. Thanks a lot for your support and for having the time to answer all these questions here.

For everyone else reading this and those who have an interest in reducing the RAM usage, I was able to do:

• Precompile a python module to a byte-code (aka .mpy file) and then copy that over the microbit. This has the advantage of skipping the precompiling phase on the microbit that could leave it without RAM resources during this process. Unfortunately, this method still requires the microbit to load the module into the RAM.

• Turning the python module to a .c file that gets compiled into the firmware itself. Has the advantage of the above method + having the benefit of running the module from the flash memory as opposed to loading it into the RAM.

I would ask you to not close this issue as I still want to bring in a short step-by-step description on how to get going with this and maybe, just maybe, I'll have some other questions down the line.

[dpgeorge]

Good to hear that it is now working. For point 1 above, this support should eventually be added to the main code. For point 2, it would be good to add some documentation for this, and we can modify the code to eliminate the number of steps required (eg even make a shell script to automate most of the process).

[RobertLucian]

@dpgeorge Have you got any idea why memoryview class is not available in microbit's implementation? Is it a technical reason, probably due to the microbit's limits or am I missing something else here?

And when developing a library that you want to embed into the firmware directly, what tips do you have for making it more memory-efficient?

1. Go with classes or functions?
2. What about the number of chained functions? I understand that this limit on the microbit is set to 8. During my tests, I noticed this actually is 5. Would it be better to just reduce as many functions as possible to just a few, thus reducing the number of recursive calls?
3. Is there a possibility of creating an alias for a bunch of instructions that they would then get replaced in the code when pre-compiled? Think of the defines we have in C or const thing in micropython.
4. How can I reduce the memory fragmentation on the microbit? memoryview seems like a very good tool, behaving very similarly to a pointer, but if it's not available on the microbit, what other options am I left with?

Edit: For both of the points you mentioned earlier, I intend on doing the same in the project I'm working on right now. It's just for the record and it doesn't mean anything else.

[RobertLucian]

I enabled the built-in memoryview object here: https://github.com/bbcmicrobit/micropython/blob/master/inc/microbit/mpconfigport.h#L72

And now I'm getting an MP_QSTR_memoryview undeclared here (not in a function); did you mean MP_QSTR_remove? error. This gets reported on line 557 in source/py/objarray.c: https://github.com/bbcmicrobit/micropython/blob/master/source/py/objarray.c#L557.

To me, it looks like MP_QSTR_memoryview has to be defined in inc/genhdr/qstrdefs.generated.h, but this file needs to be regenerated so that it can have it in. And what I see is that this file is regenerated when running tools/makeqstrhdr.sh - the problem with this one is it can only be run after having done the make all command, which also fails if I enable the memoryview in mpconfigport.h. So this kinda looks like I'm in a deadlock for now.

Do you have any insight into this problem? Thanks again for your support so far.

[dpgeorge]

And what I see is that this file is regenerated when running tools/makeqstrhdr.sh - the problem with this one is it can only be run after having done the make all command

You should be able to run tools/makeqstrhdr.sh without relying on make. So just enabled memoryview as you have done in mpconfigport.h, run makeqstrhdr.sh, then do make all. That works for me.

[RobertLucian]

The version file wasn't generated and that's why I wasn't able to build the firmware. I generated the version file and then I was able to run ./tools/makeqstrhdr.sh just fine.

Thanks a lot for your support so far George :)

[RobertLucian]

Hey there @dpgeorge. After 2 months of silence, I'm getting back to this project - I had something else going on my plate in the meantime.

Anyway, do you have any idea why the generated hex file has metadata/properties incorporated into it? I'm getting a warning when copying the firmware over to the microbit. This warning alone messes up the copying process because I've got to retry it a couple of times until it works. This is what the warning sounds like:

The file firmware.hex has properties that can't be copied to the new location.

Is there a flag I've got to set before compiling the project? What is that I'm doing different here? Thanks!

Also, I don't want to open up a new issue as I think this problem goes hand in hand with the building issues I've encountered so far in this ticket.

[dpgeorge]

Anyway, do you have any idea why the generated hex file has metadata/properties incorporated into it?

It has UICR data to describe the firmware.

I'm getting a warning when copying the firmware over to the microbit.

Where does this warning come from, where does it appear? It could be because you have an old version of the KL26 firmware, see eg https://www.mbed.com/en/platform/hardware/prototyping-production/daplink/daplink-on-kl26z/

[RobertLucian]

Hi @dpgeorge ,

Yes, it was due to an old version of the KL26 firmware. I updated it and the problem went away. Thanks.

By the way, I have successfully finished building a custom firmware based on this project of yours. We needed a way to support our modules (different kind of sensors) on our GiggleBot and since the microbit's HW is a bit limited, we had to go this route - maybe later on, I can just write the drivers in plain C, thus achieving an even smaller storage footprint. For the time being, the modules are freezed. http://gigglebot-dev.rtfd.io/

And the project is here: https://github.com/RobertLucian/micropython-gigglebot There aren't any tests on this, and the main reason is that I'd have to emulate the sensors/devices in a way, and that's a bit complex. George, do you have any suggestions on this - what would you do to ensure the firmware doesn't collapse out of the blue? How would you set up the test suite?

And the answer to this issue I initially created resides in a plain Dockerfile here (in case someone else is interested): https://github.com/RobertLucian/micropython-gigglebot/blob/master/src/Dockerfile The last part is a bit hacky, but as long as there's a git hash, it can be stable. Now, I only have to update the hash every now and then in order to update the runtime version - maybe set myself a couple of alarms to not forget.

And lastly, since I can only use 248KB of flash memory for the firmware (that's what I read), and given the current firmware, that leaves me 1936 bytes to use. I need to add a couple of other drivers which take more than 2KB, so what's left is not enough.

1. Do you know how I can figure out how much space each module takes in the firmware? I'm thinking of disabling some in the microbit package (like SPI and the Compass).
2. What things should I do to the python source in order to make it use less space once freezed? Except rewriting it. I'm only talking about the minifying process.

Thank you and Merry Christmas! :)

[dpgeorge]

1. Do you know how I can figure out how much space each module takes in the firmware? I'm thinking of disabling some in the microbit package

It's not easy to get exact sizes of modules, but if you run arm-none-eabi-size on all the object files in the build/ directory that will give you an idea.

I would suggest disabling the music module to start with, or at least the built-in tunes (see source/microbit/modmusic.cpp and source/microbit/modmusictunes.cpp), as well as the speech synthesiser (see source/microbit/modspeech.c). The simplest way to completely disable these modules is to remove their corresponding line in inc/microbit/mpconfigport.h, in the MICROPY_PORT_BUILTIN_MODULES section.

2. What things should I do to the python source in order to make it use less space once freezed?

There's not much you can do about that, except maybe shorten global identifiers (class names and function/method names).