pxt-microbit
pxt-microbit copied to clipboard
microsoft
MakeCode 2022 consumes more memory and some v4 programmes won't download anymore
Describe the bug We've got several reports from users where their micro:bit MakeCode projects stopped working after MakeCode 2022 release date.
To Reproduce Steps to reproduce the behavior:
- Download this hex file: microbit-fits-v4_does-not-v5.hex.zip
- Load it in https://makecode.microbit.org
- Click "Download"
- The error
program too big by 6184 bytes!appears
The same hex file in https://makecode.microbit.org/v4 downloads correctly.
Rather than share one of the user programmes, I create this hex file by adding a bunch of extensions and just randomly dropping blocks on the workspace until it stopped working on MakeCode v5 and worked in v4. So the programme in itself will not make much sense.
Expected behavior
Existing programmes to still work.
Screenshots
micro:bit version (please complete the following information):
N/A
Desktop (please complete the following information):
- OS: macOS 12
- Browser: Chrome
- Version: 103
Smartphone (please complete the following information):
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
Additional context
makecode.microbit.org version: 5.0.6 Microsoft MakeCode version: 8.0.3 microbit runtime version: v2.2.0-rc6 codal-microbit-v2 runtime version: v0.2.40
With the provided hex file in the MakeCode 2022 editor, if I then add the Data Logger extension, which is configured to be V2 only, then MakeCode successfully compiles/downloads the programme.
So the memory issue might mostly affect V1 (or at least is more easily encountered in V1), and as V1 DAL hasn't been updated vs MakeCode v4, it suggest all the additional memory comes from pxt updates.
@abchatra @mmoskal we've had more reports of this than anything else and it's a regression for these users, so would like to see if we can investigate. Especially because it affects V2 users even though the constraint appears to be related to V1, really?
@microbit-carlos @jaustin is it possible to create a repro without the extension? If you can do that, I would to figure out the exact checkin.
It really is just copy and paste code until it just about reaches the limit in V4 and then load the hex in MakeCode 2022: microbit-v4large.hex.zip
Regression was between 4.1.5 & 4.16. I need to see what changed between these 2 versions.
makecode.microbit.org version: 4.1.5 Microsoft MakeCode version: 7.1.4 pxt-common-packages: v9.0.1 microbit runtime version: v2.2.0-rc6 codal-microbit-v2 runtime version: v0.2.32
makecode.microbit.org version: 4.1.6 Microsoft MakeCode version: 7.2.16 pxt-common-packages": v9.2.7 microbit runtime version: v2.2.0-rc6 codal-microbit-v2 runtime version: v0.2.32
While this issue has been focused on flash consumption, it looks like there is also a higher RAM consumption by MakeCode:
- https://github.com/microsoft/pxt-microbit/issues/4808
it appears that this commit is the culprit: https://github.com/microsoft/pxt-common-packages/commit/9454016e1e34b0fc0cfc8e5c86a4fc1797637c49
well, it appears to be more than just that. delta debugging between the two versions @abchatra mentioned led me to that commit and checking out the commit before does fix the issue. however, reverting that commit on the stable branch that pxt-microbit points to does not fix the issue.
i've narrowed it down to libs/base and libs/radio, but there are no obvious suspects in the diffs between the old code and the new. checking out the old libs/base and libs/radio on the stable branch does reduce the code size quite a bit though.
Didn’t we add math.pow?
From: Richard Knoll @.> Sent: Thursday, July 21, 2022 11:35:20 PM To: microsoft/pxt-microbit @.> Cc: Subscribed @.***> Subject: Re: [microsoft/pxt-microbit] MakeCode 2022 consumes more memory and some v4 programmes won't download anymore (Issue #4788)
well, it appears to be more than just that. delta debugging between the two versions @abchatrahttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fabchatra&data=05%7C01%7Cjhalleux%40microsoft.com%7C65bc1929b98e4b4780d808da6b60e9d6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637940361230755307%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=biQLSQejS8dN0FziJTruwUPc3i%2BELK6k0%2BP1WYHtMxs%3D&reserved=0 mentioned led me to that commit and checking out the commit before does fix the issue. however, reverting that commit on the stable branch that pxt-microbit points too does not fix the issue.
i've narrowed it down to libs/base and libs/radio, but there are no obvious suspects in the diffs between the old code and the new. checking out the old libs/base and libs/radio on the stable branch does reduce the code size quite a bit though.
— Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fpxt-microbit%2Fissues%2F4788%23issuecomment-1191958042&data=05%7C01%7Cjhalleux%40microsoft.com%7C65bc1929b98e4b4780d808da6b60e9d6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637940361230755307%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wloWETgqks48gMLmQ9Id6sZ1KHYJ%2FUR3saXmrqJbiq8%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA73QKIFM7UCXYK42WX5PQTVVG7BRANCNFSM52MKRIRA&data=05%7C01%7Cjhalleux%40microsoft.com%7C65bc1929b98e4b4780d808da6b60e9d6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637940361230755307%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=haqQhM0cdp5uK3TNO0Avek%2BmTXB%2FHnX0JR0bFEmnH2A%3D&reserved=0. You are receiving this because you are subscribed to this thread.Message ID: @.***>
You could try to run this script https://github.com/microsoft/jacdac-stm32x0/blob/main/scripts/map-file-stats.js on the .map file before and after (requires local build)
well it took me awhile to figure out how to get yotta to run locally. I ended up having to use Ubuntu 18.04!
Anyway, here are the things that changed between version 4.1.5 (left side of arrow) and 5.0.13 (right side of arrow)
RAM.libs/blocksprj/built/yt/source/radio/radio.cpp.o 1 -> 2
RAM.libs/blocksprj/built/yt/source/core/pins.cpp.o 10 -> 11
RAM.TOTAL 2692 -> 2694
DATA.libs/blocksprj/built/yt/source/pointers.cpp.o 1252 -> 1268
DATA.TOTAL 9098 -> 9114
TEXT.libs/blocksprj/built/yt/source/radio/radio.cpp.o 408 -> 508
TEXT.libs/blocksprj/built/yt/source/core/serial.cpp.o 544 -> 548
TEXT.libs/blocksprj/built/yt/source/core/codal.cpp.o 820 -> 818
TEXT.libs/blocksprj/built/yt/source/core/pins.cpp.o 1064 -> 1108
TEXT.ym/microbit-dal/source/microbit-dal.a 34882 -> 34996
TEXT.TOTAL 114738 -> 114968
FLASH.libs/blocksprj/built/yt/source/radio/radio.cpp.o 408 -> 508
FLASH.libs/blocksprj/built/yt/source/core/serial.cpp.o 544 -> 548
FLASH.libs/blocksprj/built/yt/source/core/codal.cpp.o 836 -> 834
FLASH.libs/blocksprj/built/yt/source/core/pins.cpp.o 1149 -> 1193
FLASH.libs/blocksprj/built/yt/source/pointers.cpp.o 1278 -> 1294
FLASH.ym/microbit-dal/source/microbit-dal.a 37391 -> 37475
FLASH.TOTAL 123836 -> 124082
I tried reverting every file it listed as changed back to 4.1.5 and I'm still getting a "program too big by 5216 bytes". @mmoskal any pointers on how to interpret these results?
It could be that the size of compiled code increased - you could look at the assembly output in the editor - it has comments on top that specify sizes. To view the .asm file I think you have to disable the simulator otherwise it disappears very quickly. You could also diff the .asm but you may need to replace some/all numbers by a placeholder to remove "random" changes due to different AST node IDs etc.
I guess compiled code increase is more likely here - the 200 byte change above shouldn't be that big of a problem.
@microbit-carlos @abchatra Any update on this is it's definitely still an issue?
Thanks for the reminder @AlasdairAtKitronik. I will update by EOW on the progress.
@AlasdairAtKitronik https://makecode.microbit.org/stable should have a workaround with letting users to download on v2. Please take a look.
See this for the exact dialog: https://github.com/microsoft/pxt-microbit/pull/4824#issuecomment-1220953203
@abchatra I have tested the workaround and it works for downloading the program for V2 only. I take it this is just a temporary fix while you work out how to reduce the download size so programs fit on V1 as well?
Yotta package is the reason for hex being bigger in the latest release. Plan here is to move DAL (used by V1) to use CODAL as well, which gets rids of yotta package. This is going to take some time, meanwhile we will ship this workaround for V2.
@jaustin @finneyj - FYI
Thanks @abchatra.
As discussed, we are planning to deprecate yotta and replace with the CODAL build environment we use for V2 micro:bits. If that will help with avoiding regressions, i'll see if we can bump the priority of that a bit.
Just to clarify though - yotta is primarily just a package manager/build environment... but here the implication is that this is somehow causing additional FLASH use? Can you briefly explain why that is?
I think it was update of build environment to latest GCC, which was apparently somehow the only way to run latest yotta that was required due to yotta being killed by arm.
I'm not sure CODAL will use less flash than DAL, but you're the expert.
Also I my understanding is that the difference in flash usage wasn't large, it was just that the programs were already at the edge, is that right @riknoll ?
yes, it was pretty close already.
when we updated yotta, it forced us to update the version of ubuntu that our build service is using because the python libs no longer supported ssl on the older version. the gcc updates were incidental to that.
@riknoll could you point me to the old and new Dockerfile? Yotta hasn't been updated in years and I think we could find a combination of Ubuntu release and dependencies that can keep the same gcc version. Worse case scenario, if the issue is only the GCC version used, we can install whichever version we need by downloading a specific released from launchpad, as Arm GCC builds are hosted there and can be uncompressed and used directly without an apt package (which might be bound to an Ubuntu release).
Btw, I am a bit surprised that updating GCC resulted in larger flash and RAM consumption, it's usually been the other way around. But if the fix is to get a docker image with the same GCC version as before, I think that should be doable. These are the URLs for GCC v4 to v11, which I use in a GitHub action to install Arm GCC on GitHub CI:
- https://github.com/carlosperate/arm-none-eabi-gcc-action/blob/main/src/gcc.ts
And this is how I've set it up in a Dockerfile, we simply download it, md5 it, uncompress it, and add it to path:
- https://github.com/carlosperate/docker-microbit-toolchain/blob/master/Dockerfile#L16-L25
In terms of numbers, we are talking about at least 6 KBs of flash, which yes, it's a bit on the edge, but we get to that point easily by using couple of large extensions. And for RAM I'm not too sure how much extra memory is being consumed in V5, but it does surprise me that a newer version of GCC results in more RAM usage.