uuu "FB: flash -raw2sparse" consumes enormouse amount of host memory for writing .sdcard.bz2 images

[sposelenov-emcraft]

Observed on Fedora 27 running libuuu_1.3.82-0-g9c56e46 The "top" output says it all:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
14312 root      20   0 4531952 3.576g 135896 S   1.3 46.5   0:49.43 uuu        

Note the amount of the resident memory size (RES) used by uuu. It would be nice to have this fixed. My uuu.script file, for IMX8MNANO:

uuu_version 1.0.1

# Only for the non-manufactured modules
# Boot the module with the U-Boot image. U-Boot will enter the fastboot mode automatically
SDPS: boot -f imx-boot-imx8mn-ddr4-sd.bin
# Configure U-Boot variables
FB: ucmd setenv fastboot_dev mmc
FB: ucmd setenv mmcdev ${emmc_dev}
# Flash the MMC image
FB: flash -raw2sparse all image-sdcard-imx8mn-ddr4.sdcard.bz2/*
FB: Done

[nxpfrankli]

Because UUU support multi boards at the same time, it buffer the whole data. host virtual memory manage system can handle it well.

If concern memory usage, I suggest decompress bz2 before use uuu.

we already use all cores to decompress files to match download speed. if use temp file, it is equal to use virtual memory swap file.

Still not find a good way yet.

[ayurtsev]

Sometimes it is convenient to use ARM based small hosts such as raspberry pi for running the uuu. I have tried to program a decompressed 3.5 Gb image on a raspberry pi 4 (4 Gb) host, and uuu could not even open the image, returning the following error:

openat(AT_FDCWD, "image-ddr4.sdcard", O_RDONLY) = -1 EOVERFLOW (Value too large for defined data type)

I suppose this is related to the fact that the raspberry host is running a 32-bit OS, given that 4Gb RAM along with a 1 GB swap file should meet the memory requirements for opening the image. Thus, it would be good if the uuu could read the image in chunks. This would allow to use the uuu on a broader range of hosts.

[nxpfrankli]

32bit system have problem. it is not easy to change read chuck now. Or long term provide option run at small memory size.