uuu FB: flash -raw2sparse never works efficiently

[niziak]

raw2sparse is useless. Something is wrong with sparse area detection. With every data transfer to u-boot's fastboot (m_sparse_limit default 16M or modified 67M - doesn't matter) it sends raw data chunks - which is correct, but for area full of zeroes it still using raw data data blocks together with don't care blocks:

=== Sparse Image Header ===
magic: 0xed26ff3a
major_version: 0x1
minor_version: 0x0
file_hdr_sz: 28
chunk_hdr_sz: 12
blk_sz: 4096
total_blks: 81910
total_chunks: 2
Flashing Sparse Image
=== Chunk Header ===
chunk_type: 0xcac3
chunk_data_sz: 0xfff8
total_size: 0xc
=== Chunk Header ===
chunk_type: 0xcac1
chunk_data_sz: 0x3ffe
total_size: 0x3ffe00c
Wrote 81910 blocks, expected to write 81910 blocks
........ wrote 67100672 bytes to 'all'

I checked source file history and it was never implemented to detect sparse areas: https://github.com/nxp-imx/mfgtools/commit/3900f693995a724d409aee0419223b242c2961b1#diff-8347bca33dd95d8a1ac9905272e36e835b7d415348519dd6e25d4c4d98238989R155

After restoring commented line: https://github.com/nxp-imx/mfgtools/blob/24fd043225903247f71ac10666d820277c0b10b1/libuuu/sparse.cpp#L119 it generates lots of chunks in one transfer block. System boots correctly, so it solves problem for me. But I'm not expert in this tool area, so please check how to correctly add real raw2sparse file support.

=== Sparse Image Header ===
magic: 0xed26ff3a
major_version: 0x1
minor_version: 0x0
file_hdr_sz: 28
chunk_hdr_sz: 12
blk_sz: 4096
total_blks: 773385
total_chunks: 11
Flashing Sparse Image
=== Chunk Header ===
chunk_type: 0xcac3
chunk_data_sz: 0x8677
total_size: 0xc
=== Chunk Header ===
chunk_type: 0xcac1
chunk_data_sz: 0x3392
total_size: 0x339200c
=== Chunk Header ===
chunk_type: 0xcac2
chunk_data_sz: 0xb05f7
total_size: 0x10
=== Chunk Header ===
chunk_type: 0xcac1
chunk_data_sz: 0x1d
total_size: 0x1d00c
=== Chunk Header ===
chunk_type: 0xcac2
chunk_data_sz: 0xce
total_size: 0x10
=== Chunk Header ===
chunk_type: 0xcac1
chunk_data_sz: 0x5
total_size: 0x500c
=== Chunk Header ===
chunk_type: 0xcac2
chunk_data_sz: 0xff
total_size: 0x10
=== Chunk Header ===
chunk_type: 0xcac1
chunk_data_sz: 0x2
total_size: 0x200c
=== Chunk Header ===
chunk_type: 0xcac2
chunk_data_sz: 0x60f
total_size: 0x10
=== Chunk Header ===
chunk_type: 0xcac1
chunk_data_sz: 0x1
total_size: 0x100c
=== Chunk Header ===
chunk_type: 0xcac2
chunk_data_sz: 0x508
total_size: 0x10
Wrote 773385 blocks, expected to write 773385 blocks
........ wrote 3026788352 bytes to 'all'

The same image file, converted with img2simg and flashed with the same uuu works perfectly. Few block of data, and huge block of sparse data (which requires USB timeout increase)

[nxpfrankli]

CHUNK_TYPE_FILL just reduce the USB transfer time, but increase eMMC write time. Some eMMC don't like small block write. The function CHUNK_TYPE_FILL is equal to CHUNCK_TYPE_RAW except small data transfer by USB. I remember img2simg also don't check real spare area to create DON'T CARE CHUCK . The data write to EMMC should be equal to img2simg. You can use uuu crc cmd to check which block is wrong.

[niziak]

There is no wrong block. "Wrong" (unefficient) is how uuu transfers data in chunks. Using img2simg or patched -raw2sparse uuu reduces overall flashing time from 4 minute to 2 minutes (due to reduced USB transfer, reduced number of chunks = operations).

Tested with image file size 3GB where 180MB is only used.

I know that tools cannot guess where is real sparse area - it is only stream of bytes and assuming that block of zeros is DON'T CARE is a big risk. My plan is to write tool which us using wic's bitmap file to create DON'T CARE areas. This should reduce flashing time a lot - but this is not related to this issue.

[nxpfrankli]

I see. Only Performance problem. It is really update to how much data used. if only few continue zero block, emmc write will be very slow. May be we can do some trade off. Previous I tried use lz4 compress before usb transfer, then use unlz4 at board level. but most case can't get preformance gain.

[niziak]

Yes It is performance problem. But after using commented out code: (https://github.com/nxp-imx/mfgtools/blob/24fd043225903247f71ac10666d820277c0b10b1/libuuu/sparse.cpp#L119) tool works as designed (FILL blocks used). Can you check if this line can be restored (what side effects for other use cases) ? Or add parameter to enable same data detection ?

In meantime I've created own tool based on Android's img2simg where generated XML .bitmap file is used (using libxml2 library). With this new tool I can create Android's sparse file with DON'T CARE blocks so only really used date are transferred and flashed.

Can consider adding .bitmap support to this tool ?

[dnbazhenov]

@niziak Please, take a notice on https://github.com/nxp-imx/mfgtools/pull/394