Multi-memory files

[MCUdude]

I figured I'd test the USB capabilities on my AVR-DU Curiosity Nano board, so I decided to test the newly released USB CDC to UART example provided by Microchip. The 1.0.0 release contains precompiled hex files for the AVR64DU32, but to my surprise, the file contains data at address 0x820000; outside the flash memory space.

$ avrdude -cdryrun -pavr64du32 -Uflash:w:/mnt/c/Users/Hans/Downloads/hex-files/avr64du32-cnano-usb-cdc-to-usart-bridge-mplab-mcc.X_pro_1.0.0.hex:i
avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e9621 (probably 64du32)
avrdude: Note: flash memory has been specified, an erase cycle will be performed.
         To disable this feature, specify the -D option.
avrdude: erasing chip

avrdude: processing -U flash:w:/mnt/c/Users/Hans/Downloads/hex-files/avr64du32-cnano-usb-cdc-to-usart-bridge-mplab-mcc.X_pro_1.0.0.hex:i
avrdude error: Intel Hex record [0x820000, 0x82000b] out of range [0x0000, 0xffff]
        at line 726 of /mnt/c/Users/Hans/Downloads/hex-files/avr64du32-cnano-usb-cdc-to-usart-bridge-mplab-mcc.X_pro_1.0.0.hex
avrdude error: read from file /mnt/c/Users/Hans/Downloads/hex-files/avr64du32-cnano-usb-cdc-to-usart-bridge-mplab-mcc.X_pro_1.0.0.hex failed

avrdude done.  Thank you.


$ avrdude -cdryrun -pavr64du32 -t
avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e9621 (probably 64du32)

avrdude: processing -t interactive terminal
avrdude> write flash 0 0xffff /mnt/c/Users/Hans/Downloads/hex-files/avr64du32-cnano-usb-cdc-to-usart-bridge-mplab-mcc.X_pro_1.0.0.hex
avrdude error: Intel Hex record [0x820000, 0x82000b] out of range [0x0000, 0xffff]
        at line 726 of /mnt/c/Users/Hans/Downloads/hex-files/avr64du32-cnano-usb-cdc-to-usart-bridge-mplab-mcc.X_pro_1.0.0.hex
avrdude error: (write) data /mnt/c/Users/Hans/Downloads/hex-files/avr64du32-cnano-usb-cdc-to-usart-bridge-mplab-mcc.X_pro_1.0.0.hex: unable to read the Intel Hex file
avrdude>

Would it be a good idea to let the user specify "how much" of the hex file that should be written? in my case, write flash 0 0xffff file.hex? This can be useful if the user wants to flash a pre-compiled hex file that also contains a bootloader, but wants to leave this out.

[stefanrueger]

let the user specify "how much" of the hex file that should be written? in my case, write flash 0 0xffff file.hex?

The terminal write already implements the idea of writing a subset of an input file, albeit in fill mode only:

$ avrdude -qqc dryrun -p m328p -T "write -?" 2>&1 | grep len
        write <mem> <addr> <len> <data>[,] {<data>[,]} ... # Fill, see below
Ellipsis ... writes <len> bytes padded by repeating the last <data> item.
Both the <addr> and <len> can be negative numbers; a negative <addr> starts
an interval from that many bytes below the memory size; a negative <len> ends

The ellipsis ... to indicate fill mode is needed: regular write takes the number after the address to be data.

So, you got this almost: it should be write flash 0 0x10000 file.hex ... or write flash 0 -1 file.hex ...

However, this would not help here: the problem at hand is that the specified hex file has records outside the memory address space. AVRDUDE takes a dim view and balks at reading the hex file. This could be made a warning provided -F is used.

[mcuee]

In this particular case, the memory region is probably valid but avedude does not recognize it yet.

[stefanrueger]

the memory region is probably valid

Hmm, not according to the data sheet: neither MCU data space, nor MCU code space nor UPDI Space extend to 0x820000

[stefanrueger]

Seeing that it's

avrdude error: Intel Hex record [0x820000, 0x82000b]

my guess is that it's a convention on their part to stash the 12 fuse bytes into that region (the remaining 4 fuses are reserved). This is what avr-gcc .elf files do, so further guessing this is a botched conversion from .elf to .hex.

[MCUdude]

my guess is that it's a convention on their part to stash the 12 fuse bytes into that region (the remaining 4 fuses are reserved). This is what avr-gcc .elf files do, so further guessing this is a botched conversion from .elf to .hex.

@xedbg any thoughts on this? Is this something MPLAB X / the XC8 compiler does automatically, or can be instructed to do?

[xedbg]

Well, all the AVR memory types are "stashed" at magic-token locations in the hexfile (EEPROM is at 0x810000 for example) so this is nothing new. The fact that FUSES are included by default by XC8 is in my opinion an improvement - most applications won't run correctly without the correct fuses. So I would say embrace it :)

But a semi-related word of caution - new AVRs now have PDID in the fuse space, which is a great way to program your AVR for the very last time - it would probably deserve an "are you sure?" safety feature of some sort...

[mcuee]

@stefanrueger

Actuallly this issue has been discussed before. Please refer to the dicussion here.

• https://github.com/avrdudes/avrdude/discussions/1432

Low	High	Memories
0x00000000	0x007FFFFF	Flash
0x00800000	0x0080FFFF	RAM
0x00810000	0x0081FFFF	EEPROM (*)
0x00820000	0x0082FFFF	Fuses
0x00830000	0x0083FFFF	Lock bytes
0x00840000	0x0084FFFF	Signature
0x00850000	0x0085FFFF	User signatures (*)

(*) Not available in all parts

[xedbg]

Oh, and don't forget the "boot row" which starts at 0x860000! (also a *)

[stefanrueger]

@xedbg Thanks for clarifying. Yes, we discussed a container format for .hex to be used for backup and restore. One idea was to use the address offsets deployed in .elf for .hex files, too. Above table was looking at .elf files (thanks, @mcuee).

Coding .elf files in that manner is OK: for a start, it's documented and used. And AVRDUDE can extract the various memories from the .elf file.

I had not known that Microchip started using .hex files in that fashion. @xedbg is that documented somewhere? I only find mention of 0x8n0000 addresses in 329 .PIC files (from the atdf packs) with a magicoffset(!) name.

Note that AVRDUDE can only read from/write to one particular, specified memory. So when you want to write all memories from the same .elf file to a chip, one can use a bash brace expansion to create, eg, six -U memory writes like so

  avrdude -p m328p -c usbasp -U{eeprom,flash,{l,h,e}fuse,lock}:w:file.elf

There are several problems using .hex files in that fashion, though:

• It needs an agreed on convention (eg, using the same offsets as .elf files)
• So far, .hex files have always used [0, size-1] irrespective of memory type; so a 16 byte .hex file filling the address space [0, 15] might legitimately be flash, userrow, bootrow, EEPROM or fuses. Who knows? And certainly AVRDUDE needs the user to specify the memory to write to.
• The AVRDUDE code base isn't made for this (though, as anything, it can be done by substantial rewriting)
• The user needs to know what happens; the opinion on whether it is a good idea that avrdude -U flash:w:file.hex writes fuses because they are secretly coded in the data varies from "very bad idea" to "outright bonkers"

All of that can be progressed (eg, with a new memory designation all) and documentation and hard work etc.

But this isn't what this issue is about. @MCUdude came across an (undocumented?) new type of .hex file that wouldn't play with -U flash:w:file.hex because of out-of-range data. PR #1818 fixes that by giving the user a way of ignoring these.

[MCUdude]

I think we should discuss what to do with hex files containing data for more than one memory. Since the next release probably is going to be v8.0, we are allowed to introduce breaking changes.

• If the user specifies a memory (-U flash or write eeprom), we should only be writing to that specific memory. That I think we could all agree on.
• But what if the multiple_mems.hex contains data starting from address 0 and 0x810000? Should we write data starting from address 0 or 0x810000? I'm hesitant to change how -U behave, but what if we could introduce -u as a new option for writing to one or more memories? Speaking of breaking changes, -u, -s, and -y could be made available for new features, as all of these have been obsolete since v7.
  • -u multiple_mems.hex:i could write to as many memories as possible, and -u eeprom:w:multiple_mems.hex:i would write dra starting from 0x810000 if present, and 0 otherwise? This would make -u backward compatible with -U, but could also deal with "container" hex files as well.

[stefanrueger]

Grea ideas, @MCUdude. There is a bit of topic drift here, so changing the issue to multi-memory input files

Should we write data starting from address 0 or 0x810000

There is nothing AVR8 could usefully write to at flash or any other address 0x810000: the jmp/call opcodes can only address [0, 0x7FFFFF]; this is the theoretical 8 MiB limit of AVR8 flash space. This is why the .elf table reserves 8 MiB for flash, then reserves 64 kiB each for IO/SRAM, EEPROM, Fuses, lock bits, sigrow, userrow and bootrow. (That is how .elf's multi-memory trick works out with a flat address space. AVRDUDE half-heartedly supports programming flash for one(!) 32-bit part AT32xxx; that has an offset in .hex/.srec/ of way beyond above; this does not get in the way of above .elf address space.)

But AVRDUDE could treat a file automatically as multi-memory if it finds data in [0x800000, 0x86FFFF]:

• Data in [0, 0x7FFFFF] are no longer anything; they must be considered flash and only flash
• Data in [0x800000, 0x86FFFF] are mapped to SRAM, EEPROM, Fuses, lock bits, sigrow, userrow and bootrow
• Data beyond 0x870000 triggers an out-of-range error in .hex/.srec (and PR #1818 is still useful)

If an input file does not have data in [0x800000, 0x86FFFF], it would still be considered a single-memory (normal) input file, which as before expects data in [0, size-1]. This makes the distinction automatic. No need for the user to do anything or for AVRDUDE to invent new syntax or options.

We might get empty memory files for non-empty input; for example

• Input for eeprom has data in [0, 0x421] (flash) and [0x840000, 0x840002] (signature): nothing is written
• Input for eeprom has data in [0, 0x421]; this is a single-memory file and treated as before, ie, written to eeprom

Things might break, eg, can no longer rely on AVRDUDE to end reading after size bytes for raw, dec, octal, Roman, ... input

• cat eeprom.raw efuse.raw | avrdude -U eeprom:w:-:r -U efuse:w:-:r fails (but was that ever good practice?)

Some neat effects of implementing notion of multi-memory files

• No change in user syntax necessary
• Can pick any memory from multi-memory file: -U efuse:w:allmem.hex works as does -U flash:w:allmem.hex
• Can use bash brace expansion for .srec/.hex, too: -U{eeprom,flash,{l,h,e}fuse,lock}:w:file.srec
• Can force input to be considered as flash by, eg, also storing the signature of the part in the input
• Can restrict writing of a hex file to the right part: avrdude -U {signature,flash}:w:allmem.hex
  • In AVRDUDE writing to a read-only memory works if the content matches, otherwise triggers a write error
  • Writing to the wrong part (ATmega2560 instead of AVR16EB28) should exit before flash is written
• Can restrict writing to a specific part using its serial number: -U {sernum,flash}:w:allmem.hex
• This makes implementing a future memory type all easier (but still not easy and there be dragons)
• Nothing in this multi-memory file definition pertains to .hex, so could also be done for .srec, raw, oct, ...

[MCUdude]

But AVRDUDE could treat a file automatically as multi-memory if it finds data in [0x800000, 0x86FFFF]. ... If an input file does not have data in [0x800000, 0x86FFFF], it would still be considered a single-memory (normal) input file, which as before expects data in [0, size-1]. This makes the distinction automatic. No need for the user to do anything or for AVRDUDE to invent new syntax or options.

Great idea! It would be very convenient to be able to decide which memories in a multi-memory input file to write. In my case where I wanted to write the pre-compiled native USB demo to my AVR64DU32, -Ufuses:w:multi-mem.hex:i -Uflash:w::multi-mem.hex:i would "just work". And a future memory type all sounds useful, but not a deal breaker. Maybe Avrdude could reveal to the user which memories the multi-memory input file contains before writing to memory?

[stefanrueger]

future memory type all sounds useful, but not a deal breaker

A new feature such as reading a multi-memory hex file to write single memories is bound to entail a host of desires:

• Reading multi-memory srec files to write single memories
• Writing to a list of memories such as flash,ee,fuses from multi-memory files
• Writing to all memories that exist in multi-memory files
• Writing all memories except some such as all\fuses,lock
• Automatically verifying when writing to multiple memories via -U [all | <list> | <list>\<list>]:w:file.hex:i
• Verifying a multi-memory file against a multi-memory file later on via -U [<list>|all]:v:...
• Reading a list of memories to write a multi-memory file
• Reading all memories to write a multi-memory file as backup
• Checking the signature of part against the multi-memory file signature section to avoid overwriting the wrong part
• Doing so in the terminal, eg, with a backup and restore command

Given that the single most popular feature request for AVRDUDE has been creating backups that can be read back, the time might be ripe to look into this.

Maybe Avrdude could reveal to the user which memories the multi-memory input file contains before writing to memory?

Well the hope is that -c dryrun would be the natural answer:

$ avrdude -qqc dryrun -p m328p -U all:w:backup.hex
avrdude error: signature of ATmega328P does not match file (ATmega328PB)
        use -F to override this check

$ avrdude -c dryrun -p m328pb -U all:w:backup.hex
avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e9516 (probably m328pb)

avrdude: processing -U all:w:backup.hex:i
avrdude: reading 33824 bytes for multiple memories from input file backup.hex
avrdude: 1024 bytes eeprom in 1 section [0, 0x3ff]: 256 pages and 0 pad bytes
         writing 1024 bytes to eeprom ...
         writing | ################################################## | 100% 0.00 s 
         reading | ################################################## | 100% 0.00 s 
         1024 bytes of eeprom verified
avrdude: 32768 bytes flash in 1 section [0, 0x7fff]: 256 pages and 0 pad bytes
         writing 32768 bytes to flash ...
         writing | ################################################## | 100% 0.00 s 
         reading | ################################################## | 100% 0.00 s 
         32768 bytes of flash verified
avrdude: 1 byte lfuse in 1 section [0, 0]
         writing 1 byte to lfuse (0x62), 1 byte written, 1 verified
avrdude: 1 byte hfuse in 1 section [0, 0]
         writing 1 byte to hfuse (0xd9), 1 byte written, 1 verified
avrdude: 1 byte efuse in 1 section [0, 0]
         writing 1 byte to efuse (0xf7), 1 byte written, 1 verified
avrdude: 1 byte lock in 1 section [0, 0]
         writing 1 byte to lock (0xff), 1 byte written, 1 verified

avrdude done.  Thank you.

Notice that the mockup also shows the single-byte fuses that would be written? But even better you could ask for the config:

$  avrdude -qqc dryrun -p m328pb -U all:w:backup.hex -T config
config sut_cksel=intrcosc_8mhz_6ck_14ck_65ms # 34
config ckout=co_disabled # 1
config ckdiv8=by_8 # 0
config bootrst=application # 1
config bootsz=bs_2048w # 0
config eesave=ee_erased # 1
config wdton=wdt_programmable # 1
config spien=isp_enabled # 0
config dwen=dw_off # 1
config rstdisbl=external_reset # 1
config bodlevel=bod_disabled # 7
config cfd=cfd_disabled # 0
config lb=no_lock # 3
config blb0=no_lock_in_app # 3
config blb1=no_lock_in_boot # 3

And this before a single electron touches your board!