with_bootloader binary merger failes for large binaries

[rikba]

Hi community

We are trying to create the binary with bootloader of an arduino sketch using Arduino IDE 1.8.13 and upload it to an ATSAMD21G18A microprocessor (Arduino Zero) via Atmel Studio 7.0.2397.
It seems to us that the merger that fuses bootloader with sketch code that was fixed in 1.8.13 (arduino/arduino-builder#286, #744) does not work for larger sketches.

The following two sketches have been uploading via IDE and executed successfully.
However, if the binary is uploaded via Atmel Studio, only the small, first sketch works.

10992 bytes sketch, works when uploaded with IDE or Atmel as binary.with_bootloader.bin:

const char hello[13] PROGMEM = "hello world!";

void setup() {
  SerialUSB.begin(115200);
}

void loop() {
  SerialUSB.println(hello);
  delay(1000);                       // wait for a second
}

60976 bytes sketch, works when uploaded with IDE, does not work when uploaded via Atmel as binary.with_bootloader.bin:

const char hello[50000] PROGMEM = "hello world!";

void setup() {
  SerialUSB.begin(115200);
}

void loop() {
  SerialUSB.println(hello);
  delay(1000);                       // wait for a second
}

Without understanding the merging code, maybe this has to do with some maximum size assumptions https://github.com/arduino/arduino-cli/pull/744/files#diff-b0aa01210cafc5319fbc95159b62da0fR72.

[rikba]

As a workaround we can upload the bootloader.hex and program.hex successively.

[facchinm]

@rikba thanks for the report. It looks like the hex merger is maxing out on its own at 65536 bytes (16 bit).
@cmaglie should we add a test in arduino-cli for this? The last sketch shows the problem perfectly.

[cmaglie]

yes this is more an issue for arduino-cli (since the builder is now based on it), I'm moving the issue over there.

[ubidefeo]

While trying to explain a user how not to lose the bootloader when uploading using programmer I bumped into an issue after burning a compiled sketch with bootloader on an Arduino Micro.
#155 (comment)

Programming goes through (using arduinoisp) but then the board's LED breaths (like in bootloader mode) and then starts blinking regularly even thought he Sketch has no blink.
The board becomes visible to IDE/IDE 2/CLI but attempting to upload using the bootloader bricks it and a new program action is required.

Also uploading the sketch with_bootloader took a considerable amount of time, so I checked the file size and of course it does not add up

-rw-r--r--  1 ubi  staff    13B Mar 18 08:06 BlinkMe.ino.eep
-rwxr-xr-x  1 ubi  staff    45K Mar 18 08:06 BlinkMe.ino.elf
-rw-r--r--  1 ubi  staff    11K Mar 18 08:06 BlinkMe.ino.hex
-rw-r--r--  1 ubi  staff    32K Mar 18 08:06 BlinkMe.ino.with_bootloader.bin
-rw-r--r--  1 ubi  staff    88K Mar 18 08:06 BlinkMe.ino.with_bootloader.hex

The compiled sketch (no bootloader) is 11K, and the one with bootloader is 88 :(

[cmaglie]

Programming goes through (using arduinoisp) but then the board's LED breaths (like in bootloader mode) and then starts blinking regularly even thought he Sketch has no blink.

In the comment that you linked I see that you compiled a BlinkMe sketch and after that, you uploaded the BlinkMe.ino.with_bootloader.hex so you have actually uploaded a BlinkMe + bootloader. AFAICT this is the expected outcome...

Programming goes through (using arduinoisp) but then the board's LED breaths (like in bootloader mode) and then starts blinking regularly even thought he Sketch has no blink.

The big size of the ino.with_boorloader.hex may just be an artifact of the .hex: since the bootloader is in the upper area of the flash you got an .hex file that fills all the 32KB of flash BTW the most part in the middle will be just filled with "empty" space (in this case 0xFF).

[ubidefeo]

@cmaglie

The big size of the ino.with_boorloader.hex may just be an artifact of the .hex: since the bootloader is in the upper area of the flash you got an .hex file that fills all the 32KB of flash BTW the most part in the middle will be just filled with "empty" space (in this case 0xFF).

makes sense now, but this still persists on an Arduino Micro

The board becomes visible to IDE/IDE 2/CLI but attempting to upload using the bootloader bricks it and a new program action is required.

[per1234]

This was fixed in #1017 / #1212

Programming goes through (using arduinoisp) but then the board's LED breaths (like in bootloader mode) and then starts blinking regularly even thought he Sketch has no blink.

This problem is tracked at arduino/ArduinoCore-avr#411

attempting to upload using the bootloader bricks it and a new program action is required.

I did not encounter this when I reproduced arduino/ArduinoCore-avr#411. Although I did experience the problem of the exported sketch program not running when I flashed my board with the "with_bootloader" binary, I was able to upload sketches via the bootloader after that.

However, it is not clear to me exactly what happens in this case where the sketch application binary is merged with a bootloader binary that already has an embedded application. So perhaps under certain conditions this could result in a binary where the bootloader is non-functional?