Add dfu-util upload method

[chrissbarr]

This allows the dfu-util tool to be used as an upload method if specified in a board's definition. In my testing this is compatible both with the built-in DFU bootloader (only tested on an F446) and a custom software bootloader that implements DFU mode (again, on the F446).

At the moment I'm using it in my WIP branch adding my RUMBA32 board, here. It's partly dependent (on Windows) on the recent PR I submitted in the Arduino_Tools repo updating the dfu-util version (stm32duino/Arduino_Tools#29), as the older version of dfu-util did not seem to work on Windows, and the file structure has been updated slightly.

I haven't added any examples to any of the existing boards, mostly because I'm not sure which boards would benefit from using the DFU upload method - but I could get them added to this PR if anyone has suggestions.

Here are two boards.txt examples from my RUMBA32 fork. The first uses ST's built-in DFU bootloader (so BOOT0 must be held high on reset):

RUMBA32_F446.menu.upload_method.DFUnative=DFU (Native)
RUMBA32_F446.menu.upload_method.DFUnative.upload.protocol=dfuutil_upload
RUMBA32_F446.menu.upload_method.DFUnative.upload.tool=dfuutil_upload
RUMBA32_F446.menu.upload_method.DFUnative.upload.address=0x08000000

And this using my bootloader, which sits in the first page(s) of flash and is supposed to upload the user code to the page starting at 0x08010000:

RUMBA32_F446.menu.upload_method.DFUbootloader=DFU Bootloader (0x10000)
RUMBA32_F446.menu.upload_method.DFUbootloader.upload.protocol=dfuutil_upload
RUMBA32_F446.menu.upload_method.DFUbootloader.upload.tool=dfuutil_upload
RUMBA32_F446.menu.upload_method.DFUbootloader.upload.address=0x08010000
RUMBA32_F446.menu.upload_method.DFUbootloader.build.extra_flags=-D{build.product_line} {build.enable_usb} {build.xSerial} -DVECT_TAB_OFFSET=0x10000
RUMBA32_F446.menu.upload_method.DFUbootloader.build.ldscript=ldscript_0x10000.ld

I'm not sure that my bootloader in particular is any use with this core (I wrote it to set aside space for EEPROM emulation in the STM32GENERIC core - haven't yet investigated if I will still need it with this core), but because the address, VECT_TAB_OFFSET and ldscript can be set in the upload_method definition, this should be adjustable to any other bootloaders people want to use - or just with the built in DFU bootloader common to many STM32 chips.

Questions:

• Can/should we set a default for the address variable? It's likely to be 0x08000000 for most boards, unless they use a custom bootloader.
• What's the difference between upload.protocol and upload.tool in boards.txt? I'm missing something in how they map from the boards.txt to the platform.txt in my examples, and feel I probably shouldn't be repeating the same thing twice.
• Can I append to build.extra_flags, rather than overwriting it (like in my second example), where all I want is to add the DVECT_TAB_OFFSET=0x10000 flag?

Before merging:

• Someone should test the dfu-util upload method (preferably on one or more boards with the built-in bootloader)
• Update dfu-util to v0.8 Arduino_Tools#29 needs to be merged (otherwise Windows upload doesn't work)

Thoughts?

[fpistm]

There is several possibilities for DFU:

• Use the built-in on (available for some series, see AN2606 and AN3156
• Use board with a board "manufacturer" bootloader provided by default (ex maple)
• Use custom bootloader (ex bluepill)

I guess in a first step we should focus on the built-in one.

• Can/should we set a default for the address variable? It's likely to be 0x08000000 for most boards, unless they use a custom bootloader.

I don't think. Default address is ok anyway this could be handled later by using a command wrapper as done using the https://github.com/stm32duino/Arduino_Tools/blob/9d655e1880145895719a2a71e686cc3800ee203c/linux64/maple_upload#L12
Even for BluePill using Roger's bootloader, the address is not passed as an argument I guess the bootloader handle the offset to preserve itself.

• What's the difference between upload.protocol and upload.tool in boards.txt? I'm missing something in how they map from the boards.txt to the platform.txt in my examples, and feel I probably shouldn't be repeating the same thing twice.

upload.tool in board.txt is used to get the right tools to used defined in the platform.txt: tools.{upload.tool}
In the case you proposed:
upload.tool= dfuutil_upload so it search for tools.dfuutil_upload definition.
The upload.protocol is well explained here:
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5-3rd-party-Hardware-specification#sketch-upload-configuration
The same tools can be used but with a different 'protocol', example with avrdude.
The protocol in our case is not used but at mid term I think this could be used to handle this option:

-a, --alt ALT
   Specify the altsetting of the DFU interface by name or by number. 

For the built-in DFU, it requires 0 but for custom bootloader, it requires 2. So using the protocol argument, I think we can handle properly which bootloader is used and avoid to duplicate the tools definition.

• Can I append to build.extra_flags, rather than overwriting it (like in my second example), where all I want is to add the DVECT_TAB_OFFSET=0x10000 flag?
  I think this could be added as new parameters as it is done in rogers core, this will also allow to handle which linker script to use if several possibilities.
  https://github.com/rogerclarkmelbourne/Arduino_STM32/blob/61ed869caa6bf73a4ca28e325e200670fc8ef8e5/STM32F1/boards.txt#L27

[chrissbarr]

Thanks for the detailed reply!

There is several possibilities for DFU:
I guess in a first step we should focus on the built-in one.

I agree, that's probably the most useful thing for most people, and should be the most straightforward - that seems to be working fine on the boards I have tested, but it would be good if anyone else has boards to test.

Even for BluePill using Roger's bootloader, the address is not passed as an argument I guess the bootloader handle the offset to preserve itself.

I think it's good to have the option to set the flash address - the bootloader I'm using needs that, but I don't know if that is common. I'm not sure how Roger's bootloader handles that - perhaps through the ALT flag, as you mentioned.

Thanks for the explanation about upload.tool and upload.protocol - that clears things up.

For the built-in DFU, it requires 0 but for custom bootloader, it requires 2. So using the protocol argument, I think we can handle properly which bootloader is used and avoid to duplicate the tools definition.

I think you're right. The booloader I'm using requires -a 0, the same as the built-in DFU mode - so the current code, like in my examples above, is working for me. Looking at the definitions for the Maple Mini you linked in Roger's core, it looks like they need -a 1 or -a 2, depending on the bootloader version - so I agree it would be good to have that as an option, otherwise many bootloaders will not work.

So, my understanding is that we could change this line:

tools.dfuutil_upload.upload.pattern="{path}/{cmd}" -a 0 -s "{upload.address}:leave" -D "{build.path}/{build.project_name}.bin"

To:

tools.dfuutil_upload.upload.pattern="{path}/{cmd}" -a {upload.protocol} -s "{upload.address}:leave" -D "{build.path}/{build.project_name}.bin"

And then change the boards.txt definition to update the upload.protocol line:

board.menu.upload_method.DFUnative=DFU (Native)
board.menu.upload_method.DFUnative.upload.protocol=0
board.menu.upload_method.DFUnative.upload.tool=dfuutil_upload
board.menu.upload_method.DFUnative.upload.address=0x08000000

And then anyone that is using custom bootloaders can change the upload.protocol and upload.address as they need.

I guess for 90% of boards (those using built-in bootloader) the upload.protocol will be 0, and upload.address will be 0x08000000 - is there a way we can set those as default somewhere else so that they don't need to be repeated for every board unless they are different?

Then DFU upload for most boards with hardware DFU bootloaders would only need the two lines:

board.menu.upload_method.DFUnative=DFU (Native)
board.menu.upload_method.DFUnative.upload.tool=dfuutil_upload

Not sure if that is possible, but it would keep the boards.txt file cleaner.

[fpistm]

So, my understanding is that we could change this line:

tools.dfuutil_upload.upload.pattern="{path}/{cmd}" -a 0 -s "{upload.address}:leave" -D "{build.path}/{build.project_name}.bin"

To:

tools.dfuutil_upload.upload.pattern="{path}/{cmd}" -a {upload.protocol} -s "{upload.address}:leave" -D "{build.path}/{build.project_name}.bin"

And then change the boards.txt definition to update the upload.protocol line:

board.menu.upload_method.DFUnative=DFU (Native)
board.menu.upload_method.DFUnative.upload.protocol=0
board.menu.upload_method.DFUnative.upload.tool=dfuutil_upload
board.menu.upload_method.DFUnative.upload.address=0x08000000

And then anyone that is using custom bootloaders can change the upload.protocol and upload.address as they need.

Right

Then DFU upload for most boards with hardware DFU bootloaders would only need the two lines:

board.menu.upload_method.DFUnative=DFU (Native)
board.menu.upload_method.DFUnative.upload.tool=dfuutil_upload

I think it should be possible as (currently no other upload method use upload.protocol
So to do that, we should add upload.protocol=0 in the platform.txt, this will be used as default value.

I've tested with a Nucleo F429ZI. This work under Linux but not under windows. Anyway under windows I've didn't succeed with STM32CubeProgrammer. Currently, I don't know why and need to investigare as my device is well recognized as STM32DFU.

[chrissbarr]

Right, I've changed the upload pattern to include the protocol flag, so now the a interface can be specified in the boards.txt upload config.

I couldn't figure out setting upload.protocol=0 in the platform.txt file - for some reason the IDE always throws an error when I try that. Is there a specific place in the order of that file it needs to be? Not a big deal if we can't do it, it was just an idea to keep things tidier.

[chrissbarr]

So, now that the latest tools release includes the updated dfu-util for Windows, I think this is usable for DFU upload using the built-in DFU support on some STM32 chips. Custom DFU bootloaders should be supported too, but I haven't tested much with those recently. No reason they shouldn't work though.

I've tested my custom boards using an F446 (RUMBA32 and another one), along with an F407 board, and both are happy uploading via DFU from the IDE provided I jumper BOOT0 when I reset / power-on the board (i.e., the micro needs to be in bootloader mode).

I assume other people are interested in having DFU upload working. Should we see if we can get this merged?

For reference, with this merged, we can enable DFU upload targeting the built-in bootloader on boards like so:

3dprinter.menu.upload_method.dfuutil_upload=DFU Upload
3dprinter.menu.upload_method.dfuutil_upload.upload.protocol=dfuutil_upload
3dprinter.menu.upload_method.dfuutil_upload.upload.tool=dfuutil_upload
3dprinter.menu.upload_method.dfuutil_upload.upload.protocol=0
3dprinter.menu.upload_method.dfuutil_upload.upload.address=0x08000000

That's working for my application - jumper BOOT0, press reset, hit upload, and it works.

It would be great if we could get this merged, as I'm hoping to start shipping RUMBA32 boards now but at the moment will need to tell my customers to manually add in DFU support. I've been holding off on this seeing the progress that has been made in #415, but this PR is working so maybe we should merge it and it can be improved / replaced later if better methods become available.

What do you think @fpistm?

[fpistm]

Hi @chrissbarr
I'm currently on this #445.
So DFU will be available soon.
This is ok for Linux. Still Windows to end and valid also on Mac.

[fpistm]

This PR is superseded by #514