Unexpanded properties no longer deleted from recipe.preproc.macros

[per1234]

Describe the problem

The Arduino "properties" data format used by the boards platform configuration files provides a templating (AKA "pattern" or "recipe") system where properties can be referenced using a {<property name>} syntax. These references are "expanded" to the value of the referenced property.

If no property of the referenced name was set, the reference is left in the string following the expansion (e.g., if a property named bar was not set, expanding the string "foo {bar} baz" will result in "foo {bar} baz"), but it is also possible to delete any unexpanded references from a string (e.g., adding a deletion operation to the previous example would result in "foo baz").

Although unexpanded references are generally not deleted by the platform system, deletion is done on the commands generated by certain command patterns (AKA "recipes"). Previously this was done for the recipe.preproc.macros pattern.

🐛 Unexpanded reference deletion is no longer done for the recipe.preproc.macros pattern. This will break platforms that relied on the deletion for some reason.

To reproduce

$ git clone https://github.com/arduino/arduino-cli

[...]

$ cd arduino-cli

$ git checkout 0585435f~1  # checkout the last commit before the regression

$ task build

[...]

$ ./arduino-cli version

arduino-cli.exe  Version: git-snapshot Commit: 9ce2904b Date: 2023-08-11T08:22:02Z

$ export ARDUINO_DIRECTORIES_DATA="/tmp/recipe-preproc-macros-unexpanded-property-demo"  # Use a throwaway directories.data for the demo

$ ./arduino-cli core install arduino:[email protected]  # Install an arbitrary platform to use for the demo

[...]

$ echo 'recipe.preproc.macros={nonexistent_property} "{compiler.path}{compiler.cpp.cmd}" {compiler.cpp.flags} {preproc.macros.flags} -mmcu={build.mcu} -DF_CPU={build.f_cpu} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} {compiler.cpp.extra_flags} {build.extra_flags} {includes} "{source_file}" -o "{preprocessed_file_path}"' > "${ARDUINO_DIRECTORIES_DATA}/packages/arduino/hardware/avr/1.8.6/platform.local.txt"  # Override the platform's `recipe.preproc.macros` property with one that has a reference to an nonexistent property

$ ./arduino-cli sketch new /tmp/FooSketch  # Create an arbitrary sketch to compile

$ ./arduino-cli compile --fqbn arduino:avr:uno /tmp/FooSketch  # compilation is successful due to deletion of the unexpanded property from the `recipe.preproc.macros` command

Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.

Used platform Version Path
arduino:avr   1.8.6   C:\Users\per\AppData\Local\Temp\recipe-preproc-macros-unexpanded-property-demo\packages\arduino\hardware\avr\1.8.6

$ git checkout master  # checkout a version of Arduino CLI with the regression

$ task build

[...]

$ ./arduino-cli version

arduino-cli.exe  Version: git-snapshot Commit: 246adf95 Date: 2023-08-11T08:00:04Z

$ ./arduino-cli compile --fqbn arduino:avr:uno --verbose /tmp/FooSketch

FQBN: arduino:avr:uno
Using board 'uno' from platform in folder: C:\Users\per\AppData\Local\Temp\recipe-preproc-macros-unexpanded-property-demo\packages\arduino\hardware\avr\1.8.6
Using core 'arduino' from platform in folder: C:\Users\per\AppData\Local\Temp\recipe-preproc-macros-unexpanded-property-demo\packages\arduino\hardware\avr\1.8.6

Detecting libraries used...
{nonexistent_property} C:\Users\per\AppData\Local\Temp\recipe-preproc-macros-unexpanded-property-demo\packages\arduino\tools\avr-gcc\7.3.0-atmel3.6.1-arduino7/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR -IC:\Users\per\AppData\Local\Temp\recipe-preproc-macros-unexpanded-property-demo\packages\arduino\hardware\avr\1.8.6\cores\arduino -IC:\Users\per\AppData\Local\Temp\recipe-preproc-macros-unexpanded-property-demo\packages\arduino\hardware\avr\1.8.6\variants\standard C:\Users\per\AppData\Local\Temp\arduino\sketches\2405C4C9F90028537C79B0744BCE536E\sketch\FooSketch.ino.cpp -o nul


Used platform Version Path
arduino:avr   1.8.6   C:\Users\per\AppData\Local\Temp\recipe-preproc-macros-unexpanded-property-demo\packages\arduino\hardware\avr\1.8.6
Error during build: exec: "{nonexistent_property}": executable file not found in %PATH%

🐛 Compilation now fails when there is a reference to a nonexistent property in the platform's recipe.preproc.macros property.

Expected behavior

Document change in handling of recipe.preproc.macros in upgrading guide and changelog.

-OR-

Restore historical behavior by going back to deleting unexpanded properties in the command generated from recipe.preproc.macros

Arduino CLI version

246adf9

Operating system

All

Operating system version

Any

Additional context

I bisected the regression to 0585435 (does not occur when using the build from the previous commit 9ce2904).

0585435 is described as a refactoring that is not expected to change behavior or be breaking (#2194), which leads me to believe this was unintentional.

---

A significant real world example of a platform impacted by this change is the teensy:avr boards platform (CC: @PaulStoffregen).

Its recipe.preproc.macros property:

recipe.preproc.macros="{compiler.path}{build.toolchain}{build.command.g++}" -E -CC -x c++ -w {compiler.cpp.flags} {build.flags.common} {build.flags.cpp} {build.flags.cpu} {build.flags.defs} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DF_CPU={build.fcpu} -D{build.usbtype} -DLAYOUT_{build.keylayout} {includes} "{source_file}" -o "{preprocessed_file_path}"

references a nonexistent property compiler.cpp.flags, which causes compilation for any Teensy board to fail:

$ ./arduino-cli compile --fqbn teensy:avr:teensy41 --verbose /tmp/FooSketch
FQBN: teensy:avr:teensy41
Using board 'teensy41' from platform in folder: C:\Users\per\AppData\Local\Arduino15\packages\teensy\hardware\avr\1.58.1
Using core 'teensy4' from platform in folder: C:\Users\per\AppData\Local\Arduino15\packages\teensy\hardware\avr\1.58.1

Detecting libraries used...
C:\Users\per\AppData\Local\Arduino15\packages\teensy\tools\teensy-compile\11.3.1/arm/bin/arm-none-eabi-g++ -E -CC -x c++ -w {compiler.cpp.flags} -g -Wall -ffunction-sections -fdata-sections -nostdlib -std=gnu++14 -fno-exceptions -fpermissive -fno-rtti -fno-threadsafe-statics -felide-constructors -Wno-error=narrowing -mthumb -mcpu=cortex-m7 -mfloat-abi=hard -mfpu=fpv5-d16 -D__IMXRT1062__ -DTEENSYDUINO=158 -DARDUINO=10607 -DARDUINO_TEENSY41 -DF_CPU=600000000 -DUSB_SERIAL -DLAYOUT_US_ENGLISH -IC:\Users\per\AppData\Local\Arduino15\packages\teensy\hardware\avr\1.58.1\cores\teensy4 C:\Users\per\AppData\Local\Temp\arduino\sketches\2405C4C9F90028537C79B0744BCE536E\sketch\FooSketch.ino.cpp -o nul
arm-none-eabi-g++: fatal error: cannot specify '-o' with '-c', '-S' or '-E' with multiple files
compilation terminated.
Alternatives for cannot specify '-o' with '-c', '-S' or '-E' with multiple files: []
ResolveLibrary(cannot specify '-o' with '-c', '-S' or '-E' with multiple files)
  -> candidates: []
arm-none-eabi-g++: fatal error: cannot specify '-o' with '-c', '-S' or '-E' with multiple files
compilation terminated.


Used platform Version Path
teensy:avr    1.58.1  C:\Users\per\AppData\Local\Arduino15\packages\teensy\hardware\avr\1.58.1
Error during build: exit status 1

The presence of this reference could be considered a bug in the teensy:avr platform and could easily be fixed either by removing the reference from the property:

recipe.preproc.macros="{compiler.path}{build.toolchain}{build.command.g++}" -E -CC -x c++ -w {build.flags.common} {build.flags.cpp} {build.flags.cpu} {build.flags.defs} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DF_CPU={build.fcpu} -D{build.usbtype} -DLAYOUT_{build.keylayout} {includes} "{source_file}" -o "{preprocessed_file_path}"

Or if there is use of this reference as an "extra flags" type of application, where users are using it to inject arbitrary flags into the generated command, by adding a fallback property value to platform.txt:

compiler.cpp.flags=

Workaround

Remove references to nonexistent properties from the recipe.preproc.macros property in your platform.txt file.

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the nightly build
• My report contains all necessary details