Arduino ide does not put variant files in the archive and Sloeber does

[RickKimball]

I'm using the core from https://github.com/rogerclarkmelbourne/Arduino_STM32, when I compile and link in the Arduino IDE it produces a good binary. When compiled and linked in sloeber it doesn't produce the same results.

I narrowed it down to the way it links. With the Arduino IDE, all the .o files from the variant directory end up in the {object_files} array. With sloeber, those files get stuff into the archive file. When it links, you can see that it is not placing the variant .o files in the proper order:

arduino link command:
"/home/user1/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++" -Os -Wl,--gc-sections -mcpu=cortex-m3 "-T/home/user1/Arduino/hardware/arduino_stm32/STM32F1/variants/generic_stm32f103c/ld/bootloader_20.ld" "-Wl,-Map,/tmp/arduino_build_476970/zloe1.ino.map" "-L/home/user1/Arduino/hardware/arduino_stm32/STM32F1/variants/generic_stm32f103c/ld" -o "/tmp/arduino_build_476970/zloe1.ino.elf" "-L/tmp/arduino_build_476970" -lm -lgcc -mthumb -Wl,--cref -Wl,--check-sections -Wl,--gc-sections -Wl,--unresolved-symbols=report-all -Wl,--warn-common -Wl,--warn-section-align -Wl,--warn-unresolved-symbols -Wl,--start-group "/tmp/arduino_build_476970/sketch/zloe1.ino.cpp.o" "/tmp/arduino_build_476970/core/wirish/start.S.o" "/tmp/arduino_build_476970/core/wirish/start_c.c.o" "/tmp/arduino_build_476970/core/wirish/syscalls.c.o" "/tmp/arduino_build_476970/core/board.cpp.o" "/tmp/arduino_build_476970/core/wirish/boards.cpp.o" "/tmp/arduino_build_476970/core/wirish/boards_setup.cpp.o" "/tmp/arduino_cache_244275/core/core_arduino_stm32_STM32F1_genericSTM32F103C_device_variant_STM32F103C8,upload_method_DFUUploadMethod,cpu_speed_speed_72mhz,opt_osstd_f86f3cb45e926d46fff8a1442c383d28.a" -Wl,--end-group

sloeber link command:
"/home/user1/sloeber//arduinoPlugin/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++" -Os -Wl,--gc-sections -mcpu=cortex-m3 "-T/home/user1/Arduino/hardware/arduino_stm32/STM32F1/variants/generic_stm32f103c/ld/bootloader_20.ld" "-Wl,-Map,/home/user1/Documents/sloeber-workspace/zloe1/Release/zloe1.map" "-L/home/user1/Arduino/hardware/arduino_stm32/STM32F1/variants/generic_stm32f103c/ld" -o "/home/user1/Documents/sloeber-workspace/zloe1/Release/zloe1.elf" "-L/home/user1/Documents/sloeber-workspace/zloe1/Release" -lm -lgcc -mthumb -Wl,--cref -Wl,--check-sections -Wl,--gc-sections -Wl,--unresolved-symbols=report-all -Wl,--warn-common -Wl,--warn-section-align -Wl,--warn-unresolved-symbols -Wl,--start-group ./sloeber.ino.cpp.o /home/user1/Documents/sloeber-workspace/zloe1/Release/arduino.ar -Wl,--end-group Finished building: zloe1.elf

The difference between the two is that the variant object files (start.S.o, start_c.c.o, syscalls.c.o ... etc) get put into the archive when using sloeber and aren't individually provided via the {object_files} This causes the sketch to fail because some of the functions in those .o files are marked weak so the user can override them in their sketch. However when the user doesn't override them, and sloeber puts them in an archive file, they end up coming from the libc library instead. The functions from the variant *.o files are meant to override the newlib functions like (_sbrk(), _open(), _close() ... etc).

I can get around this problem by copying the variant *.c files into my sketch directory. However, it doesn't seem like sloeber is mimicking arduino-builder behavior in this case.

I have the "aggressively cache" flag checked in the Arduino IDE and the variant files are always provided as individual .o files.

At that point I started looking at the arduino-builder source. It always seems to puts the files from the variant in the object_files list not in the archive file.

In the arduino-builder you can see the function:
archiveFile, objectFiles, err := compileCore(ctx, coreBuildPath, coreBuildCachePath, buildProperties)
https://github.com/arduino/arduino-builder/blob/3f9e2f06c6346b559c20f9e3591f14966cd33097/phases/core_builder.go#L62

You can see that the variantFiles are placed in the objectFiles, not the archiveFiles:
return archiveFile, variantObjectFiles, nil
https://github.com/arduino/arduino-builder/blob/3f9e2f06c6346b559c20f9e3591f14966cd33097/phases/core_builder.go#L133

[jantje]

I know. The core/variants folder should not be put in the archive. Most boards do not have a problem with this but some do.

Sloeber does this now hard coded.
A fix is needed at 2 locations
https://github.com/Sloeber/arduino-eclipse-plugin/blob/master/io.sloeber.core/src/io/sloeber/core/toolchain/ArchiveNameProvider.java#L30
https://github.com/Sloeber/arduino-eclipse-plugin/blob/master/io.sloeber.core/src/io/sloeber/core/toolchain/LinkNameProvider.java#L29

[stevstrong]

Arduino solution: https://github.com/arduino/ArduinoCore-sam/blob/master/platform.txt#L38

[jantje]

@stevstrong
I don't see the relationship between the issue and the code you refer to.

[jantje]

I'm currently testing the fix.
This fix should also fix the "UDOO Neo (Cortex M4)" build issues.

[stevstrong]

You probably figured out, those functions from my link are the ones in syscalls.c mentioned by Rick.

[jantje]

You probably figured out, those functions from my link are the ones in syscalls.c mentioned by Rick.

No I still have not figured it out yet. I fear I need more explanation to figure it out. (read I need it spelled out ;-) )

[jantje]

I have been diving a little deeper into this issue and this is my current understanding:
1)https://github.com/rogerclarkmelbourne/Arduino_STM32 does not provide a json file.
2)A problem at hand is that files in the variants folder are added to the archive (which on its turn is send to the linker) in Sloeber; where Arduino IDE adds them directly to the linker (bypassing the archiving)
3) Previous testing has shown that the board "UDOO Neo (Cortex M4)" has problems with 2 as well (basically they put the core in the variants folder to avoid it to be archived before linking)
4) I assume @stevstrong sees another issue related to toolchain commands

I'm currently testing the fix for 2. Because of 1 I can not add the board to my backwards compatibility test.
Because of 3 I can test the fix fixes issue 2
FYI: For backwards compatibility 564 boards are tested in total and a bit less than 500 build successfully. It takes a long time to run on my dual core low power consumption I5 intel processor.
If the test is successful I will add the fix to the nightly.
At that point in time I need you guys to update your Sloeber's to the nightly and look at 4.

[RickKimball]

If you check it in, or provide a patch I'll test it. I just got the source building and running.

[RickKimball]

@stevstrong has the same problem. He just chose to fix it differently.

[jantje]

@stevstrong has the same problem. He just chose to fix it differently.

OK. I'm not good with gcc compile commands so it simply didn't make sense to me.
Sloeber compilation should be close to what Arduino IDE does so this is a bug that needs fixing.
The test ran by now but I have to look at the tests results. From a high level they look good 500 tests ok :-)

[jantje]

I pushed the fix. if all goes well the build engine will pick it up tonight and tomorrow it will be available on the nightly update site.

[RickKimball]

I recompiled the plugin from source and it all works as expected now! Thanks!

The sloeber link now looks correctly like the arduino one:
"/mnt/vbox/shared/github/arduino-eclipse-plugin/io.sloeber.product/target/products/io.sloeber.product/linux/gtk/x86_64/sloeber//arduinoPlugin/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++" -Os -Wl,--gc-sections -mcpu=cortex-m3 "-T/home/user1/Arduino/hardware/arduino_stm32/STM32F1/variants/generic_stm32f103c/ld/jtag.ld" "-Wl,-Map,/home/user1/Documents/sloberdev_workspace/devtest/Release/devtest.map" "-L/home/user1/Arduino/hardware/arduino_stm32/STM32F1/variants/generic_stm32f103c/ld" -o "/home/user1/Documents/sloberdev_workspace/devtest/Release/devtest.elf" "-L/home/user1/Documents/sloberdev_workspace/devtest/Release" -lm -lgcc -mthumb -Wl,--cref -Wl,--check-sections -Wl,--gc-sections -Wl,--unresolved-symbols=report-all -Wl,--warn-common -Wl,--warn-section-align -Wl,--warn-unresolved-symbols -Wl,--start-group ./sloeber.ino.cpp.o ./core/variant/wirish/boards.cpp.o ./core/variant/wirish/boards_setup.cpp.o ./core/variant/wirish/start.S.o ./core/variant/wirish/start_c.c.o ./core/variant/wirish/syscalls.c.o ./core/variant/board.cpp.o /home/user1/Documents/sloberdev_workspace/devtest/Release/arduino.ar -Wl,--end-group
Thanks for your prompt attention!

[jantje]

good :-)