Eclipse Sloeber - cannot build using 3.1.0 on system without arduino IDE installed; because pre-build needs preferences.txt

[jantje]

While investigating #8803 Sloeber/arduino-eclipse-plugin#1547 , and Sloeber/arduino-eclipse-plugin#1546 I found that the prebuild command needs a preferences.txt file (see logs below).
If the user is on a system that has Sloeber or any other third party IDE installed but has no Arduino IDE (Like on a build server) this preferences.txt will not be found.
Apparently when the py script fails to do a workaround for the Arduino IDE the copy command of the build.opt is not executed resulting in a build failure.

Can you remove the dependency on the preferences.txt?

Thanks
Jantje

Failure during Sloeber Build https://github.com/Sloeber/arduino-eclipse-plugin/actions/runs/3924285135/jobs/6708444570#step:4:2500

Failure during regression testing on windows

18:41:57 **** Incremental Build of configuration Release for project issue1047_Board_Names_Can_Be_used_as_Strings ****
"e:\\platform\\arduinoPlugin\\tools\\make\\make" all 
e:/platform/arduinoPlugin/tools/make/make --no-print-directory pre-build
"e:\platform\arduinoPlugin\packages\esp8266\tools\python3\3.7.2-post1/python3" -I "e:\platform\arduinoPlugin\packages\esp8266\hardware\esp8266\3.1.1/tools/mkbuildoptglobals.py" "e:\platform" 10812 "L:\test\junit-workspace\issue1047_Board_Names_Can_Be_used_as_Strings\Release" "L:\test\junit-workspace\issue1047_Board_Names_Can_Be_used_as_Strings\Release/core/build.opt" "L:\test\junit-workspace\issue1047_Board_Names_Can_Be_used_as_Strings/issue1047_Board_Names_Can_Be_used_as_Strings.globals.h" "e:/platform/arduinoPlugin/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/CommonHFile.h" 

*** File preferences.txt not found on Windows

*** 'preferences.txt' file missing from well known locations.
make[1]: [makefile:134: pre-build] Error 1 (ignored)
' '
e:/platform/arduinoPlugin/tools/make/make --no-print-directory main-build
'Building file: ..\sloeber.ino.cpp'
'Starting C++ compile'
"e:\platform\arduinoPlugin\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec/bin/xtensa-lx106-elf-g++" -D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE -DESP8266 "@L:\test\junit-workspace\issue1047_Board_Names_Can_Be_used_as_Strings\Release/core/build.opt" "-Ie:\platform\arduinoPlugin\packages\esp8266\hardware\esp8266\3.1.1/tools/sdk/include" "-Ie:\platform\arduinoPlugin\packages\esp8266\hardware\esp8266\3.1.1/tools/sdk/lwip2/include" "-Ie:\platform\arduinoPlugin\packages\esp8266\hardware\esp8266\3.1.1/tools/sdk/libc/xtensa-lx106-elf/include" "-IL:\test\junit-workspace\issue1047_Board_Names_Can_Be_used_as_Strings\Release/core" -c "@e:\platform\arduinoPlugin\packages\esp8266\hardware\esp8266\3.1.1/tools/warnings/none-g++"  -Os -g -free -fipa-pta -Werror=return-type -mlongcalls -mtext-section-literals -fno-rtti -falign-functions=4 -std=gnu++17 -MMD -ffunction-sections -fdata-sections -fno-exceptions  -DMMU_IRAM_SIZE=0x8000 -DMMU_ICACHE_SIZE=0x8000  -DNONOSDK22x_190703=1 -DF_CPU=80000000L -DLWIP_OPEN_SRC -DTCP_MSS=536 -DLWIP_FEATURES=1 -DLWIP_IPV6=0   -DARDUINO=10812 -DARDUINO_ESP8266_NODEMCU_ESP12 -DARDUINO_ARCH_ESP8266 "-DARDUINO_BOARD=\"ESP8266_NODEMCU_ESP12\"" -DARDUINO_BOARD_ID=""  -DFLASHMODE_QIO   -I"E:\platform\arduinoPlugin\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266" -I"E:\platform\arduinoPlugin\packages\esp8266\hardware\esp8266\3.1.1\variants\nodemcu" -MMD -MP -MF"sloeber.ino.cpp.d" -MT"sloeber.ino.cpp.o" -D__IN_ECLIPSE__=1 "..\sloeber.ino.cpp" -o "sloeber.ino.cpp.o"
xtensa-lx106-elf-g++: error: L:\test\junit-workspace\issue1047_Board_Names_Can_Be_used_as_Strings\Release/core/build.opt: No such file or directory
make[1]: *** [subdir.mk:20: sloeber.ino.cpp.o] Error 1
make: *** [makefile:90: all] Error 2
"e:/platform/arduinoPlugin/tools/make/make all" terminated with exit code 2. Build might be incomplete.

18:41:58 Build Failed. 3 errors, 0 warnings. (took 1s.284ms)

[mhightower83]

.../tools/mkbuildoptglobals.py brings global build options to the Arduino IDE. If I followed the features described for "Eclipse Sloeber" this has already been addressed and is not needed.

https://arduino-esp8266.readthedocs.io/en/latest/faq/a06-global-build-options.html#custom-build-environments

[jantje]

This is a change that that was introduced after ESP8266 version 3.0.2
When mkbuildoptglobals.py does not find a preferences.txt the opt file is not copied. I think that is a bug in mkbuildoptglobals.py as this makes that proper functioning of ESP8266 is dependent on finding a file that may not be needed.

Simply changing the script so that the copying of the opt file is done before trying to do the workaround should fix this issue.

[mhightower83]

Yes, the feature was added after 3.0.2

The compiler.cache_core setting is needed for mkbuildoptglobals.py to operate optimally and in some cases build reliability can be affected.

With at least two sketches open when the state of compiler.cache_core is not accurately known and you are using a ...globals.h file in one or more sketches, cached results from one build can be inappropriately applied to another sketch's build.

It is possible to cause the core files to always be rebuilt every time; however, that slows down development time. It is better to know there is a problem and have the opportunity to address it.

For reliable builds, you need to know how you build environment-defined compiler.cache_core and pass that info on to mkbuildoptglobals.py through platform.local.txt with lines like these:

mkbuildoptglobals.extra_flags=--no_cache_core
mkbuildoptglobals.extra_flags=--cache_core

Details on other options can be found here.

Or the feature can be disabled as described before.

[jantje]

I have read up on the doc you provided me. Basically this feature introduced after ESP8266 V3.0.2 tries to work around a "difficulty" introduced in Arduino IDE 1.8.2
Sloeber had this "difficulty" in version 1.0 and I fixed this "difficulty" by having the core archive creation inside the sketch creation. (That was a hard nut to crack for me but I did that and it has been working fine since then.)

How I see the corrent situation in ESP8266 talk: Sloeber works with
mkbuildoptglobals.extra_flags=--cache_core
But every sketch has his own persistent cache (The same cache is used after stopping and restarting Sloeber )

Therefore the fact that the workaround can not be applied is not an issue at all for Sloeber. The fact the opt file is not copied is a problem.

My conclusion
As the Arduino IDE needs to have a preferences.txt at a very dedicated (set of) location(s). If the preferences.txt file is not where it should be the mkbuildoptglobals.py script can safely assume it is not started by Arduino IDE and as such does not need to apply the workaround and should not throw an Error but simply continue.

[mhightower83]

But every sketch has his own persistent cache (The same cache is used after stopping and restarting Sloeber )

Since your sketches have a private copy of core.a, I think for this case where core.a is not shared it is safe to use: mkbuildoptglobals.extra_flags=--no_cache_core. The build process will succeed either way; however, a forced rebuild of core files is avoided.

If the preferences.txt file is not where it should be the mkbuildoptglobals.py script can safely assume it is not started by Arduino IDE and as such does not need to apply the workaround and should not throw an Error but simply continue.

After 8 months, my recollection of all the nuances and concerns with this enhancement has faded a bit. My exposure to alternative build platforms has been limited. I felt at the time with the uncertainty of how an "alternative build platform" might be implemented, it was best to fail rather than silently create a possibly errant build. But, allow for methods that could override the failure by hinting at the correct answer.

BTW: If you are using arduino-builder, passing in an earlier version number, {runtime.ide.version}, prior to 10802 would bypass trying to find preferences.txt. For example, the CI platform is using 10607. Also if you remove the -ide-version=... parameter, arduino-builder will default to an old version number prior to aggressive caching.

While composing this, I see you have a solution path already worked out. I'll post this anyway in case it has content useful for others.

[JAndrassy]

@mhightower83 we would like to use the esp8266 Arduino without preferences.txt

[jantje]

, I see you have a solution

I do not have a solution. I have a workaround with some serious drawbacks.

I felt at the time with the uncertainty of how an "alternative build platform" might be implemented, it was best to fail rather than silently create a possibly errant build.

Sloeber workaround is ignoring any error/failure from the pre build actions. Unfortunately I can not do this only for ESP8266 so now any pre build failure will go unnoticed. 😢

Again: 1) As you implement a platform provider in the arduino framework you should not assume you are started by the Arduino IDE. As such throwing an error because you do not find Arduino stuff is not the way to go.
2) Not being able to apply a workaround to one of the framework implementations should not lead to "not doing the core business" (In casu copying the opt file)

[mcspr]

Was closed in #8803