Using github o update IDF SDK on Arduino IDE leads to compilation errors

[RoCorbera]

Hardware:

Board: ESP32 Dev Module
Core Installation version: 2,0,0
IDE name: Arduino IDE 1.8.15
Flash Frequency: 80MHz
PSRAM enabled: no
Upload Speed: 115200?
Computer OS: Windows 10

Description:

When replacing Arduino 2.0.0 with latest Master, I got an error.

Steps to reproduce:
a) Install Arduino ESP32 Board with V2.0.0
b) Clone Arduino github on a separated directory
c) Copy contents from GitHub cloned folder /tools/sdk/esp32/* to PC Arduino IDE folder Arduino15/packages/esp32/hardware/esp32/2.0.0/tools/sdk/esp32/
d) At this point the IDF sdk is now updated to the latest github master branch. All set to build an application.
e) Compile any sketch and get an error message as in Debug Messages below:

Sketch:

//Any sketch, even an empty sketch such as
#include <Arduino.h>

void setup() {
}

void loop() {
}

Debug Messages:

In file included from C:\Users\rocor\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0/tools/sdk/esp32/include/freertos/include/freertos/FreeRTOS.h:57,
                 from C:\Users\rocor\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0\cores\esp32/Arduino.h:33,
                 from C:\Users\rocor\AppData\Local\Temp\arduino_build_735887\sketch\Blink.ino.cpp:1:
C:\Users\rocor\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0/tools/sdk/esp32/include/freertos/include/freertos/FreeRTOSConfig.h:78:2: error: #error "Needs architecture-speific FreeRTOSConfig.h!"
 #error "Needs architecture-speific FreeRTOSConfig.h!"
  ^~~~~
In file included from C:\Users\rocor\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0\cores\esp32/Arduino.h:34,
                 from C:\Users\rocor\AppData\Local\Temp\arduino_build_735887\sketch\Blink.ino.cpp:1:
C:\Users\rocor\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0/tools/sdk/esp32/include/freertos/include/freertos/task.h:3454:10: fatal error: freertos/task_snapshot.h: No such file or directory
 #include "freertos/task_snapshot.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
exit status 1
Error compiling for board ESP32 Dev Module.

[RoCorbera]

To solve this issue it's necessary to copy:

Arduino15\packages\esp32\hardware\esp32\2.0.0\tools\sdk\esp32\include\freertos\include\esp_additions\freertos_tasks_c_additions.h

and

Arduino15\packages\esp32\hardware\esp32\2.0.0\tools\sdk\esp32\include\freertos\include\esp_additions\freertos\task_snapshot.h

to this folder:
Arduino15\packages\esp32\hardware\esp32\2.0.0\tools\sdk\esp32\include\freertos\include\freertos

[chegewara]

How did you replace v2.0.0 with master?
Maybe you got git conflicts and not updated arduino-esp32 properly?
I just tested master branch which i am updating with git clone/pull and i dont have any errors.

[RoCorbera]

@chegewara, The steps to reproduce the issue are in the description.

[lbernstone]

Step c is incorrect. The github version should be put in your sketchbook folder. This is typically in your Documents folder on Windows, so c:\users\username\Documents\Arduino\hardware\espressif\esp32
https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html#windows

[RoCorbera]

I want to update IDF libraries as the same as in Master latest.
I'm using Windows 10 and installing ESP32 from board manager in the IDE.
It installs Arduino folders in C:\Users\rocor\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0\

In the step to reproduce the issue, I just refer to Arduino15/.... to make it easier.

The link you pointed talks about [ARDUINO_SKETCHBOOK_DIR]/hardware/espressif/esp32 --- it doesn't exist in my Windows.
Maybe https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html#windows is outdated?

[lbernstone]

Yes, don't do that. Let board manager manage its files. You can manage the ones in your sketchbook folder. Make the espressif folder and clone esp32 into it.

[RoCorbera]

@lbernstone, I did exatcly as you said.
Deleted and reinstalled 2.0.0 again using Arduino IDE Board Manager - this step restored the files back to what it was.
Created folder C:\Users\rocor\Documents\Arduino\hardware\espressif\esp32 and cloned Arduino-Esp32 github to it.
C:\Users\rocor\Documents\Arduino\ is my sketchbook folder.

Compiled the sketch from #5107 and tested it ---> it didn't work at all.
When I follow the steps I described in this issue, #5107 works because I force Arduino to link using latest IDF libxxxx.a

Looking into the log from Arduino IDE while compiling and linking, I only see references to C:\Users\rocor\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0\ folder. None to the sketchbook folder.

My issue is regarding Linking the latest IDF libxxx.a files found in C:\Users\rocor\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0\tools\sdk\esp32\lib folder.

For some reason https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html#windows doesn't work.
I use IDE 1.8.15. - the right version as per the instructions.

[RoCorbera]

This is the linking command line from Arduino IDE:

Linking everything together...
"C:\\Users\\rocor\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\xtensa-esp32-elf-gcc\\gcc8_4_0-esp-2021r1/bin/xtensa-esp32-elf-g++" "-Wl,--Map=C:\\Users\\rocor\\AppData\\Local\\Temp\\arduino_build_491404/issue_5107_Serial_Wakeup_LightSleep.ino.map" "-LC:\\Users\\rocor\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\2.0.0/tools/sdk/esp32/lib" "-LC:\\Users\\rocor\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\2.0.0/tools/sdk/esp32/ld" -T esp32.rom.redefined.ld -T memory.ld -T sections.ld -T esp32.rom.ld -T esp32.rom.api.ld -T esp32.rom.libgcc.ld -T esp32.rom.newlib-data.ld -T esp32.rom.syscalls.ld -T esp32.peripherals.ld -mlongcalls -Wno-frame-address -Wl,--cref -Wl,--gc-sections -fno-rtti -fno-lto -u _Z5setupv -u _Z4loopv -Wl,--wrap=mbedtls_mpi_exp_mod -u esp_app_desc -u pthread_include_pthread_impl -u pthread_include_pthread_cond_impl -u pthread_include_pthread_local_storage_impl -u ld_include_panic_highint_hdl -u start_app -u start_app_other_cores -u __ubsan_include -Wl,--wrap=longjmp -u __assert_func -u vfs_include_syscalls_impl -Wl,--undefined=uxTopUsedPriority -u app_main -u newlib_include_heap_impl -u newlib_include_syscalls_impl -u newlib_include_pthread_impl -u __cxa_guard_dummy -DESP32 -DCORE_DEBUG_LEVEL=0 -DARDUINO_RUNNING_CORE=1 -DARDUINO_EVENT_RUNNING_CORE=1 -DARDUINO_USB_CDC_ON_BOOT=0 -Wl,--start-group "C:\\Users\\rocor\\AppData\\Local\\Temp\\arduino_build_491404\\sketch\\issue_5107_Serial_Wakeup_LightSleep.ino.cpp.o" "C:\\Users\\rocor\\AppData\\Local\\Temp\\arduino_build_491404\\core\\core.a" -lesp_ringbuf -lefuse -lesp_ipc -ldriver -lesp_pm -lmbedtls -lapp_update -lbootloader_support -lspi_flash -lnvs_flash -lpthread -lesp_gdbstub -lespcoredump -lesp_phy -lesp_system -lesp_rom -lhal -lvfs -lesp_eth -ltcpip_adapter -lesp_netif -lesp_event -lwpa_supplicant -lesp_wifi -llwip -llog -lheap -lsoc -lesp_hw_support -lxtensa -lesp_common -lesp_timer -lfreertos -lnewlib -lcxx -lapp_trace -lasio -lbt -lcbor -lunity -lcmock -lcoap -lconsole -lnghttp -lesp-tls -lesp_adc_cal -lesp_hid -ltcp_transport -lesp_http_client -lesp_http_server -lesp_https_ota -lesp_lcd -lprotobuf-c -lprotocomm -lmdns -lesp_local_ctrl -lsdmmc -lesp_serial_slave_link -lesp_websocket_client -lexpat -lwear_levelling -lfatfs -lfreemodbus -ljsmn -ljson -llibsodium -lmqtt -lopenssl -lperfmon -lspiffs -lulp -lwifi_provisioning -lbutton -ljson_parser -ljson_generator -lesp_schedule -lesp_rainmaker -lqrcode -lws2812_led -lesp-dsp -lesp-face -lesp32-camera -lesp_littlefs -lfb_gfx -lasio -lcbor -lcmock -lunity -lcoap -lesp_hid -lesp_lcd -lesp_local_ctrl -lesp_websocket_client -lexpat -lfreemodbus -ljsmn -llibsodium -lperfmon -lesp_adc_cal -lfatfs -lwear_levelling -lopenssl -lspiffs -lesp_rainmaker -lmqtt -lwifi_provisioning -lprotocomm -lbt -lbtdm_app -lprotobuf-c -lmdns -lconsole -ljson -ljson_parser -ljson_generator -lesp_schedule -lqrcode -lpe -lfd -lfr -ldetection_cat_face -ldetection -ldl -lesp_ringbuf -lefuse -lesp_ipc -ldriver -lesp_pm -lmbedtls -lapp_update -lbootloader_support -lspi_flash -lnvs_flash -lpthread -lesp_gdbstub -lespcoredump -lesp_phy -lesp_system -lesp_rom -lhal -lvfs -lesp_eth -ltcpip_adapter -lesp_netif -lesp_event -lwpa_supplicant -lesp_wifi -llwip -llog -lheap -lsoc -lesp_hw_support -lxtensa -lesp_common -lesp_timer -lfreertos -lnewlib -lcxx -lapp_trace -lnghttp -lesp-tls -ltcp_transport -lesp_http_client -lesp_http_server -lesp_https_ota -lsdmmc -lesp_serial_slave_link -lulp -lmbedtls -lmbedcrypto -lmbedx509 -lcoexist -lcore -lespnow -lmesh -lnet80211 -lpp -lsmartconfig -lwapi -lesp_ringbuf -lefuse -lesp_ipc -ldriver -lesp_pm -lmbedtls -lapp_update -lbootloader_support -lspi_flash -lnvs_flash -lpthread -lesp_gdbstub -lespcoredump -lesp_phy -lesp_system -lesp_rom -lhal -lvfs -lesp_eth -ltcpip_adapter -lesp_netif -lesp_event -lwpa_supplicant -lesp_wifi -llwip -llog -lheap -lsoc -lesp_hw_support -lxtensa -lesp_common -lesp_timer -lfreertos -lnewlib -lcxx -lapp_trace -lnghttp -lesp-tls -ltcp_transport -lesp_http_client -lesp_http_server -lesp_https_ota -lsdmmc -lesp_serial_slave_link -lulp -lmbedtls -lmbedcrypto -lmbedx509 -lcoexist -lcore -lespnow -lmesh -lnet80211 -lpp -lsmartconfig -lwapi -lesp_ringbuf -lefuse -lesp_ipc -ldriver -lesp_pm -lmbedtls -lapp_update -lbootloader_support -lspi_flash -lnvs_flash -lpthread -lesp_gdbstub -lespcoredump -lesp_phy -lesp_system -lesp_rom -lhal -lvfs -lesp_eth -ltcpip_adapter -lesp_netif -lesp_event -lwpa_supplicant -lesp_wifi -llwip -llog -lheap -lsoc -lesp_hw_support -lxtensa -lesp_common -lesp_timer -lfreertos -lnewlib -lcxx -lapp_trace -lnghttp -lesp-tls -ltcp_transport -lesp_http_client -lesp_http_server -lesp_https_ota -lsdmmc -lesp_serial_slave_link -lulp -lmbedtls -lmbedcrypto -lmbedx509 -lcoexist -lcore -lespnow -lmesh -lnet80211 -lpp -lsmartconfig -lwapi -lesp_ringbuf -lefuse -lesp_ipc -ldriver -lesp_pm -lmbedtls -lapp_update -lbootloader_support -lspi_flash -lnvs_flash -lpthread -lesp_gdbstub -lespcoredump -lesp_phy -lesp_system -lesp_rom -lhal -lvfs -lesp_eth -ltcpip_adapter -lesp_netif -lesp_event -lwpa_supplicant -lesp_wifi -llwip -llog -lheap -lsoc -lesp_hw_support -lxtensa -lesp_common -lesp_timer -lfreertos -lnewlib -lcxx -lapp_trace -lnghttp -lesp-tls -ltcp_transport -lesp_http_client -lesp_http_server -lesp_https_ota -lsdmmc -lesp_serial_slave_link -lulp -lmbedtls -lmbedcrypto -lmbedx509 -lcoexist -lcore -lespnow -lmesh -lnet80211 -lpp -lsmartconfig -lwapi -lesp_ringbuf -lefuse -lesp_ipc -ldriver -lesp_pm -lmbedtls -lapp_update -lbootloader_support -lspi_flash -lnvs_flash -lpthread -lesp_gdbstub -lespcoredump -lesp_phy -lesp_system -lesp_rom -lhal -lvfs -lesp_eth -ltcpip_adapter -lesp_netif -lesp_event -lwpa_supplicant -lesp_wifi -llwip -llog -lheap -lsoc -lesp_hw_support -lxtensa -lesp_common -lesp_timer -lfreertos -lnewlib -lcxx -lapp_trace -lnghttp -lesp-tls -ltcp_transport -lesp_http_client -lesp_http_server -lesp_https_ota -lsdmmc -lesp_serial_slave_link -lulp -lmbedtls -lmbedcrypto -lmbedx509 -lcoexist -lcore -lespnow -lmesh -lnet80211 -lpp -lsmartconfig -lwapi -lphy -lesp_phy -lphy -lesp_phy -lphy -lrtc -lxt_hal -lm -lnewlib -lstdc++ -lpthread -lgcc -lcxx -lapp_trace -lgcov -lapp_trace -lgcov -lc -Wl,--end-group -Wl,-EL -o "C:\\Users\\rocor\\AppData\\Local\\Temp\\arduino_build_491404/issue_5107_Serial_Wakeup_LightSleep.ino.elf"

It's easy to see that it links using C:\\Users\\rocor\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\2.0.0/ folder. No reference to SketchBook folder.

[chegewara]

Please accept the fact you are doing it wrong.
I never before tried method @lbernstone described until now and it works:

As you can see there is 2 versions of esp32 boards. One from normal installation and another from custom installation.
Only difference in my setup is that i am using ubuntu 20 and you windows, but arduino IDE does not care about that.

[RoCorbera]

@chegewara, yes, I hadn't seen that there was another board (in sketchbook). Thanks for the screenshots.
It did work fine and #5107 worked correctly.

@lbernstone and @chegewara Thanks for the help!