ninja process hanging forever when building esp32 arduino lib builder with component having ${CMAKE_BINARY_DIR} in INCLUDE_DIRS, idf_component_register (IDFGH-14412)

[bhcuong2008]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.4

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

What is the expected behavior?

Build successfully

What is the actual behavior?

ninja hanging forever at step [2820/2821] Linking CXX executable arduino-lib-builder.elf

Steps to reproduce.

1. Install requirements following https://github.com/espressif/esp32-arduino-lib-builder for tag idf-release_v5.4, for example, at folder /esp32/lib5.4
2. Run initial to install all tools, libraries, esp-idf, and wait until the end
   /esp/lib5.4/build.sh -A idf-release/v5.4 -I release/v5.4 -i v5.4 -t 'esp32s3' -b idf-libs
3. Create test component in directory components inside lib-builder folder, in this folder /esp32/lib5.4/components with command
   source ../esp-idf/export.sh
idf.py create-component test
4. Edit created test/CMakeLists.txt as following:

idf_component_register(
    SRCS "test.c"
    INCLUDE_DIRS "${CMAKE_BINARY_DIR}" "include"
)

5. Save and go to folder /esp32/lib5.4, run command again with option -s to avoid download tools, libs,...again.
   build.sh -s -A idf-release/v5.4 -I release/v5.4 -i v5.4 -t 'esp32s3' -b idf-libs
6. The build process will run and hang at 2820/2821

[2819/2821] Building C object CMakeFiles/arduino-lib-builder.elf.dir/project_elf_src_esp32s3.c.obj
[2820/2821] Linking CXX executable arduino-lib-builder.elf

7. Now, remove "${CMAKE_BINARY_DIR}" in file CMakeLists.txt, and run again, it will be successful.
   build.sh -s -A idf-release/v5.4 -I release/v5.4 -i v5.4 -t 'esp32s3' -b idf-libs

So, why ${CMAKE_BINARY_DIR} or ${BUILD_DIR} causes this hang?

Build or installation Logs.

...
[2799/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Zigbee/src/ZigbeeHandlers.cpp.obj
[2800/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Zigbee/src/ep/ZigbeeDimmableLight.cpp.obj
[2801/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Zigbee/src/ep/ZigbeeSwitch.cpp.obj
[2802/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Zigbee/src/ep/ZigbeeLight.cpp.obj
[2803/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Matter/src/MatterEndpoints/MatterHumiditySensor.cpp.obj
[2804/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Zigbee/src/ep/ZigbeeTempSensor.cpp.obj
[2805/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Zigbee/src/ep/ZigbeeThermostat.cpp.obj
[2806/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Zigbee/src/ep/ZigbeeOccupancySensor.cpp.obj
[2807/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Zigbee/src/ep/ZigbeeFlowSensor.cpp.obj
[2808/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Zigbee/src/ep/ZigbeeCarbonDioxideSensor.cpp.obj
[2809/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Zigbee/src/ep/ZigbeePressureSensor.cpp.obj
[2810/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Matter/src/MatterEndpoints/MatterThermostat.cpp.obj
[2811/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Matter/src/MatterEndpoints/MatterPressureSensor.cpp.obj
[2812/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Matter/src/MatterEndpoints/MatterOnOffPlugin.cpp.obj
[2813/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Matter/src/MatterEndpoints/MatterOccupancySensor.cpp.obj
[2814/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Matter/src/MatterEndpoints/MatterContactSensor.cpp.obj
[2815/2821] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Matter/src/Matter.cpp.obj
[2816/2821] Linking C static library esp-idf/arduino/libarduino.a
[2817/2821] Linking C static library esp-idf/main/libmain.a
[2818/2821] Generating ld/sections.ld
warning: default on the choice symbol SEC_CERT_DAC_PROVIDER (defined at /cfm/esp32/lib5.4_mp1.24/managed_components/espressif__esp_matter/connectedhomeip/connectedhomeip/config/esp32/components/chip/Kconfig:929, /cfm/esp32/lib5.4_mp1.24/managed_components/espressif__esp_matter/components/esp_matter/Kconfig:48) will have no effect, as defaults do not affect choice symbols
warning: the choice symbol SEC_CERT_DAC_PROVIDER (defined at /cfm/esp32/lib5.4_mp1.24/managed_components/espressif__esp_matter/connectedhomeip/connectedhomeip/config/esp32/components/chip/Kconfig:929, /cfm/esp32/lib5.4_mp1.24/managed_components/espressif__esp_matter/components/esp_matter/Kconfig:48) is defined with a prompt outside the choice
info: INFO: Symbol SEC_CERT_DAC_PROVIDER defined in multiple locations (see below). Please check if this is a correct behavior or a random name match:
    /cfm/esp32/lib5.4_mp1.24/managed_components/espressif__esp_matter/components/esp_matter/Kconfig:48
    /cfm/esp32/lib5.4_mp1.24/managed_components/espressif__esp_matter/connectedhomeip/connectedhomeip/config/esp32/components/chip/Kconfig:929
[2819/2821] Building C object CMakeFiles/arduino-lib-builder.elf.dir/project_elf_src_esp32s3.c.obj
[2820/2821] Linking CXX executable arduino-lib-builder.elf

More Information.

No response

[me-no-dev]

It's actually hanging while trying to collect all files for Arduino. Having that directive there is probably making it search through folders that is not prepared to go through and is running in circles. You can check the out folder for what is being put there and maybe add some prints in the build script to try and figure out where is it looping.

[bhcuong2008]

Is there anyway to control this behavior, such as excluding some folders for searching? Because some components could include build dir in INCLUDE_DIRS rather than PRIV_INCLUDE_DIRS, for example, micropython, micropython-ulab use INTERFACE library while building. It refers to runtime created header files inside build dir. So if including build dir (CMAKE_BINARY_DIR) in INCLUDE_DIRS, the whole build process could not finished. I waited for several hours but it is still there.

[me-no-dev]

The tooling here is tailored specifically for what Arduino needs. It extracts the files it also needs from the build dir (sdkconfig.h, bootloaders, etc), so maybe you need to be more specific than just adding the whole build dir to includes.

[bhcuong2008]

Could you help me which codes, in which files, to search through include dirs to collect files for Arduino? I will modify the script to exclude build dir while collecting files but still existing when compile codes. Currently my compiling/linking is successful, but stuck at collecting files phase that you describe.

[bhcuong2008]

I found it in file tools/copy-libs.sh, I will investigate it.

[bhcuong2008]

I tried but ninja still hanging at phase "Linking CXX executable arduino-lib-builder.elf" before executing copy-libs.sh. I don't know whether it is ninja process or arduino build/collect scripts.