Skip to content

Using most IDF Managed Components instead of git-clone #145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Nov 9, 2023
Merged

Using most IDF Managed Components instead of git-clone #145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
79598ee
Update CMakeLists.txt
SuGlider Nov 6, 2023
3a5b40c
Tool fix
SuGlider Nov 6, 2023
5ceed1d
Update idf_component.yml
SuGlider Nov 6, 2023
597754f
Adds Camera Component
SuGlider Nov 8, 2023
dbea3d4
DL component only for S3
SuGlider Nov 8, 2023
9eb7aea
get Camera component from GH
SuGlider Nov 8, 2023
ddec8a0
Camera component for any Version + Public
SuGlider Nov 8, 2023
07b90dd
Test for SR Models building in CI
SuGlider Nov 8, 2023
6ea0c57
add SR component to the CI
SuGlider Nov 8, 2023
7b4ea7c
fix idf_component.yml format
SuGlider Nov 8, 2023
cbeb969
Reverting not necessary change about ESP-SR component
SuGlider Nov 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,6 @@
# CMakeLists in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.5)

set(RMAKER_PATH ${CMAKE_SOURCE_DIR}/components/esp-rainmaker)
set(INSIGHTS_PATH ${RMAKER_PATH}/components/esp-insights)

set(EXTRA_COMPONENT_DIRS ${INSIGHTS_PATH}/components ${RMAKER_PATH}/components)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(arduino-lib-builder)

Expand All @@ -15,7 +10,7 @@ idf_build_get_property(elf EXECUTABLE GENERATOR_EXPRESSION)
add_custom_command(
OUTPUT "idf_libs"
COMMAND ${CMAKE_SOURCE_DIR}/tools/copy-libs.sh ${IDF_TARGET} "${CONFIG_LIB_BUILDER_FLASHMODE}" "${CONFIG_SPIRAM_MODE_OCT}" "${CONFIG_IDF_TARGET_ARCH_XTENSA}"
DEPENDS all
DEPENDS ${elf}
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
VERBATIM
)
Expand Down
13 changes: 13 additions & 0 deletions build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,19 @@ for target_json in `jq -c '.targets[]' configs/builds.json`; do
idf.py -DIDF_TARGET="$target" -DSDKCONFIG_DEFAULTS="$idf_libs_configs" idf_libs
if [ $? -ne 0 ]; then exit 1; fi

if [ "$target" == "esp32s3" ]; then
idf.py -DIDF_TARGET="$target" -DSDKCONFIG_DEFAULTS="$idf_libs_configs" srmodels_bin
Copy link
Contributor Author

@SuGlider SuGlider Nov 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@me-no-dev - It sounds like CI is failing here... It works in my computer, though! :-)

https://github.com/espressif/esp32-arduino-lib-builder/actions/runs/6776484660/job/18438277663?pr=145

-- Build files have been written to: /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/build
ninja: error: unknown target 'srmodels_bin'
ninja failed with exit code 1, output of the command is in the /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/build/log/idf_py_stderr_output_20115 and /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/build/log/idf_py_stdout_output_20115
Error: Process completed with exit code 1.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not nice. We need a proper workaround

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try to figure out a way to fix CI.

Copy link
Contributor Author

@SuGlider SuGlider Nov 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@me-no-dev - I found out what is the issue here.
In order to add SR component to the CI here (current PR), I need that the PR #8842 is merged, because it adds SR to the Arduino yml file.
After SR is listed in the managed components(from Arduino side), it will run CI corretly here at Lib Builder CI.

As a way to test it, I've added SR to the local Lib Bulder yml file in a "testing" commit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now it shows an Arduino esp32-hal-sr.c error:

-c /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/components/arduino/libraries/ESP_SR/src/esp32-hal-sr.c
/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/components/arduino/libraries/ESP_SR/src/esp32-hal-sr.c:10:2: warning: #warning Compatible partition must be selected for ESP_SR to work [-Wcpp]
10 | #warning Compatible partition must be selected for ESP_SR to work
| ^~~~~~~
/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/components/arduino/libraries/ESP_SR/src/esp32-hal-sr.c:27:10: fatal error: esp_mn_speech_commands.h: No such file or directory
27 | #include "esp_mn_speech_commands.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My suggestion is that we merge Arduino espressif/arduino-esp32#8842 and then check how CI will work here at Lib Builder.

Copy link
Contributor Author

@SuGlider SuGlider Nov 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/espressif/arduino-esp32/actions/runs/6800936891/job/18490657761?pr=8842

Specific CI for the S3 is passing fine in the Arduino PR #8842
It shall run fine here as well.

Eveidence from the CI:
[1237/1323] Building CXX object esp-idf/arduino-esp32/CMakeFiles/__idf_arduino-esp32.dir/libraries/ESP_SR/src/ESP_SR.cpp.obj
With no error.

if [ $? -ne 0 ]; then exit 1; fi
AR_SDK="$AR_TOOLS/esp32-arduino-libs/$target"
# sr model.bin
if [ -f "build/srmodels/srmodels.bin" ]; then
echo "$AR_SDK/esp_sr"
mkdir -p "$AR_SDK/esp_sr"
cp -f "build/srmodels/srmodels.bin" "$AR_SDK/esp_sr/"
cp -f "partitions.csv" "$AR_SDK/esp_sr/"
fi
fi

# Build Bootloaders
for boot_conf in `echo "$target_json" | jq -c '.bootloaders[]'`; do
bootloader_configs="$main_configs"
Expand Down
21 changes: 17 additions & 4 deletions main/idf_component.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,23 @@
dependencies:
# Required IDF version
idf: ">=5.1"

mdns: "^1.1.0"
chmorgan/esp-libhelix-mp3: "1.0.3"
esp-dsp: "^1.3.4"
espressif/esp32-camera:
version: "*"
git: https://github.com/espressif/esp32-camera.git
require: public
espressif/esp-tflite-micro:
version: ">=1.2.0"
require: public
espressif/esp-dl:
version: ">=2.0.0"
path: ../components/esp-dl
require: public
rules:
- if: "target in [esp32s3]"
espressif/esp-sr:
Copy link
Contributor Author

@SuGlider SuGlider Nov 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This component (esp-sr) is already in Arduino component list from PR #8842.
As soon as the #8842 is merged, I'll remove it and make it run CI again.

version: ">=1.4.2"
rules:
- if: "target in [esp32s3]"

# esp-sr: "^1.3.1"
# esp32-camera: "^2.0.4"
Expand Down
7 changes: 0 additions & 7 deletions tools/copy-libs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,13 +478,6 @@ echo -n "$LD_FLAGS" > "$FLAGS_DIR/ld_flags"
echo -n "$LD_SCRIPTS" > "$FLAGS_DIR/ld_scripts"
echo -n "$AR_LIBS" > "$FLAGS_DIR/ld_libs"

# sr model.bin
if [ -f "build/srmodels/srmodels.bin" ]; then
mkdir -p "$AR_SDK/esp_sr"
cp -f "build/srmodels/srmodels.bin" "$AR_SDK/esp_sr/"
cp -f "partitions.csv" "$AR_SDK/esp_sr/"
fi

# sdkconfig
cp -f "sdkconfig" "$AR_SDK/sdkconfig"

Expand Down
126 changes: 63 additions & 63 deletions tools/update-components.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,107 +13,107 @@ TFLITE_REPO_URL="https://github.com/espressif/tflite-micro-esp-examples.git"
#
# CLONE/UPDATE ESP32-CAMERA
#
echo "Updating ESP32 Camera..."
if [ ! -d "$AR_COMPS/esp32-camera" ]; then
git clone $CAMERA_REPO_URL "$AR_COMPS/esp32-camera"
else
git -C "$AR_COMPS/esp32-camera" fetch && \
git -C "$AR_COMPS/esp32-camera" pull --ff-only
fi
if [ $? -ne 0 ]; then exit 1; fi
#echo "Updating ESP32 Camera..."
#if [ ! -d "$AR_COMPS/esp32-camera" ]; then
# git clone $CAMERA_REPO_URL "$AR_COMPS/esp32-camera"
#else
# git -C "$AR_COMPS/esp32-camera" fetch && \
# git -C "$AR_COMPS/esp32-camera" pull --ff-only
#fi
#if [ $? -ne 0 ]; then exit 1; fi

#
# CLONE/UPDATE ESP-DL
#
echo "Updating ESP-DL..."
if [ ! -d "$AR_COMPS/esp-dl" ]; then
git clone $DL_REPO_URL "$AR_COMPS/esp-dl"
#this is a temp measure to fix build issue
mv "$AR_COMPS/esp-dl/CMakeLists.txt" "$AR_COMPS/esp-dl/CMakeListsOld.txt"
echo "idf_build_get_property(target IDF_TARGET)" > "$AR_COMPS/esp-dl/CMakeLists.txt"
echo "if(NOT \${IDF_TARGET} STREQUAL \"esp32c6\" AND NOT \${IDF_TARGET} STREQUAL \"esp32h2\")" >> "$AR_COMPS/esp-dl/CMakeLists.txt"
cat "$AR_COMPS/esp-dl/CMakeListsOld.txt" >> "$AR_COMPS/esp-dl/CMakeLists.txt"
echo "endif()" >> "$AR_COMPS/esp-dl/CMakeLists.txt"
rm -rf "$AR_COMPS/esp-dl/CMakeListsOld.txt"
git clone $DL_REPO_URL "$AR_COMPS/esp-dl"
#this is a temp measure to fix build issue
mv "$AR_COMPS/esp-dl/CMakeLists.txt" "$AR_COMPS/esp-dl/CMakeListsOld.txt"
echo "idf_build_get_property(target IDF_TARGET)" > "$AR_COMPS/esp-dl/CMakeLists.txt"
echo "if(NOT \${IDF_TARGET} STREQUAL \"esp32c6\" AND NOT \${IDF_TARGET} STREQUAL \"esp32h2\")" >> "$AR_COMPS/esp-dl/CMakeLists.txt"
cat "$AR_COMPS/esp-dl/CMakeListsOld.txt" >> "$AR_COMPS/esp-dl/CMakeLists.txt"
echo "endif()" >> "$AR_COMPS/esp-dl/CMakeLists.txt"
rm -rf "$AR_COMPS/esp-dl/CMakeListsOld.txt"
else
git -C "$AR_COMPS/esp-dl" fetch && \
git -C "$AR_COMPS/esp-dl" pull --ff-only
git -C "$AR_COMPS/esp-dl" fetch && \
git -C "$AR_COMPS/esp-dl" pull --ff-only
fi
if [ $? -ne 0 ]; then exit 1; fi
#this is a temp measure to fix build issue
if [ -f "$AR_COMPS/esp-dl/idf_component.yml" ]; then
rm -rf "$AR_COMPS/esp-dl/idf_component.yml"
rm -rf "$AR_COMPS/esp-dl/idf_component.yml"
fi

#
# CLONE/UPDATE ESP-SR
#
echo "Updating ESP-SR..."
if [ ! -d "$AR_COMPS/esp-sr" ]; then
git clone $SR_REPO_URL "$AR_COMPS/esp-sr"
else
git -C "$AR_COMPS/esp-sr" fetch && \
git -C "$AR_COMPS/esp-sr" pull --ff-only
fi
if [ $? -ne 0 ]; then exit 1; fi
#echo "Updating ESP-SR..."
#if [ ! -d "$AR_COMPS/esp-sr" ]; then
# git clone $SR_REPO_URL "$AR_COMPS/esp-sr"
#else
# git -C "$AR_COMPS/esp-sr" fetch && \
# git -C "$AR_COMPS/esp-sr" pull --ff-only
#fi
#if [ $? -ne 0 ]; then exit 1; fi

#
# CLONE/UPDATE ESP-RAINMAKER
#
echo "Updating ESP-RainMaker..."
if [ ! -d "$AR_COMPS/esp-rainmaker" ]; then
git clone $RMAKER_REPO_URL "$AR_COMPS/esp-rainmaker" && \
git -C "$AR_COMPS/esp-rainmaker" reset --hard d8e93454f495bd8a414829ec5e86842b373ff555 && \
git -C "$AR_COMPS/esp-rainmaker" submodule update --init --recursive
#echo "Updating ESP-RainMaker..."
#if [ ! -d "$AR_COMPS/esp-rainmaker" ]; then
# git clone $RMAKER_REPO_URL "$AR_COMPS/esp-rainmaker" && \
# git -C "$AR_COMPS/esp-rainmaker" reset --hard d8e93454f495bd8a414829ec5e86842b373ff555 && \
# git -C "$AR_COMPS/esp-rainmaker" submodule update --init --recursive
# else
# git -C "$AR_COMPS/esp-rainmaker" fetch && \
# git -C "$AR_COMPS/esp-rainmaker" pull --ff-only && \
# git -C "$AR_COMPS/esp-rainmaker" fetch && \
# git -C "$AR_COMPS/esp-rainmaker" pull --ff-only && \
# git -C "$AR_COMPS/esp-rainmaker" submodule update --init --recursive
fi
if [ $? -ne 0 ]; then exit 1; fi
#fi
#if [ $? -ne 0 ]; then exit 1; fi

#this is a temp measure to fix build issue
if [ -f "$AR_COMPS/esp-rainmaker/components/esp-insights/components/esp_insights/scripts/get_projbuild_gitconfig.py" ] && [ `cat "$AR_COMPS/esp-rainmaker/components/esp-insights/components/esp_insights/scripts/get_projbuild_gitconfig.py" | grep esp32c6 | wc -l` == "0" ]; then
echo "Overwriting 'get_projbuild_gitconfig.py'"
cp -f "tools/get_projbuild_gitconfig.py" "$AR_COMPS/esp-rainmaker/components/esp-insights/components/esp_insights/scripts/get_projbuild_gitconfig.py"
fi
#if [ -f "$AR_COMPS/esp-rainmaker/components/esp-insights/components/esp_insights/scripts/get_projbuild_gitconfig.py" ] && [ `cat "$AR_COMPS/esp-rainmaker/components/esp-insights/components/esp_insights/scripts/get_projbuild_gitconfig.py" | grep esp32c6 | wc -l` == "0" ]; then
# echo "Overwriting 'get_projbuild_gitconfig.py'"
# cp -f "tools/get_projbuild_gitconfig.py" "$AR_COMPS/esp-rainmaker/components/esp-insights/components/esp_insights/scripts/get_projbuild_gitconfig.py"
#fi

#
# CLONE/UPDATE ESP-LITTLEFS
#
echo "Updating ESP-LITTLEFS..."
if [ ! -d "$AR_COMPS/esp_littlefs" ]; then
git clone $LITTLEFS_REPO_URL "$AR_COMPS/esp_littlefs" && \
git -C "$AR_COMPS/esp_littlefs" submodule update --init --recursive
else
git -C "$AR_COMPS/esp_littlefs" fetch && \
git -C "$AR_COMPS/esp_littlefs" pull --ff-only && \
git -C "$AR_COMPS/esp_littlefs" submodule update --init --recursive
fi
if [ $? -ne 0 ]; then exit 1; fi
#echo "Updating ESP-LITTLEFS..."
#if [ ! -d "$AR_COMPS/esp_littlefs" ]; then
# git clone $LITTLEFS_REPO_URL "$AR_COMPS/esp_littlefs" && \
# git -C "$AR_COMPS/esp_littlefs" submodule update --init --recursive
#else
# git -C "$AR_COMPS/esp_littlefs" fetch && \
# git -C "$AR_COMPS/esp_littlefs" pull --ff-only && \
# git -C "$AR_COMPS/esp_littlefs" submodule update --init --recursive
#fi
#if [ $? -ne 0 ]; then exit 1; fi

#
# CLONE/UPDATE TINYUSB
#
echo "Updating TinyUSB..."
if [ ! -d "$AR_COMPS/arduino_tinyusb/tinyusb" ]; then
git clone $TINYUSB_REPO_URL "$AR_COMPS/arduino_tinyusb/tinyusb"
git clone $TINYUSB_REPO_URL "$AR_COMPS/arduino_tinyusb/tinyusb"
else
git -C "$AR_COMPS/arduino_tinyusb/tinyusb" fetch && \
git -C "$AR_COMPS/arduino_tinyusb/tinyusb" pull --ff-only
git -C "$AR_COMPS/arduino_tinyusb/tinyusb" fetch && \
git -C "$AR_COMPS/arduino_tinyusb/tinyusb" pull --ff-only
fi
if [ $? -ne 0 ]; then exit 1; fi

#
# CLONE/UPDATE TFLITE MICRO
#
echo "Updating TFLite Micro..."
if [ ! -d "$AR_COMPS/tflite-micro" ]; then
git clone $TFLITE_REPO_URL "$AR_COMPS/tflite-micro"
git -C "$AR_COMPS/tflite-micro" submodule update --init --recursive
else
git -C "$AR_COMPS/tflite-micro" fetch && \
git -C "$AR_COMPS/tflite-micro" pull --ff-only
git -C "$AR_COMPS/tflite-micro" submodule update --init --recursive
fi
if [ $? -ne 0 ]; then exit 1; fi
#echo "Updating TFLite Micro..."
#if [ ! -d "$AR_COMPS/tflite-micro" ]; then
# git clone $TFLITE_REPO_URL "$AR_COMPS/tflite-micro"
# git -C "$AR_COMPS/tflite-micro" submodule update --init --recursive
#else
# git -C "$AR_COMPS/tflite-micro" fetch && \
# git -C "$AR_COMPS/tflite-micro" pull --ff-only
# git -C "$AR_COMPS/tflite-micro" submodule update --init --recursive
#fi
#if [ $? -ne 0 ]; then exit 1; fi