Partition tables, CSV in Variants folder, file not found

[WanaGo]

Board

ESP32-S3 DEV

Device Description

Using a custom module, however its the same for the ESP32-S3 DEV

Hardware Configuration

N/A

Version

latest master (checkout manually)

IDE Name

Arduino IDE 2.1.1

Operating System

Windows 11

Flash frequency

80Mhz

PSRAM enabled

yes

Upload speed

921600

Description

https://github.com/espressif/arduino-esp32/blob/master/docs/source/tutorials/partition_table.rst

This tutorial describes how to set up a Partition scheme CSV file, however it does not touch on at all how to put the CSV's into the Variants folder inside the core, rather than in the tools/partitions/ folder.

Wanting to submit a pull request soon to add our custom board into the boards.txt etc, however we just cannot get the partitions working out of the variants folder, it fails to compile as it cannot fine 'partitions.csv'.
Doing the exact same thing with an Adafruit board, which has its CSV file in its Variants folder, the compile looks for its CSV file in the variants folder, and copies it to the temp folder partitions.csv just fine - but why ours doesnt, I have no idea.

I dont know what I am missing.

I have edited the boards.txt file, and added in the partition listings.
I have created my board variant folder, which is referenced in the boards.txt file.

When it goes to compile, it says it cant find the partitions.csv file though.

When I select say the Adafruit MagTag board, which is set up similar for partitions, its CSV file is in its variants folder, nothing too different in its boards.txt file, but when that compiles, the compile text is quite different and it specifically looks at the Variants folder and the specific CSV selected in the Partition scheme menu, and copies that as is required to make the partitions.csv file.

Mine does not do that, it never looks in the variants folder for the CSV file, to copy over, and I have no idea why. I must be missing something, but I just cant figure out what it is.

If I have the partition csv files in the tools/partitions/ folder, it works fine - however that is not where it 'should' be I think, as I want to eventually submit this to merge into the ESP32 core, and it needs to then be in the Variants folder for my custom board.

Does anyone have any idea what is missing?

Error is:

FQBN: esp32:esp32:xxx-ESP32-S3R8n16
Using board 'xxx-ESP32-S3R8n16' from platform in folder: C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.9
Using core 'esp32' from platform in folder: C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.9

cmd /c if exist "C:\Users\xxx\Documents\Arduino\MinimalTest\partitions.csv" COPY /y "C:\Users\xxx\Documents\Arduino\MinimalTest\partitions.csv" "C:\Users\xxx\AppData\Local\Temp\arduino\sketches\A811B3FA3DA6C70399255D8FABE9EA9C\partitions.csv"
cmd /c if not exist "C:\Users\xxx\AppData\Local\Temp\arduino\sketches\A811B3FA3DA6C70399255D8FABE9EA9C\partitions.csv" if exist "C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.9\variants\esp32_s3r8n16\partitions.csv" COPY "C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.9\variants\esp32_s3r8n16\partitions.csv" "C:\Users\xxx\AppData\Local\Temp\arduino\sketches\A811B3FA3DA6C70399255D8FABE9EA9C\partitions.csv"
cmd /c if not exist "C:\Users\xxx\AppData\Local\Temp\arduino\sketches\A811B3FA3DA6C70399255D8FABE9EA9C\partitions.csv" COPY "C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.9\tools\partitions\partitions-16MB-config1.csv" "C:\Users\xxx\AppData\Local\Temp\arduino\sketches\A811B3FA3DA6C70399255D8FABE9EA9C\partitions.csv"
The system cannot find the file specified.

exit status 1

Compilation error: exit status 1

The CSV file I have selected is called "partitions-16MB-config1.csv" and its in the "C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.9\variants\xxx-esp32-s3r8n16" folder

I just cant find what I am missing.

When I select the Adafruit board, and compile, it shows this (correctly getting their custom CSV file from their variants folder, and copying that):

cmd /c if exist "C:\\Users\\xxx\\Documents\\Arduino\\MinimalTest\\partitions.csv" COPY /y "C:\\Users\\xxx\\Documents\\Arduino\\MinimalTest\\partitions.csv" "C:\\Users\\xxx\\AppData\\Local\\Temp\\arduino\\sketches\\A811B3FA3DA6C70399255D8FABE9EA9C\\partitions.csv"
cmd /c if not exist "C:\\Users\\xxx\\AppData\\Local\\Temp\\arduino\\sketches\\A811B3FA3DA6C70399255D8FABE9EA9C\\partitions.csv" if exist **"C:\\Users\\xxx\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\2.0.9\\variants\\adafruit_magtag29_esp32s2\\partitions-4MB-tinyuf2.csv"** COPY **"C:\\Users\\xxx\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\2.0.9\\variants\\adafruit_magtag29_esp32s2\\partitions-4MB-tinyuf2.csv"** "C:\\Users\\xxx\\AppData\\Local\\Temp\\arduino\\sketches\\A811B3FA3DA6C70399255D8FABE9EA9C\\partitions.csv"
        **1 file(s) copied.**
cmd /c if not exist "C:\\Users\\xxx\\AppData\\Local\\Temp\\arduino\\sketches\\A811B3FA3DA6C70399255D8FABE9EA9C\\partitions.csv" COPY "C:\\Users\\xxx\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\2.0.9\\tools\\partitions\\default.csv" "C:\\Users\\xxx\\AppData\\Local\\Temp\\arduino\\sketches\\A811B3FA3DA6C70399255D8FABE9EA9C\\partitions.csv"
cmd /c IF EXIST "C:\\Users\\xxx\\Documents\\Arduino\\MinimalTest\\bootloader.bin" ( COPY /y "C:\\Users\\xxx\\Documents\\Arduino\\MinimalTest\\bootloader.bin" "C:\\Users\\xxx\\AppData\\Local\\Temp\\arduino\\sketches\\A811B3FA3DA6C70399255D8FABE9EA9C\\MinimalTest.ino.bootloader.bin" ) ELSE ( IF EXIST "C:\\Users\\xxx\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\2.0.9\\variants\\adafruit_magtag29_esp32s2\\bootloader-tinyuf2.bin" ( COPY "C:\\Users\\xxx\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\2.0.9\\variants\\adafruit_magtag29_esp32s2\\bootloader-tinyuf2.bin" "C:\\Users\\xxx\\AppData\\Local\\Temp\\arduino\\sketches\\A811B3FA3DA6C70399255D8FABE9EA9C\\MinimalTest.ino.bootloader.bin" ) ELSE ( "C:\\Users\\xxx\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esptool_py\\4.5.1/esptool.exe" --chip esp32s2 elf2image --flash_mode dio --flash_freq 80m --flash_size 4MB -o "C:\\Users\\xxx\\AppData\\Local\\Temp\\arduino\\sketches\\A811B3FA3DA6C70399255D8FABE9EA9C\\MinimalTest.ino.bootloader.bin" "C:\\Users\\xxx\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\2.0.9\\tools\\sdk\\esp32s2\\bin\\bootloader_qio_80m.elf" ) )
        1 file(s) copied.
cmd /c if exist "C:\\Users\\xxx\\Documents\\Arduino\\MinimalTest\\build_opt.h" COPY /y "C:\\Users\\xxx\\Documents\\Arduino\\MinimalTest\\build_opt.h" "C:\\Users\\xxx\\AppData\\Local\\Temp\\arduino\\sketches\\A811B3FA3DA6C70399255D8FABE9EA9C\\build_opt.h"
cmd /c if not exist "C:\\Users\\xxx\\AppData\\Local\\Temp\\arduino\\sketches\\A811B3FA3DA6C70399255D8FABE9EA9C\\build_opt.h" type nul > "C:\\Users\\xxx\\AppData\\Local\\Temp\\arduino\\sketches\\A811B3FA3DA6C70399255D8FABE9EA9C\\build_opt.h"

I don't know what I am missing to make the compile know to look at the variants folder, like the adafruit board does. The tutorial doesnt cover this, and I am failing to find any information even with google searches. Its obviously possible to do, but I just cannot find the information on what I am missing.

Is anyone able to help, and then also trigger the Tutorial to be updated with this information please?

Sketch

Sketch is N/A, it doesnt matter what it is, its the compiling failing immediately due to not finding the partition file in the tools/partitions/ folder, when I want it to be looking at the variants/ folder instead.

Debug Message

FQBN: esp32:esp32:xxx-ESP32-S3R8n16
Using board 'xxx-ESP32-S3R8n16' from platform in folder: C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.9
Using core 'esp32' from platform in folder: C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.9

cmd /c if exist "C:\Users\xxx\Documents\Arduino\MinimalTest\partitions.csv" COPY /y "C:\Users\xxx\Documents\Arduino\MinimalTest\partitions.csv" "C:\Users\xxx\AppData\Local\Temp\arduino\sketches\A811B3FA3DA6C70399255D8FABE9EA9C\partitions.csv"
cmd /c if not exist "C:\Users\xxx\AppData\Local\Temp\arduino\sketches\A811B3FA3DA6C70399255D8FABE9EA9C\partitions.csv" if exist "C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.9\variants\esp32_s3r8n16\partitions.csv" COPY "C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.9\variants\esp32_s3r8n16\partitions.csv" "C:\Users\xxx\AppData\Local\Temp\arduino\sketches\A811B3FA3DA6C70399255D8FABE9EA9C\partitions.csv"
cmd /c if not exist "C:\Users\xxx\AppData\Local\Temp\arduino\sketches\A811B3FA3DA6C70399255D8FABE9EA9C\partitions.csv" COPY "C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.9\tools\partitions\partitions-16MB-config1.csv" "C:\Users\xxx\AppData\Local\Temp\arduino\sketches\A811B3FA3DA6C70399255D8FABE9EA9C\partitions.csv"
The system cannot find the file specified.

exit status 1

Compilation error: exit status 1

Other Steps to Reproduce

Putting the CSV files in the tools/partitions/ folder, work correctly.
What I cant figure out is what I need to do to tell it to look at the variants/ folder instead, like the Adafruit product does.
Nothing is obvious as to what I am missing in the boards.txt

All I am wanting to change is the partition schemes, I am not changing bootloader etc

I have tried with 2.0.9 and 2.0.11, and the results are the same. The above code was based on the 2.0.9 output - it is however identical.

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[lbernstone]

They are using the UF2 bootloader

[WanaGo]

I found the problem
In the boards.txt file, you simply have to have the flag custom_partitions instead of partitions for lines like:

boardXXX.menu.PartitionScheme.huge_app=Huge APP (3MB No OTA/1MB SPIFFS)
boardXXX.menu.PartitionScheme.huge_app.build.partitions=huge_app
boardXXX.menu.PartitionScheme.huge_app.upload.maximum_size=3145728

boardXXX.menu.PartitionScheme.huge_app=Huge APP (3MB No OTA/1MB SPIFFS)
boardXXX.menu.PartitionScheme.huge_app.build.custom_partitions=custom_huge_app
boardXXX.menu.PartitionScheme.huge_app.upload.maximum_size=3145728

it then looks in the variant folder.

I couldnt find this documented anywhere

[WanaGo]

https://forum.arduino.cc/t/arduino-2-1-1-custom-partitions-with-esp32-core/1156402

Confirmed here.

This can be closed, but it still needs to be updated in the Documentation I feel - as I could not find this anywhere

[VojtechBartoska]

Closing as solved.