Commandline too long for xtensa-esp32-elf-gcc ?

[Bascy]

When my sizable project gets compiled, the combiner phase uses the following command:

'Starting combiner' "D:\sloeber4.3\/arduinoPlugin/packages/arduino-esp32/tools/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc" -nostdlib "-LD:\sloeber4.3\/arduinoPlugin/packages/arduino-esp32/tools/sdk/lib" "-LD:\sloeber4.3\/arduinoPlugin/packages/arduino-esp32/tools/sdk/ld" -T esp32_out.ld -T esp32.common.ld -T esp32.rom.ld -T esp32.peripherals.ld -T esp32.rom.spiram_incompatible_fns.ld -u ld_include_panic_highint_hdl -u call_user_start_cpu0 -Wl,--gc-sections -Wl,-static -Wl,--undefined=uxTopUsedPriority -u __cxa_guard_dummy -u __cxx_fatal_exception -Wl,--start-group .\CSenseAirPage.cpp.o .\CSenseAlarmPage.cpp.o .\CSenseBatteryPage.cpp.o .\CSenseGasSmokeCalibratePage.cpp.o .\CSenseHomePage.cpp.o .\CSenseIBCMaster.cpp.o .\CSenseLevelPage.cpp.o .\CSenseLightsPage.cpp.o .\CSenseLocationPage.cpp.o .\CSenseNumKeypadPage.cpp.o .\CSensePageHandler.cpp.o .\CSenseSettingsPage.cpp.o .\CSenseShuntCanbusSensor.cpp.o .\CSenseStoredSettings.cpp.o .\CSenseTemperaturePage.cpp.o .\CSenseTimerPage.cpp.o .\CSenseWaterPage.cpp.o .\CSenseWaterflowCanbusSensor.cpp.o .\CSenseWeatherPage.cpp.o .\SenseBuzzer.cpp.o .\SenseEdit.cpp.o .\SenseTimerAlertSensor.cpp.o .\csIntruderAlert.cpp.o .\csLeavePage.cpp.o .\csMainMaster.cpp.o .\csMainMasterMain.cpp.o .\csMasterOTAUpdater.cpp.o .\csWiFiManager.cpp.o .\libraries\Wire\src\Wire.cpp.o .\libraries\WiFi\src\ETH.cpp.o .\libraries\WiFi\src\WiFi.cpp.o .\libraries\WiFi\src\WiFiAP.cpp.o .\libraries\WiFi\src\WiFiClient.cpp.o .\libraries\WiFi\src\WiFiGeneric.cpp.o .\libraries\WiFi\src\WiFiMulti.cpp.o .\libraries\WiFi\src\WiFiSTA.cpp.o .\libraries\WiFi\src\WiFiScan.cpp.o .\libraries\WiFi\src\WiFiServer.cpp.o .\libraries\WiFi\src\WiFiUdp.cpp.o .\libraries\Update\src\Updater.cpp.o .\libraries\Timer\Event.cpp.o .\libraries\Timer\Timer.cpp.o .\libraries\SenseNextionFramework\src\SenseNeoNextion.cpp.o .\libraries\SenseNextionFramework\src\SenseNextionComponents.cpp.o .\libraries\SenseNextionFramework\src\SenseNextionFramework.cpp.o .\libraries\SenseNextionFramework\src\SenseNextionUpload.cpp.o .\libraries\SenseNextionFramework\src\SenseNextionVersionChecker.cpp.o .\libraries\SenseNextionFramework\src\SensePage.cpp.o .\libraries\SenseNextionFramework\src\SensePageDefinitions.cpp.o .\libraries\SenseNextionFramework\src\SensePageHandler.cpp.o .\libraries\SenseNextionFramework\src\SenseWaveForm.cpp.o .\libraries\SenseIBCFramework\src\RemoteBooleanSensor.cpp.o .\libraries\SenseIBCFramework\src\RemoteDateTimeSensor.cpp.o .\libraries\SenseIBCFramework\src\RemoteFloatSensor.cpp.o .\libraries\SenseIBCFramework\src\RemoteIntegerSensor.cpp.o .\libraries\SenseIBCFramework\src\RemoteYawPitchRollSensor.cpp.o .\libraries\SenseIBCFramework\src\SenseIBCConstants.cpp.o .\libraries\SenseIBCFramework\src\SenseIBCFramework.cpp.o .\libraries\SenseIBCFramework\src\SenseLightZoneRemoteActuator.cpp.o .\libraries\SenseIBCFramework\src\SenseRemoteActuator.cpp.o .\libraries\SenseIBCFramework\src\SenseRemoteSensor.cpp.o .\libraries\SenseFramework\src\CohenSutherlandAlgorthm.cpp.o .\libraries\SenseFramework\src\DigitalPinActuator.cpp.o .\libraries\SenseFramework\src\ISenseGPSValued.cpp.o .\libraries\SenseFramework\src\ISensePWMValued.cpp.o .\libraries\SenseFramework\src\ISenseShuntValued.cpp.o .\libraries\SenseFramework\src\ISenseYawPitchRollValued.cpp.o .\libraries\SenseFramework\src\PWMPinActuator.cpp.o .\libraries\SenseFramework\src\SenseActuator.cpp.o .\libraries\SenseFramework\src\SenseActuatorSerial.cpp.o .\libraries\SenseFramework\src\SenseAlertHandler.cpp.o .\libraries\SenseFramework\src\SenseAnalogSensor.cpp.o .\libraries\SenseFramework\src\SenseBase.cpp.o .\libraries\SenseFramework\src\SenseBuzzerActuator.cpp.o .\libraries\SenseFramework\src\SenseCanBusNode.cpp.o .\libraries\SenseFramework\src\SenseCanBusSensor.cpp.o .\libraries\SenseFramework\src\SenseCircularBufferObserver.cpp.o .\libraries\SenseFramework\src\SenseDigitalPinSensor.cpp.o .\libraries\SenseFramework\src\SenseFramework.cpp.o .\libraries\SenseFramework\src\SenseFreeMemSensor.cpp.o .\libraries\SenseFramework\src\SenseI2CBusSensor.cpp.o .\libraries\SenseFramework\src\SenseI2CWireUser.cpp.o .\libraries\SenseFramework\src\SenseInterruptCountSensor.cpp.o .\libraries\SenseFramework\src\SenseInterruptManager.cpp.o .\libraries\SenseFramework\src\SenseNode.cpp.o .\libraries\SenseFramework\src\SenseObserver.cpp.o .\libraries\SenseFramework\src\SenseSensor.cpp.o .\libraries\SenseFramework\src\SenseSensorInitialisor.cpp.o .\libraries\SenseFramework\src\SenseSerialPort.cpp.o .\libraries\SenseFramework\src\SenseTimedTaskList.cpp.o .\libraries\SenseFramework\src\SenseTimestampProvider.cpp.o .\libraries\SenseFramework\src\SenseValueSensor.cpp.o .\libraries\SenseFramework\src\SenseWatchdog.cpp.o .\libraries\SenseFramework\src\TemperatureSensor.cpp.o .\libraries\SenseBLEFramework\BLEManager.cpp.o .\libraries\SenseBLEFramework\BLEObserver.cpp.o .\libraries\SPI\src\SPI.cpp.o .\libraries\SPIFFS\src\SPIFFS.cpp.o .\libraries\RTClib\RTClib.cpp.o .\libraries\Preferences\src\Preferences.cpp.o .\libraries\NeoNextion\INextionColourable.cpp.o .\libraries\NeoNextion\INextionFontStyleable.cpp.o .\libraries\NeoNextion\INextionTouchable.cpp.o .\libraries\NeoNextion\INextionWidget.cpp.o .\libraries\NeoNextion\Nextion.cpp.o .\libraries\NeoNextion\NextionCrop.cpp.o .\libraries\NeoNextion\NextionPage.cpp.o .\libraries\NeoNextion\NextionPicture.cpp.o .\libraries\NeoNextion\NextionSlidingText.cpp.o .\libraries\NeoNextion\NextionTimer.cpp.o .\libraries\NeoNextion\NextionWaveform.cpp.o .\libraries\NeoNextion\sscanf.cpp.o .\libraries\MCP_CAN_lib\mcp_can.cpp.o .\libraries\FS\src\FS.cpp.o .\libraries\FS\src\vfs_api.cpp.o .\libraries\ESPmDNS\src\ESPmDNS.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLE2902.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLE2904.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEAddress.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEAdvertisedDevice.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEAdvertising.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEBeacon.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLECharacteristic.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLECharacteristicMap.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEClient.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEDescriptor.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEDescriptorMap.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEDevice.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEEddystoneTLM.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEEddystoneURL.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEExceptions.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEHIDDevice.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLERemoteCharacteristic.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLERemoteDescriptor.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLERemoteService.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEScan.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLESecurity.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEServer.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEService.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEServiceMap.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEUUID.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEUtils.cpp.o .\libraries\ESP32_BLE_Arduino\src\BLEValue.cpp.o .\libraries\ESP32_BLE_Arduino\src\FreeRTOS.cpp.o .\libraries\ESP32_BLE_Arduino\src\GeneralUtils.cpp.o .\libraries\ArduinoOTA\src\ArduinoOTA.cpp.o .\libraries\Arduino-MPU6050\MPU6050.cpp.o .\libraries\Adafruit_INA219\Adafruit_INA219.cpp.o .\libraries\Adafruit_BMP280_Library\Adafruit_BMP280.cpp.o D:/C-Sense2.0/source/csMainMaster/Debug/arduino.ar -lgcc -lopenssl -lbtdm_app -lfatfs -lwps -lcoexist -lwear_levelling -lesp_http_client -lprotobuf-c -lhal -lnewlib -ldriver -lbootloader_support -lpp -lfreemodbus -lmesh -lsmartconfig -ljsmn -lwpa -lethernet -lphy -lapp_trace -lconsole -lulp -lwpa_supplicant -lfreertos -lbt -lmicro-ecc -lesp32-camera -lcxx -lxtensa-debug-module -ltcp_transport -lmdns -lvfs -lesp_ringbuf -lsoc -lcore -lsdmmc -llibsodium -lcoap -ltcpip_adapter -lprotocomm -lesp_event -lc_nano -lesp-tls -lasio -lrtc -lspi_flash -lwpa2 -lwifi_provisioning -lesp32 -lapp_update -lnghttp -lspiffs -lunity -lesp_https_server -lespnow -lnvs_flash -lesp_adc_cal -llog -lsmartconfig_ack -lexpat -lm -lmqtt -lc -lheap -lmbedtls -llwip -lnet80211 -lesp_http_server -lpthread -ljson -lesp_https_ota -lstdc++ -Wl,--end-group -Wl,-EL -o "D:/C-Sense2.0/source/csMainMaster/Debug/csMainMaster.elf"

This results in the following error message:

d:/sloeber4.3/arduinoplugin/packages/arduino-esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld.exe: cannot open output file D:/C-Sense2.0/source/csMainMaser/Debug/csMainMaster.elf: No such file or directory
collect2.exe: error: ld returned 1 exit status
makefile:101: recipe for target 'csMainMaster.elf' failed

When I try to copy paste the command into a cmd window, it doesnt paste the whole command, (it stops after -lesp_hhtp_server) and I cannot manually add anything either, indicating probably the the max-length of command is reached.

How can i get this to work again?

[EDIT] If i paste the command into a bash prompt (after replacing all \ with / ) it does run succesfully

[jantje]

This is a known issue with windows having a maximum command line length.
If windows doesn't have a way to extend it I don't really know how to fix this.

[Bascy]

Maybe there is a parameter with elf.gcc that lets you specify the parameters in a file?

[jantje]

Maybe there is a parameter with elf.gcc that lets you specify the parameters in a file?

That would break the way arduino boards.txt and platform.txt works.

[jantje]

Seems like this would be fixed with a different make toolset.
https://mcuoneclipse.com/2015/03/29/solving-the-8192-character-command-line-limit-on-windows/
Yoiu could try to replace make.exe in D:\sloeber4.3/arduinoPlugin/tools/make with this set.
No guarantee it will work though as make has some "variants"

[Bascy]

Replaced the make.exe with the one from https://gnu-mcu-eclipse.github.io/windows-build-tools/install/
And this seems to solve the problem!

hmmm .. hold the horses .... It does seem to have messed with the project. All selected libraries are out of the project. it also did not include all selected libraries in the build.

So not a solution, alas :(

[jantje]

I do not understand how replacing external tools could influence the internal project structure.
I'll give it a try here though I can't really test as I do not have a project with generates to long command lines.

[jantje]

I did a quick test here and it worked fine.
I did have the "missing libraries" as you described but that was caused by something causing the make file to be recreated and no longer having the platform version installed. Please check the project properties of the "missing libraries" project.

I assume you are on a windows 64 machine. If so please try the following.
Rename [sloeber]/arduinoPlugin/downloads/make.zip to [sloeber]/arduinoPlugin/downloads/make.zip.org
download and save the file attached here in [sloeber]/arduinoPlugin/downloads/
delete the folder [sloeber]/arduinoPlugin/tools/make
In sloeber install a platform (doesn't really care which one and you can remove it later)
This issue should be fixed now :-)
Note that this is a 64 bit windows make so it needs a 64 bit windows OS
make.zip

[jantje]

The make.zip attached in the previous comment is part of the content downloaded here https://github.com/gnu-mcu-eclipse/windows-build-tools/releases in the format Sloeber expects it.
Thanks @ilg-ul

[ilg-ul]

You're welcome!

I guess the secret to solve this issue is the use of sh.exe intead of cmd.exe.

[Bascy]

but i only need the replacement make.exe?

[ilg-ul]

I don't know the details of your setup, but for maximum results you need both make.exe and sh.exe.

make.exe normally has no limitations, but it does not start the compiler directly, but via a shell. if it finds the sh.exe, it uses it, otherwise it used the native cmd.exe.

unfortunately on previous Windows, cmd.exe was limited to 8192. I don't know how the latest version behaves.

it is true that third party make.exe may also be built with limitations, but if you use the binaries I distribute, the limitation comes from the shell, so for best result you need the entire package, with sh.exe, and preferably with rm.exe, echo.exe, etc.

[ilg-ul]

FYI, there is a simple way of telling if the build uses cmd.exe or sh.exe: when cmd.exe is selected, all the lines echoed during the build are quoted, probably because the internal echo in cmd.exe uses quotes.

[jantje]

I don't know the details of your setup, but for maximum results you need both make.exe and sh.exe.

I took the whole bin folder from your package, so sh.exe, make.exe and echo.exe are there. I did so because I have had serious issues with make/cmd/bash/sh combinations which made me modify the environment variable path used during compilation to the bare minimum.

unfortunately on previous Windows, cmd.exe was limited to 8192. I don't know how the latest version behaves.

Microsoft reports limitations are decreasing (in other words longer commands are possible) but it is unclear to me which limitation hits when. For sure OP did get hit.

[jantje]

but i only need the replacement make.exe?

No you need the whole package.

[Bascy]

Did as you suggested, but make is still run in cmd

[Bascy]

Where should the make.zip be extracted to?
if its arduinoPlugin\tools\make .. nothing has changed there
... Oops .. I forgot to delete that folder :(

[Bascy]

Make folder is now replaced with the contents of the zip file, (make -v says version 4.2.1) but the build process is still using cmd i think:

20:21:14 **** Incremental Build of configuration Debug for project csMainMaster ****
"D:\sloeber4.3\arduinoPlugin\tools\make\make" -j4 all
'Building file: ..\CSenseAirPage.cpp'
'Building file: ..\CSenseAlarmPage.cpp'
'Building file: ..\CSenseBatteryPage.cpp'
'Starting C++ compile'

[jantje]

It looks like it is cmd.exe that is used indeed.
Can you go to the project properties->C/C++build->environment and modify PATH (at the bottom) from
${A.COMPILER.PATH}${PathDelimiter}${A.BUILD.GENERIC.PATH}${PathDelimiter}${SystemRoot}\system32${PathDelimiter}${SystemRoot}${PathDelimiter}${SystemRoot}\system32\Wbem${PathDelimiter}${sloeber_path_extension}
to
D:\sloeber4.3\arduinoPlugin\tools\make;${A.COMPILER.PATH}${PathDelimiter}${A.BUILD.GENERIC.PATH}${PathDelimiter}${SystemRoot}\system32${PathDelimiter}${SystemRoot}${PathDelimiter}${SystemRoot}\system32\Wbem${PathDelimiter}${sloeber_path_extension}

[Bascy]

No, still using cmd

[jantje]

I tried here and it works .... I mean no more quotes.
However ... the linking fails due to .\filename.cpp.o which is not properly interpreted by the sh in the package.

[Bascy]

Is that caused by the backslash?

[ilg-ul]

You can double the backslash or, even better, use POSIX paths.

[jantje]

You can double the backslash or, even better, use POSIX paths.

I know, it is just that the change is not a quick fix as it is in a ugly piece of code :-( which means lots of testing as well.
I also have other fish to fry right now (like making Christmas presents :-( )
But IMHO it is the way to go.

I assume replacing the toOSString in something more appropriate in this file https://github.com/Sloeber/arduino-eclipse-plugin/blob/master/io.sloeber.core/src/io/sloeber/core/toolchain/ArduinoGnuMakefileGenerator.java is a good start.
Maybe op can give this a shot

[judge2005]

My projects don't even get as far as the linker, because (at least in 1.0.1 of the ESP32 package) C:\Users\mpand\eclipse\arduinoPlugin\packages\esp32\hardware\esp32\1.0.1\tools\sdk\include has a ton of subdirectories, each one of which is included on the compilation command line with a -I. After adding few more of my own libraries, this easily breaks whatever command line length limit windows is imposing and very strange things start to happen (by which I mean, it just randomly removes characters from the output filename!). I will try the package mentioned above, but I was going to suggest that perhaps it should only include those subdirectories that contain headers that are actually referenced. Or maybe something in the distribution is referencing them all? Although, if it will then just break at the linker stage, that doesn't really help either.

[judge2005]

That make.zip worked for me. Many thanks.