Sourcing built-in libraries when not bundled w/ classic IDE is undocumented or unintended

[per1234]

During library discovery, Arduino CLI searches several locations, which are documented here. One of those locations can be the libraries subfolder of the Arduino IDE folder. The documented behavior:

https://arduino.github.io/arduino-cli/dev/sketch-build-process/#location-priority

This location is only used by Arduino CLI when it's located in the Arduino IDE installation folder

(Note this is referring to the classic Arduino IDE only. There is no special behavior when Arduino CLI is bundled with Arduino IDE 2.x)

Although that statement in the documentation is correct in regards to arduino-cli lib list, it turns out arduino-cli compile uses a completely different mechanism:

arduino-cli/commands/compile/compile.go

Lines 188 to 205 in bf4a784

	// Check if Arduino IDE is installed and get it's libraries location.
	dataDir := paths.New(configuration.Settings.GetString("directories.Data"))
	preferencesTxt := dataDir.Join("preferences.txt")
	ideProperties, err := properties.LoadFromPath(preferencesTxt)
	if err == nil {
	lastIdeSubProperties := ideProperties.SubTree("last").SubTree("ide")
	// Preferences can contain records from previous IDE versions. Find the latest one.
	var pathVariants []string
	for k := range lastIdeSubProperties.AsMap() {
	if strings.HasSuffix(k, ".hardwarepath") {
	pathVariants = append(pathVariants, k)
	}
	}
	sort.Strings(pathVariants)
	ideHardwarePath := lastIdeSubProperties.Get(pathVariants[len(pathVariants)-1])
	ideLibrariesPath := filepath.Join(filepath.Dir(ideHardwarePath), "libraries")
	builderCtx.BuiltInLibrariesDirs = paths.NewPathList(ideLibrariesPath)
	}

The value of an internal data property from the classic Arduino IDE preferences file at {directories.data}/preferences.txt is used to define an additional libraries folder located at:

{last.ide.<*>.hardwarepath}/../libraries

Unlike the completely different mechanism used by arduino-cli lib list, the one used by arduino-cli compile adds the IDE built-in libraries folder even if Arduino CLI is not located inside a classic Arduino IDE installation.

Steps to reproduce

Generate the preferences.txt file

1. Install a recent version of the classic Arduino IDE (e.g., 1.8.16) if an installation is already present on your computer.
2. Start the classic IDE in normal mode (not in portable mode).
3. Exit the classic IDE.

Demonstration of the use of the IDE built-in libraries by Arduino CLI

$ cat ~/AppData/Local/Arduino15/preferences.txt | grep "hardwarepath"
last.ide.1.8.16.hardwarepath=C:\Program Files (x86)\Arduino\hardware

$ which arduino-cli  # Arduino CLI is not bundled with IDE
/c/program-files/arduino/cli/arduino-cli_nightly/arduino-cli

$ arduino-cli version
arduino-cli.exe alpha Version: nightly-20211112 Commit: bf4a784 Date: 2021-11-12T01:26:50Z

$ arduino-cli lib uninstall Ethernet  # Clean up in case you already had the lib installed
$ arduino-cli core install arduino:avr
$ arduino-cli lib list --all Ethernet  # Output is as expected
No libraries installed.

$ mkdir /tmp/SomeEthernetSketch
$ printf "#include <Ethernet.h>\nvoid setup() {}\nvoid loop() {}\n" > /tmp/SomeEthernetSketch/SomeEthernetSketch.ino  # sketch that uses the Ethernet lib
$ arduino-cli compile -b arduino:avr:uno -v /tmp/SomeEthernetSketch
[...]
Using library Ethernet at version 2.0.0 in folder: C:\Program Files (x86)\Arduino\libraries\Ethernet
[...]

Note that, even though lib list doesn't consider the "Ethernet" library to be installed, and Arduino CLI is not bundled with the classic IDE installation, Arduino CLI still uses the classic IDE's built-in "Ethernet" library when compiling.

Recommendations

Since Arduino IDE 2.x does not have libraries built-in to its installation, I think it is best to just remove this capability from Arduino CLI.

If it is to be kept, then it is essential for the following two things to be done:

• Correctly document the behavior in https://github.com/arduino/arduino-cli/blob/master/docs/sketch-build-process.md#dependency-resolution
• Unify the behavior of arduino-cli compile and arduino-cli lib list. The list of libraries presented by lib list must exactly match the list used by the dependency resolution system.

Additional information

This also occurs when using Arduino IDE 2.x, meaning that compilations in Arduino IDE 2.x use libraries from any installation of the classic Arduino IDE you might happen to have on your system. Although that is very unexpected behavior, it may currently be mitigating the impact of Arduino IDE 2.x not having built-in libraries. However, the built-in library capability will be replaced via a different mechanism in Arduino IDE 2.x. So, if at all, this is only a temporary benefit.

[per1234]

Reopening because there was a regression (I bisected to 427af56)

$ cat ~/AppData/Local/Arduino15/preferences.txt | grep "hardwarepath"
last.ide.1.8.19.hardwarepath=C:\Program Files (x86)\Arduino\hardware

$ which arduino-cli  # Arduino CLI is not bundled with IDE
/e/arduino/cli/arduino-cli_nightly/arduino-cli

$ arduino-cli version
arduino-cli.exe  Version: nightly-20220620 Commit: 64bc4dc Date: 2022-06-20T01:34:24Z

$ arduino-cli lib uninstall Ethernet  # Clean up in case you already had the lib installed
Library Ethernet is not installed

$ arduino-cli core install arduino:avr
Platform arduino:[email protected] already installed

$ arduino-cli lib list --all Ethernet  # Output is as expected
Name             Installed Available   Location                    Description
Ethernet         1.0.0     2.0.0       arduino:[email protected] Enables network connection (local and...
 "               2.0.0     -           esp32:[email protected]           -
 "               2.0.0     -           esp8266:[email protected]       -
Ethernet(Edison) 1.0       -           Intel:[email protected]+1.0        -
Ethernet         2.0.0     -           m5stack:[email protected]         -
 "               2.0.0     -           teensy:[email protected]          -


$ mkdir /tmp/SomeEthernetSketch

$ printf "#include <Ethernet.h>\nvoid setup() {}\nvoid loop() {}\n" > /tmp/SomeEthernetSketch/SomeEthernetSketch.ino  # sketch that uses the Ethernet lib

$ arduino-cli compile -b arduino:avr:uno -v /tmp/SomeEthernetSketch

[...]

Using library Ethernet at version 2.0.0 in folder: C:\Program Files (x86)\Arduino\libraries\Ethernet