Increase "location priority" score of library locations set via --library flag

[per1234]

Describe the request

When the user passes a specific library path to the compilation command via the --library flag, give that library priority over any other candidates.

🙂 The user will have more power to influence library selection.

Describe the current behavior

Arduino CLI automatically resolves library dependencies during compilation by searching the library installation locations for libraries that contain a file matching the #include directive.

Multiple matching libraries may be found. In this case, Arduino CLI must decide which library to use. Multiple factors are considered. One of these factors is the library's location category:

1. Unmanaged: under a location specified by the --library or --libraries flags (with the --library flag locations having higher priority than the --libraries flag locations if the command used both)
2. User: under the libraries subfolder of the user folder
3. PlatformBuiltIn: bundled with the board platform
4. ReferencedPlatformBuiltIn: bundled with the core platform
5. IDEBuiltIn: under the "built-in" folder

Since configuring the compile command to search a specific custom location for libraries is a clear indication of intent from the user, libraries from the "Unmanaged" location are given the highest location priority score. However, location priority is a less significant factor than other factors, so a library with a lower location priority may be chosen even when a matching library was found in the "Unmanaged" location:

$ arduino-cli version
arduino-cli.exe  Version: git-snapshot Commit: fbeb2718 Date: 2023-03-13T05:29:37Z

$ export ARDUINO_DIRECTORIES_USER="/tmp/foo-sketchbook" # Use a throwaway directories.user for the demo.

$ mkdir --parents "$ARDUINO_DIRECTORIES_USER/libraries/SomeHeader"

$ touch "$ARDUINO_DIRECTORIES_USER/libraries/SomeHeader/SomeHeader.h"

$ mkdir --parents "$ARDUINO_DIRECTORIES_USER/libraries/Mismatch"

$ touch "$ARDUINO_DIRECTORIES_USER/libraries/Mismatch/SomeHeader.h"

$ mkdir "$ARDUINO_DIRECTORIES_USER/DependentSketch"

$ printf "#include <SomeHeader.h>\nvoid setup() {}\nvoid loop() {}" > "$ARDUINO_DIRECTORIES_USER/DependentSketch/DependentSketch.ino"

$ arduino-cli compile --fqbn arduino:avr:uno "$ARDUINO_DIRECTORIES_USER/DependentSketch"

[...]

Used library Version Path
SomeHeader           C:\Users\per\AppData\Local\Temp\foo-sketchbook\libraries\SomeHeader

[...]

🙂 All other factors being equal, the library with the highest name priority was selected.

$ arduino-cli compile --fqbn arduino:avr:uno --library "$ARDUINO_DIRECTORIES_USER/libraries/Mismatch" "$ARDUINO_DIRECTORIES_USER/DependentSketch"

[...]

Used library Version Path
SomeHeader           C:\Users\per\AppData\Local\Temp\foo-sketchbook\libraries\SomeHeader

[...]

🙁 The user gave a clear sign of preference for the "Mismatch" library, yet the "SomeHeader" library was still selected because its higher name priority score outweighed its lower location priority score.

Arduino CLI version

fbeb271

Operating system

All

Operating system version

Any

Additional context

This proposal is potentially breaking in that it is possible there are cases where it would change the selected library where the previous weights were causing the intended library to be chosen.

It is likely that only a small portion of direct users of Arduino CLI are using the more niche --library/--libraries flags so probably the chances of significant impact on the direct users is low.

Arduino IDE doesn't make any use of "Unmanaged" locations, so there is no possible impact on those users.

Arduino Cloud does make significant use of "Unmanaged" locations:

• The path to the globally installed libraries is specified via a --libraries flag
• The path to each of the "CUSTOM" libraries are specified via a --library flag
• The path to each of the sketch's pinned libraries are specified via a --library flag

So the chance of impact on the Arduino Cloud users is quite high. However, they also can benefit the most from the change because it would allow them to force the intended library dependency to be selected in an environment of thousands of constantly updating libraries that is otherwise not under their control.

---

If the --libraries flag is to continue to be used to specify the paths to globally installed libraries in the Arduino Cloud compilation commands (vs. setting that path via directories.user as is standard), then an additional location category must be added so that the libraries from the path specified via the --library flag can assigned a different location category than the libraries from the paths specified via --libraries flag.

This would be necessary to allow the weight increase to be bestowed only upon the location category associated with the --library flag.

---

Real world example of how difficult/annoying it can be to force library selection in Arduino Cloud with the current weights:

https://forum.arduino.cc/t/sketch-compiles-on-arduino-laptop-ide-but-not-web-ide-library-issue/1095090

Issue checklist

• I searched for previous requests in the issue tracker
• I verified the feature was still missing when using the latest nightly build
• My request contains all necessary details

[maxgerhardt]

+1. I actually have a use case in the Arduino-IRemote and the Teensy(4) core, using arduino-cli it always preferes the older IRemote library version that is in the core and doesn't use the new library version we try to run the CI for.

Arduino-IRremote/Arduino-IRremote#1110