Skip to content

Add prioritization to tool selection in absence of explicit dependencies #1887

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jan 31, 2023

Conversation

cmaglie
Copy link
Member

@cmaglie cmaglie commented Sep 21, 2022

Please check if the PR fulfills these requirements

  • The PR has no duplicates (please search among the Pull Requests
    before creating one)
  • The PR follows
    our contributing guidelines
  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • UPGRADING.md has been updated with a migration guide (for breaking changes)
  • What kind of change does this PR introduce?
    It removes some ambiguities in compile/upload tools selection when the explicit dependencies set (from package_index.json) is missing.
  • What is the current behavior?
    In a situation when different platforms provide the same tool (for example arm-gcc-none-eabi from arduino and other 3rd parties), and in absence of the package_index.json, the selection of the tool in the compile/upload may not be deterministic (AKA: random).
  • What is the new behavior?
    In the situation above where a tool is provided by multiple packages then the compiler will pick in order of priority:
    • the tool provided by the board's package
    • in absence of it, the tool (from a 3rd party) with the greatest version according to semver
    • in case of a tie, the tool from the packager whose name comes first in alphabetic order
      The last rule is merely a way to make the tool selection fully deterministic even in the most uncommon situation.

@cmaglie cmaglie self-assigned this Sep 21, 2022
@cmaglie cmaglie added priority: high Resolution is a high priority topic: code Related to content of the project itself criticality: low Of low impact type: imperfection Perceived defect in any part of project labels Sep 21, 2022
@cmaglie cmaglie force-pushed the fix_tools_selection branch 2 times, most recently from 810ad0d to 5bf23b9 Compare September 21, 2022 19:41
@codecov
Copy link

codecov bot commented Sep 21, 2022

Codecov Report

Base: 36.56% // Head: 36.66% // Increases project coverage by +0.10% 🎉

Coverage data is based on head (27a8394) compared to base (4b70e02).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1887      +/-   ##
==========================================
+ Coverage   36.56%   36.66%   +0.10%     
==========================================
  Files         228      228              
  Lines       19376    19396      +20     
==========================================
+ Hits         7084     7111      +27     
+ Misses      11463    11455       -8     
- Partials      829      830       +1     
Flag Coverage Δ
unit 36.66% <100.00%> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
arduino/cores/packagemanager/package_manager.go 68.19% <100.00%> (+2.36%) ⬆️
commands/debug/debug_info.go 45.04% <100.00%> (ø)
commands/upload/upload.go 54.73% <100.00%> (ø)
legacy/builder/target_board_resolver.go 73.52% <100.00%> (ø)
arduino/cores/packagemanager/download.go 34.61% <0.00%> (+5.12%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@cmaglie
Copy link
Member Author

cmaglie commented Sep 23, 2022

I still need to update documentation, I'll do that after validating the changes.

@jantje
Copy link

jantje commented Oct 13, 2022

the tool provided by the board's package

Should this not be the boards platform (instead of the board's package)?
Is that the selected platform only or the selected platform and the referenced platform?
And if the selected platform and the referenced platform both have the tool (Which is possible but I have never seen it) which is selected?

in absence of it, the tool (from a 3rd party) with the greatest version according to semver

FYI
If I remember correctly I had a similar implementation in Sloeber one day and it caused lots of boards to fail build yet they build successfully in Arduino IDE.

@cmaglie
Copy link
Member Author

cmaglie commented Oct 13, 2022

Should this not be the boards platform (instead of the board's package)?

It's the same since the tools are defined at the package level (and referenced at platform level as toolsDependecies).
After re-reading the PR I see that my first explanation was a bit rushed... I made a much better explanation in the documentation changes: https://github.com/arduino/arduino-cli/pull/1887/files#diff-29b8b99faa5765ef9cc15dbe6010806716089af4475d8483766b02c237f33753R274-R288
here is the extract:

When the IDE needs a tool, it downloads the corresponding archive file and unpacks the content into a private folder
that can be referenced from platform.txt using one of the following properties:

  • {runtime.tools.TOOLNAME-VERSION.path}
  • {runtime.tools.TOOLNAME.path}

For example, to obtain the avr-gcc 4.8.1 folder we can use {runtime.tools.avr-gcc-4.8.1.path} or
{runtime.tools.avr-gcc.path}.

In general the same tool may be provided by different platforms (for example the Arduino AVR platform may provide an
arduino:avr-gcc and another 3rd party platform may provide their own 3rdparty:avr-gcc). The rules to disambiguate
are as follows:

  • The property {runtime.tools.TOOLNAME.path} points, in order of priority, to:
  1. the tool, version and packager specified via toolsDependencies in the package_index.json
  2. the highest version of the tool provided by the packager of the current platform
  3. the highest version of the tool provided by any other packager (in case of tie, the first packager in alphabetical
    order wins)
  • The property {runtime.tools.TOOLNAME-VERSION.path} points, in order of priority, to:
    1. the tool and version provided by the packager of the current platform
    2. the tool and version provided by any other packager (in case of tie, the first packager in alphabetical order wins)

I should change the sentence:

In general the same tool may be provided by different platforms (for example the Arduino AVR platform may provide an
arduino:avr-gcc and another 3rd party platform may provide their own 3rdparty:avr-gcc). The rules to disambiguate
are as follows:

to

In general the same tool may be provided by different packagers (for example the Arduino packager may provide an
arduino:avr-gcc and another 3rd party platform may provide their own 3rdparty:avr-gcc). The rules to disambiguate
are as follows:

In practice the idea is that when the toolsDependencies is not defined or not available, we first try to select the tools from the same packager or, if not found, from another packager.


About the other question:

Is that the selected platform only or the selected platform and the referenced platform?

That is a really good point, I didn't consider the referenced platform, and I must add that in this PR.

And if the selected platform and the referenced platform both have the tool (Which is possible but I have never seen it) which is selected?

The selected platform of course.

Thanks for the feedback @jantje

@cmaglie cmaglie added this to the Arduino CLI 1.0 milestone Oct 14, 2022
@cmaglie cmaglie marked this pull request as draft October 20, 2022 13:02
@cmaglie cmaglie force-pushed the fix_tools_selection branch 3 times, most recently from 579f4bf to 449770d Compare November 24, 2022 10:48
@cmaglie cmaglie marked this pull request as ready for review November 24, 2022 12:13
@cmaglie cmaglie force-pushed the fix_tools_selection branch 2 times, most recently from 5bc5bbb to 87abb68 Compare January 17, 2023 19:05
@cmaglie cmaglie force-pushed the fix_tools_selection branch from 87abb68 to 8a44930 Compare January 30, 2023 15:05
Copy link
Contributor

@umbynos umbynos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM

@cmaglie cmaglie merged commit b894e12 into arduino:master Jan 31, 2023
@cmaglie cmaglie deleted the fix_tools_selection branch January 31, 2023 08:45
@matthijskooijman
Copy link
Collaborator

I just tested git master and this also solves an issue I had with a git checkout of the stm32duino core that had been annoying for quite some time (but I could fix it by just retrying and it would randomly work or not work, so wasn't annoying enough to investigate ;-p). Thanks!

@cmaglie
Copy link
Member Author

cmaglie commented Feb 6, 2023

so wasn't annoying enough to investigate

We should increase the annoyance then :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
criticality: low Of low impact priority: high Resolution is a high priority topic: code Related to content of the project itself type: imperfection Perceived defect in any part of project
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incorrect compiler selected by Arduino-cli Issue with platform tool dependency resolving
4 participants