Boards manager install of older avr-gcc breaks compilation for all AVR boards in 1.6.10.

[damellis]

A user of the attiny boards package encountered this: damellis/attiny#105

The previous ATtiny package json (https://raw.githubusercontent.com/damellis/attiny/f026e80bc8fa4e33714bfe28ceafdd8829d85293/package_damellis_attiny_index.json) included explicit dependencies on the avr-gcc and avrdude versions that came with Arduino 1.6.9 (and previous). With 1.6.10, newer versions of these tools are included, so installing the ATtiny package also installed these older tools. These older tools then get used for all AVR boards.

It appears the older tools don't support LTO or plugins, so compiling in 1.6.10 gives the error:

sorry - this program has been built without plugin support
exit status 1

I think I fixed this for the ATtiny package by removing the explicit tool dependency, and assuming that avr-gcc and avrdude will be included with the Arduino IDE.

This may remain a problem for other third-party hardware packages, though. Maybe the IDE should use the bundled avr-gcc and avrdude for bundled boards even if another version has been installed for other boards? Maybe only if the newly-installed version is older than the bundled version?

[damellis]

Also, is there a way for a third-party hardware package to specify dependency on a tool, but not a particular version of it? Or to depend on a range of versions (e.g. some particular version or later)?

[jantje]

Work around: install 1.11 avr arduino toolchain
Note currently you can not install 1.12.
This means the optimisations are no longer available. As soon as 1.13 comes out (or is specified in the json file) you can use 1.13 and have the optimisations.
Root cause of the issues: preinstalled toolchains are not properly identified.

[facchinm]

Hi everyone,
we spot the bug too late (the fact that the PR adding the new toolchains had a hack to overcome an IDE limitation is the key here) and we are working on a fast solution (making the builder more intelligent). Expect some bugfix release really soon. Sorry for any inconvenience.

[jantje]

Thanks for confirming this is a bug.

[MCUdude]

@per1234 does this means that I can just leave the JSON file as it is, or just upgrade the Tooldependencies field?

[per1234]

@MCUdude I won't know for sure until there is an hourly build with a fix that I can test. It does possibly offer the option of just letting the issue be fixed in the IDE. The problem with that solution is anyone who has installed Arduino IDE 1.6.10 in the meantime remains vulnerable to having their Arduino AVR Boards core broken by installing any of your cores.

I'm glad to see facchinm is working on a fix.

[cmaglie]

The attiny:avr package uses the core from the arduino:avr package as specificed in the boards txt with the line:

ATtinyX5.build.core=arduino:arduino

This means that the package attiny:avr needs the same set of tools but also the platform.txt and the arduino core from the package arduino:avr. When we changed platform.txt of arduino:avr, to enable LTO, we also break the attiny:avr package because it referenced explicitly the older set of tools that did not support LTO. So in this case removing the explicit reference to the old tools is a good fix.

Also, is there a way for a third-party hardware package to specify dependency on a tool, but not a particular version of it? Or to depend on a range of versions (e.g. some particular version or later)?

The real issue is that the package_index.json misses the information that attiny:avr uses arduino:avr and, at the moment, there is no way to explicitly state this.

A better solution IMHO is to add a new field coreDependencies in the package_index.json , something like this:

        {
          "name": "attiny",
          "architecture": "avr",
          "version": "1.0.1",
          "category": "attiny",
          "help": {
            "online": ""
          },
          "url": "https://github.com/damellis/attiny/archive/76ca88ce3c195fcba8e8a645084e352965dc80e0.zip",
          "archiveFileName": "76ca88ce3c195fcba8e8a645084e352965dc80e0.zip",
          "checksum": "SHA-256:49a4dfc3796caf2533050b0b962ae0affb2471b8ded1283936d7b712cc1fdff1",
          "size": "5719",
          "boards": [
            {"name": "ATtiny45"},
            {"name": "ATtiny85"},
            {"name": "ATtiny44"},
            {"name": "ATtiny84"}
          ],
+         "coreDependencies": [
+             { "packager": "arduino" }
+         ],
          "toolsDependencies": [
-           {
-             "packager": "arduino",
-             "name": "avr-gcc",
-             "version": "4.8.1-arduino5"
-           },
-           {
-             "packager": "arduino",
-             "name": "avrdude",
-             "version": "6.0.1-arduino5"
-           }
          ]
        }

Doing this way we know explicitly that attiny:avr uses arduino:avr and therefore inherits all the tools from arduino:avr (no need to specify them).

[jantje]

Doing this way we know explicitly that attiny:avr uses arduino:avr and therefore inherits all the tools from arduino:avr (no need to specify them).

... for the arduino IDE.

[cmaglie]

... for the arduino IDE.

@jantje
may you be a bit more explicit?

You may be surprised, but I proposesd this change to help 3rd party tool developers, in particular you, to gather all the information needed exclusively from the json package_indexes.

[jantje]

@cmaglie
I'm just stating that the Arduino eclipse plugin named sloeber does need to have the tools specified to work properly. I assume there are other IDE's around that need this as well.
As the data is static I do not think there is a big gain in not specifying.

[cmaglie]

Arduino eclipse plugin named sloeber does need to have the tools specified to work properly.

I know, that's why I proposed to remove toolDependency but at the same time add coreDependencies.

Look at the attiny:avr example above: it uses the core (and consequently the same tools) of arduino:avr.
So why repeating? just state that with:

         "coreDependencies": [
             { "packager": "arduino" }
         ],

and you know that attiny:avr needs the same tools from arduino:avr (that are specified in the main package_index.json).

from package_index.json:

{
  "packages": [
    {
      "name": "arduino",
      "maintainer": "Arduino",
      "websiteURL": "http://www.arduino.cc/",
      "email": "[email protected]",
      "help": {
        "online": "http://www.arduino.cc/en/Reference/HomePage"
      },
      "platforms": [
         ...SNIP...
        {
          "name": "Arduino AVR Boards",
          "architecture": "avr",
          "version": "1.6.12",
          ...SNIP...
          "toolsDependencies": [
            {
              "packager": "arduino",
              "name": "avr-gcc",
              "version": "4.9.2-atmel3.5.3-arduino2"
            },
            {
              "packager": "arduino",
              "name": "avrdude",
              "version": "6.3.0-arduino2"
            }
          ]
        },

As the data is static I do not think there is a big gain in not specifying

The problem is that this data is not always static, in particular packages that uses cores from another package (like attiny but basically 90% of the packages around) needs the same tools from the referenced package that may change when the referenced package is upgraded!

[jantje]

@cmaglie
It is once again clear to me that we come from 2 different planets.
Because of that I decided a long time ago to stop discussing and work with what comes out of the Arduino core team and live with it.
My body advices me to stick to this policy.

[PaulStoffregen]

Expect some bugfix release really soon. Sorry for any inconvenience.

Version 1.6.11 ??

[cmaglie]

It is once again clear to me that we come from 2 different planets.

@jantje
may I know what could be a reasonable solution to this issue for you?

Version 1.6.11

@PaulStoffregen
that's the idea

[jantje]

@cmaglie
As I don't see the difference between this issue and MCUdude/MightyCore#31 (comment) I don't think I can help.

[cmaglie]

The difference is that MightyCore is a totally independent package because it has this:

https://github.com/MCUdude/MightyCore/tree/master/avr/cores/MightyCore

in the MightyCore case specifing the tools is the correct thing to do.

In the attiny case you won't fint the "core" folder as in the MightyCore, look by yourself:

https://github.com/damellis/attiny

this is because the attiny package provides only some variants, but uses the core from arduino. That's why you can't write a static set of tool for attiny: it depends on another core that may change tools when upgraded.

[jantje]

@cmaglie
To answer your question: "may I know what could be a reasonable solution to this issue for you?"
I think we should first agree on the root cause of the issue. Given the solution you propose I think you think the root cause is in the json specification.
I on the other hand think the root cause is the Arduino IDE implementation of the boards manager in regards to the "preinstalled boards".

Basically I think the boardsmanager does not handle the preinstalled tools and boards in the same way as "additional tools and boards". A obvious proofs of this are the different folder structures and the lack of a json file.

Due to the missing json file the Arduino IDE "has to guess which version of the toolchain and uploader it will use". Which is what OP is pointing at when saying.

The previous ATtiny package json included explicit dependencies on the avr-gcc and avrdude versions that came with Arduino 1.6.9 (and previous). With 1.6.10, newer versions of these tools are included, so installing the ATtiny package also installed these older tools. These older tools then get used for all AVR boards.

In other words: When multiple toolchains are installed for avr, the Arduino IDE is missing the json file of the "pre installed boards" to be able to pick the "correct" toolchain.

[cmaglie]

In other words: When multiple toolchains are installed for avr, the Arduino IDE is missing the json file of the "pre installed boards" to be able to pick the "correct" toolchain.

The json file exists, it's here:

http://downloads.arduino.cc/packages/package_index.json

here the snippet that defines the preinstalled AVR for IDE 1.6.10:

        {
          "name": "Arduino AVR Boards",
          "architecture": "avr",
          "version": "1.6.12",
          "category": "Arduino",
          "help": {
            "online": "http://www.arduino.cc/en/Reference/HomePage"
          },
          "url": "http://downloads.arduino.cc/cores/avr-1.6.12.tar.bz2",
          "archiveFileName": "avr-1.6.12.tar.bz2",
          "checksum": "SHA-256:4a296a3304a51d820e680e2022fb9b9f5706144abf1e5d60c7876a0f9fc7e6d9",
          "size": "4993586",
          "boards": [
            {"name": "Arduino Yún"},
            {"name": "Arduino/Genuino Uno"},
            {"name": "Arduino Diecimila"},
            {"name": "Arduino Nano"},
            {"name": "Arduino/Genuino Mega"},
            {"name": "Arduino MegaADK"},
            {"name": "Arduino Leonardo"},
            {"name": "Arduino/Genuino Micro"},
            {"name": "Arduino Esplora"},
            {"name": "Arduino Mini"},
            {"name": "Arduino Ethernet"},
            {"name": "Arduino Fio"},
            {"name": "Arduino BT"},
            {"name": "Arduino LilyPadUSB"},
            {"name": "Arduino Lilypad"},
            {"name": "Arduino Pro"},
            {"name": "Arduino ATMegaNG"},
            {"name": "Arduino Robot Control"},
            {"name": "Arduino Robot Motor"},
            {"name": "Arduino Gemma"}
          ],
          "toolsDependencies": [
            {
              "packager": "arduino",
              "name": "avr-gcc",
              "version": "4.9.2-atmel3.5.3-arduino2"
            },
            {
              "packager": "arduino",
              "name": "avrdude",
              "version": "6.3.0-arduino2"
            }
          ]
        },

as you can see the tools are specified, but there is a bug in the builder that do not select the correct ones. This is already explained here: arduino/Arduino#5042
Please note that arduino/Arduino#5042 OP referes to two different SAMD packages (not AVR!) that are not preinstalled. So the presumed unfairness of the IDE is an hypotesys that doesn't hold for me.

[jantje]

Ok... I have been investigating this json story a bit deeper due to Sloeber/arduino-eclipse-plugin#516 and .....
I finally understand what is going on here.
I apologize for some reactions caused by my misunderstanding. (sorry for not understanding)

As to the proposed change:

        {
          "name": "attiny",
          "architecture": "avr",
          "version": "1.0.1",
          "category": "attiny",
          "help": {
            "online": ""
          },
          "url": "https://github.com/damellis/attiny/archive/76ca88ce3c195fcba8e8a645084e352965dc80e0.zip",
          "archiveFileName": "76ca88ce3c195fcba8e8a645084e352965dc80e0.zip",
          "checksum": "SHA-256:49a4dfc3796caf2533050b0b962ae0affb2471b8ded1283936d7b712cc1fdff1",
          "size": "5719",
          "boards": [
            {"name": "ATtiny45"},
            {"name": "ATtiny85"},
            {"name": "ATtiny44"},
            {"name": "ATtiny84"}
          ],
+         "coreDependencies": [
+             { "packager": "arduino" }
+         ],
          "toolsDependencies": [
-           {
-             "packager": "arduino",
-             "name": "avr-gcc",
-             "version": "4.8.1-arduino5"
-           },
-           {
-             "packager": "arduino",
-             "name": "avrdude",
-             "version": "6.0.1-arduino5"
-           }
          ]
        }

I would counter propose (may contain formatting issues)

        {
          "name": "attiny",
          "architecture": "avr",
          "version": "1.0.1",
          "category": "attiny",
          "help": {
            "online": ""
          },
          "url": "https://github.com/damellis/attiny/archive/76ca88ce3c195fcba8e8a645084e352965dc80e0.zip",
          "archiveFileName": "76ca88ce3c195fcba8e8a645084e352965dc80e0.zip",
          "checksum": "SHA-256:49a4dfc3796caf2533050b0b962ae0affb2471b8ded1283936d7b712cc1fdff1",
          "size": "5719",
          "boards": [
            {"name": "ATtiny45"},
            {"name": "ATtiny85"},
            {"name": "ATtiny44"},
            {"name": "ATtiny84"}
          ],
+         "platformDependencies": [
+             { "packager": "arduino",
+                 "supportedVersions":[
+                  {"version:""1.6.10"},
+                  {"version:""1.6.11"}
+                  ]
+             }
+         ],
          "toolsDependencies": [
-           {
-             "packager": "arduino",
-             "name": "avr-gcc",
-             "version": "4.8.1-arduino5"
-           },
-           {
-             "packager": "arduino",
-             "name": "avrdude",
-             "version": "6.0.1-arduino5"
-           }
          ]
        }

The problem I have with "coreDependencies" is that currently "core" is not an entity in the json file. So referencing a core is very confusing (to me). -Notice that I do think referencing a core is the right way-
Secondly I would add an optional list of supported versions. When no versions are specified all versions are supposed to be ok.

As to the 2 different planets. I think core should be something platform references. Basically because a core is a library that must be compiled in one (or more -like the teensy core- ) specific way.
The boards.txt is a set of specifications for a specific hardware.
Whether you want to do the same for the hardware dependent libraries is a question that should be asked. My knowledge in this area is insufficient to make a statement about this.

It may be a good idea to make a schema file of the json as it is a pretty complex structure.

[cmaglie]

I apologize for some reactions caused by my misunderstanding. (sorry for not understanding)

No worries. And thanks for looking deeply into to this issue!

The problem I have with "coreDependencies" is that currently "core" is not an entity in the json file. So referencing a core is very confusing (to me). -Notice that I do think referencing a core is the right way-

now that you make me notice I agree, platformDependencies is the correct name.
I've used coreDependencies initially because the need for this dependecy originates from boards.txt with a configuration like:

...
ATtinyX5.build.core=arduino:arduino
...

so the it's the board configuration that is actually referencing a core outside of the board's platform. BTW considering that:

• a core is contained in a platform (to be precise a platform is a container of cores and may have from 0 to N cores)
• the package_index.json doesn't have the concept of core (as you pointed out)

the correct thing to do in the json is to reference the container platform, indeed.

Secondly I would add an optional list of supported versions. When no versions are specified all versions are supposed to be ok.

Version constraints may lead to complex algorithms for conflicts resolution, IMHO we should do a first release without them. Some of the problems that comes to mind are:

• How to handle two different platforms ("A" and "B") that depends on two different version of a 3rd platform "C"?
• How to handle updates? you probably may want to specify ranges something lik 1.6.* or have an operator that selects all the compatible versions following semver like ~1.6.7 that means 1.6.7 <= X < 2.0.0

[jantje]

Version constraints may lead to complex algorithms for conflicts resolution, IMHO we should do a first release without them. Some of the problems that comes to mind are:

This issue popped up because of a upgrade of a toolchain. So if versions are specified I would not allow wildcards. When versions are specified it simply means "We have tested these versions and they work".

I do agree that a first release without them is a good idea.

[rlogiacco]

About version constraining you might want to use the same approach Maven uses: (min.ver,max.ver) and [min.ver,max.ver], but also [min.ver,) using the very same notation used in mathematics for sets.
It's natural and very comprehensive, but it works only if semver is adopted.

[facchinm]

New IDEs use the json definitions to select the right toolchain if no version is specified in platform.txt, thus closing as fixed