How to use an (avr) core from git?

[matthijskooijman]

I often work on the AVR core files, so I have a checkout of the ArduinoCore-avr repository on my system that I want to usually use. Previously, I just had a symlink from ~/Arduino/hardware/arduino/avr to the ArduinoCore-avr checkout and that was used as the AVR core by the IDE.

However, I've just updated to the latest git (I'm also running the IDE from git) and this no longer works. I'm not sure what change broke it exactly, since reverting to (what I think was) the previous version does not unbreak it (so maybe I had some extra local changes before in e.g. the build/linux folder, or maybe a custom arduino-builder version too?).

The problem I'm seeing now is that:

• Arduino-builder gets -hardware options in order "build/linux/work/hardware", "~/.arduino15/packages", "Sketchbook/hardware":

  Arduino/arduino-core/src/cc/arduino/Compiler.java

  Lines 241 to 243 in 6116a8e

  	addPathFlagIfPathExists(cmd, "-hardware", BaseNoGui.getHardwareFolder());
  	addPathFlagIfPathExists(cmd, "-hardware", installedPackagesFolder);
  	addPathFlagIfPathExists(cmd, "-hardware", BaseNoGui.getSketchbookHardwareFolder());

It seems that this also determines the priority, meaning that any bundled AVR-core, or any boards-manager installed core is used before my core from git in my sketchbook. I do not have any boards-manager installed core, so to use my custom core I removed the build/linux/work/hardware/arduino folder, which makes arduino-builder use the sketchbook core, but then the IDE seems to no longer know about avrdude:

java.io.IOException: Cannot run program "{runtime.tools.avrdude.path}/bin/avrdude": error=2, No such file or directory

This is confirmed by --get-pref, running arduino --get-pref 2>/dev/null |grep avrdude returns nothing.

I guess this makes sense: Tools are defined by .json files, but I have none (for avr) in my ~/.arduino15 directory. I do have build/linux/work/hardware/package_index_bundled.json which defines an avrdude tool, but (I guess) since there is no corresponding AVR core directory (I deleted it), the tool is not loaded.

I tried:

• Putting a symlink to the git avr core in build/linux/work/hardware/arduino/avr, which works, but only if the version number in the package_index_bundled.json file matches the version number in the core's platform.txt. If not, the avrdude is not made available. Since I sometimes jump around different versions, this is not really convenient.
• Keeping the original build/linux/work/hardware/arduino/avr directory, but giving my core from git a higher version number in its platform.txt (hoping that arduino-builder would use the highest version number), but that did not work (avrdude works, but arduino-builder uses the bundled avr core).
• Installing an avr core with the boards manager and replace build/linux/work/hardware/arduino/avr with a symlink to the git core. I thought this woujld let the board-manager installed version make its tools available (so avrdude would work), but the symlink in build/linux/... has higher priority for arduino-builder, so that core is used. However, I now get java.io.IOException: Cannot run program "___REMOVE___/bin/avrdude": error=2, No such file or directory, so apparently it does recognize the tool, but it has a null path somehow...
• Keeping the original bundled core and linking to the git core as ~/Arduino/hardware/arduino-git/avr. This changes the core name and lets the core be installed alongside the regular avr core rather than overriding it (which causes all boards to show up in the boards menu twice). This needs a change in the git core's name in platform.txt to be able to distinguish them and makes the board menu even longer, but I guess this is a reasonable way to do this.

What is noteworthy is that this problem does not seem to occur in arduino-builder, which has no trouble finding avr-gcc (which is also a tool and is also referenced through runtime.tools.avr-gcc.path), so it seems the IDE and arduino-builder load their tools differently (but I've never really dug in to this stuff to really understand it...).

I guess it comes down to the handling of cores that do not have a corresponding .json file to define their dependencies. I guess this also goes for other third-party cores installed directly into the sketchbook, though those typically use the tools from the e.g. the official AVR or SAM cores. Since these third-party cores have a distinct name and offer boards distinct from the official boards, the priority used by arduino-builder is not a problem for them. My ~/Arduino/hardware/arduino-git approach above essentially turns the git core into a distinct core as well, which is why that works.

So I wonder: Are there better ways to support this usecase? If not, is there anything we should or could change to make this easier?

[facchinm]

I use the ~/Arduino/hardware/arduino-git approach, which duplicates the entries but doesn't mess with the ARDUINO_ARCH_XXXX defines as long as the subbolder gets renamed from ArduinoCore-xxx to xxx.

About the difference between previous workflow and the 1.8.10., I suspect the culprit could be one between #9023 (which reworks the json db) and arduino/arduino-builder#319 (which has the side effect of being picky about paths).

I'll investigate a bit more as soon as the merge week is over 🙂

[Technetium1]

@facchinm Just a reminder that this issue still exists, and it's also after merge week 🙂

[matthijskooijman]

I've since solved this problem by using the ~/Arduino/hardware/arduino-git approach (i.e. clone this repo as ~/Arduino/hardware/arduino-git/avr so it is available in addition to the board-manager installed core rather than replacing it. A while ago, another PR was merged which marks these cores with "(in sketchbook)" so you don't have to modify the name in platform.txt anymore, and with the boards now showing in submenus per core, this option is a lot more feasible.

Having said that, I believe that simply using ~/Arduino/hardware/arduino/avr for the clone should behave consistently (either it should replace the board manager or bundled core and work, or be ignored, but IIRC the behavior was that it would be used but with the tools breaking (and @Technetium1 suggests this is still the case).