arduino:avr platform version conflict between Arduino CLI and Arduino IDE 1.8.6/1.8.7

[bxparks]

I'm using the latest binary built from $ go get -u github.com/arduino/arduino-cli.

There seems to be a version conflict between the cores used by Arduino-CLI and the built-in cores in the Arduino IDE.

• IDE 1.8.6 and 1.8.7 come bundled with AVR core 1.6.22/1.6.23 (respectively)
• $ arduino-cli core install arduino:avr installs AVR core 1.6.21, since AVR core 1.6.22/1.6.23 have not been released (don't know why)
• this causes IDE 1.8.6/1.8.7 to revert back to AVR 1.6.21
• now, use the IDE Board Manager to upgrade to AVR 1.6.22/1.6.23
• this causes $HOME/.arduino15/packages/arduino/hardware/avr/1.6.21 to be removed
• which causes $ arduino-cli core list to return an empty list

Possibly related to Issue #28.

(Edited: clarify that IDE 1.8.6 comes bundled with AVR 1.6.22, and IDE 1.8.7 comes with AVR 1.6.23).

[nelsonov]

In the discussion below:
IDE = Arduino IDE 1.8.7
arduino-cli = arduino-cli version 0.3.0-alpha.preview

wget http://downloads.arduino.cc/cores/avr-1.6.23.tar.bz2 works, so the file exists, but it is not listed in https://downloads.arduino.cc/packages/package_index.json.

1.6.21 isn't even available through the list of versions for "Arduino AVR Boards" in the IDE's Board Manager. It jumps from 1.6.20 to 1.6.23. However, according to my testing, you can install 1.6.21 through arduino-cli and the IDE accepts it. If, afterwards, you upgrade through the IDE, then you will be back at the situation that OP describes.

The following process should undo the "damage" done by the IDE and leave you with 1.6.21 in both the IDE and arduino-cli. The arduino-cli core install command is the only one that should really be necessary, but if that doesn't work, then try the whole process. Also, the version number in the install command isn't necessary, but it removes doubt about what is getting installed.

> arduino-cli core update-index
Updating index: package_index.json downloaded

> arduino-cli core search avr
Searching for platforms matching 'avr'

ID                  	Version	Name                          
arduino:avr         	1.6.21 	Arduino AVR Boards            
atmel-avr-xminis:avr	0.6.0  	Atmel AVR Xplained-minis      
emoro:avr           	3.2.2  	EMORO 2560                    
littleBits:avr      	1.0.0  	littleBits Arduino AVR Modules

> arduino-cli core install arduino:[email protected]
Downloading arduino:[email protected]...
arduino:[email protected] downloaded                            
Downloading arduino:[email protected]...
arduino:[email protected] downloaded                                       
Downloading arduino:[email protected]...
arduino:[email protected] downloaded                                             
Downloading arduino:[email protected]...
arduino:[email protected] downloaded                                                   
Installing arduino:[email protected]
arduino:[email protected] installed
Installing arduino:[email protected]
arduino:[email protected] installed
Installing arduino:[email protected]
arduino:[email protected] installed
Updating arduino:[email protected] with arduino:[email protected]...
arduino:[email protected] installed

Just for kicks, I tried to be clever and try this, but, as expected, it didn't work. There's no 1.6.23 for avr in package_index.json

> arduino-cli core install arduino:[email protected]
Error: required version 1.6.23 not found for platform arduino:avr
Could not determine platform dependencies

[cmaglie]

The AVR core version 1.6.23 was missing from the package_index.json for an error on our server, now it has been added, and everything should be back to normal, so I'm going forward and closing the issue.

If the problem persists please comment further and I'll reopen.

[bxparks]

I don't want to be too OCD about this, but it is not clear to me that updating package_index.json is the correct solution. As far as I can tell, arduino-cli looks only in $HOME/.arduino15, so it does not know about the cores built-in to the Arduino IDE. This can cause confusion, and here are two examples:

1. Disappearing AVR core:

• IDE 1.8.7 come with AVR core 1.6.23, but arduino-cli does not know about it.
• Use arduino-cli core install arduino:avr, which installs a duplicate in $HOME/.arduino15/.../avr/1.6.23/.
• Use the IDE board manager to downgrade to 1.6.22. The .../avr/1.6.23 is removed and replaced with .../avr/1.6.22.
• Use the IDE board manager to upgrade to 1.6.23. The .../avr/1.6.22 is removed, and replaced with nothing because the IDE knows that it has 1.6.23 builtin.
• Now arduino-cli no longer knows about 1.6.23.

2. Upgrading to new IDE:

• Suppose a new version of the IDE (say 1.8.8), comes with a new version of the AVR core (say 1.6.24).
• Since this is a builtin to the IDE, arduino-cli does not know about it, and continues to use 1.6.23 (or whatever old version it is stuck on).

The fundamental problem seems to be that arduino-cli does not know about the built-in cores of the IDE. I don't know if that is an intentional design of the arduino-cli, but I think if arduino-cli is meant to be used as command line independent of the IDE, the version mismatch problem could be confusing to some users.

[cmaglie]

Currently the CLI detects if it is bundled with the IDE by checking if the executable is inside the Arduino IDE folder, you can see with the --debug option for example:

$ arduino-cli version --debug
INFO[0000] Initiating configuration                     
INFO[0000] Unserializing configurations from /home/megabug/Workspace/go/bin/.cli-config.yml 
INFO[0000] Checking if CLI is Bundled into the IDE      
INFO[0000] Candidate IDE Directory: /home/megabug/Workspace/go/bin 
INFO[0000] CLI is not bundled into the IDE              <--------
INFO[0000] Configuration set                            
INFO[0000] Starting root command preparation (`arduino`) 
INFO[0000] Formatter set                                
INFO[0000] Calling version command on `arduino`         
arduino-cli version 0.3.1-alpha.preview
$

The cli determine if it is bundled by checking if the arduino-cli executable is in the same folder with the classic arduino from the java IDE (and some other simple checks on the files).
If the above check is successful then the CLI will use the core present in the hardware directory of the Arduino IDE, in other words it will actually use the AVR core that is bundled with the Java IDE.

Anyway, the CLI commands core install/uninstall will still work only on the DATA_DIR (.arduino15/packages/... on linux) and they will install/uninstall the AVR core as you described in your last comment, so the behavior in this case will be the same.

The fundamental problem seems to be that arduino-cli does not know about the built-in cores of the IDE. I don't know if that is an intentional design of the arduino-cli, but I think if arduino-cli is meant to be used as command line independent of the IDE, the version mismatch problem could be confusing to some users.

I see your point, but how can this be improved?

[bxparks]

Thanks for taking the time to explain things.

It seems that my expectation of the Arduino-CLI behavior is different than what you describe. I expected Arduino-CLI to follow the same logic as the Arduino IDE when looking for the various cores, so I expected the CLI to know about both the built-in Cores in {arduino-ide-install-dir}/hardware/ and the additional cores in $HOME/.arduino15/.

It seems to me that the arduino_data: parameter in .cli-config.yml should point to the install directory of the Arduino IDE, not the $HOME/.arduino15 directory. And the Arduino-CLI should follow the exact same search path and algorithm as the Arduino IDE to find the latest matching core of the given board type.

For testing purposes, I currently have 4 separate installations of the Arduino IDE: 1.8.5, 1.8.6, 1.8.7, 1.8.7-clean (fresh install w/o modifications). I use the Portable IDE configuration so that each IDE installation uses the {arduino-ide-dir}/portable/ directory instead of the $HOME/.arduino15 directory. This set up prevents one version of the IDE from interfering with another version of the IDE. I don't know if different Cores versions of different boards are actually compatible with different versions of the IDE, and this setup keeps everything separate and isolated.

In the mental model that I have for the Arduino-CLI workflow, I want to point it to a specific IDE installation directory, and have it build and compile in an environment that mimics the target Arduino IDE installation as closely as possible. Right now, the Arduino-CLI does not know about the Cores built-in to the IDE, so this is not possible.

I understand that eventually you want the Arduino IDE to use the built-in Arduino-CLI tools. It seems like my suggested change to the arduino_data: is compatible with that goal, and actually might make it easier to achieve that.

[cmaglie]

It seems to me that the arduino_data: parameter in .cli-config.yml should point to the install directory of the Arduino IDE, not the $HOME/.arduino15 directory. And the Arduino-CLI should follow the exact same search path and algorithm as the Arduino IDE to find the latest matching core of the given board type.

We always called data_dir the .arduino15 folder, I'd like to avoid to change this meaning now, but nothing stops us to add a new directive arduino_ide_dir for example. Moreover in "portable" mode the IDE will use the portable folder as data_dir.

So following your suggestions a possible solution would be:

• if you set arduino_ide_dir=IDEDIR in the cli-config the CLI will check for bundled hardware in IDEDIR/hardware
• if IDEDIR/portable directory exists then data_dir is set to IDEDIR/portable

this will allow you to "link" the CLI to a specific IDE, even if the arduino-cli executable is not placed in the same folder of the IDE.

How does it sounds?

[bxparks]

It sounds like there are 3 use cases:

1. Arduino IDE uses its own Arduino-CLI binaries. In this case, I think the IDE should provide the entire configuration context for the Arduino-CLI. Conceptually, the IDE is defining the .cli-config.yml and passes that along to the Arduino-CLI. (Implementation wise, this could be achieved by the arduino-cli recognizing that it is installed within the Arduino IDE, and determining its configuration based upon that information.) It seems to me in this use case, arduino-cli should NOT read $HOME/.cli-config.yml because the IDE is in charge of its configuration.

2. User uses the Arduinio-CLI binaries installed by the Arduino IDE. I think you have 2 options here:

   • Option 1: Allow arduino-cli to read the $HOME/.cli-config.yml to determine its configuration, or
   • Option 2: Disallow arduino-cli to read the $HOME/.cli-config.yml, and forcing it to use only its enclosing Arduino IDE configuration.

   I think Option 2 makes the most sense, because it is too confusing to allow Option 1. Option 1 allows me to run the {arduino-ide-1.8.5}/arduino-cli binary, and point it to the install directory of {arduino-ide-1.8.7}/ if my $HOME/.cli-config.yml was configured that way. But I think that is too flexible and potentially confusing.

   It is far simpler to explain to users that if they run the arduino-cli binary that is installed with a particular Arduino IDE, then the arduino-cli will act just like the IDE, and see exact same list of Cores that the IDE can see.

3. User installs a stand-alone version of Arduino-CLI. Here the arduiino-cli should read the $HOME/.cli-config.yml file, because it needs to know where to find the various Cores.

With regards to the new arduino_ide_dir parameter, yes that makes sense. In the Use Case 1 and 2 above, the IDE is effectively settings it to ./ or ../ (depending on where the arduino-cli is installed). But it may interact with the data_dir parameter in unexpected ways. I haven't thought all the details, but at a high level, my expectation is that arduino-cli follows the same search path as the IDE, which I think is the following:

• If data_dir is defined, then it uses that, otherwise,
• If ./portable exists, then it uses that, otherwise,
• If $HOME/.arduino15 exists, then it uses that,

[Perlovka]

Currently the CLI detects if it is bundled with the IDE by checking if the executable is inside the Arduino IDE folder
This is the wrong way. Usually Linux distribution packages will not install bundles at all. For example in Gentoo, Arduino IDE and arduno-builder are two separate packages, and will never be installed to the same path. Same will apply to arduino-cli.

I think it is better just to add configuration options for custom hardware/libraries paths.
This will also allow to define directories located on removable media.
Like this:

directories:
  cache: /home/perlovka/.cache/arduino
  config: /home/perlovka/.config/arduino
  data: /home/perlovka/.local/share/arduino
  downloads: /home/perlovka/.cache/arduino/staging
  user: /home/perlovka/develop/arduino
  custom_hardware:
    - /usr/share/arduino/hardware
    - /mnt/portable/arduino_hardware
  custom_libraries:
    - /usr/share/arduino/libraries
    - /mnt/portable/arduino_libs

here is a quick hack to allow additional hardware directories
(not completely tested, but at least arduino-cli could see my avr core, located in /usr/share/arduino/hardware):

diff -ur arduino-cli-0.19.3-orig/configuration/directories.go arduino-cli-0.19.3-hw/configuration/directories.go
--- arduino-cli-0.19.3-orig/configuration/directories.go	2021-10-11 18:14:04.000000000 +0300
+++ arduino-cli-0.19.3-hw/configuration/directories.go	2021-11-06 10:51:32.546885802 +0200
@@ -47,6 +47,13 @@
 		}
 	}
 
+    if settings.IsSet("directories.Custom_hardware") {
+        chDirs := settings.GetStringSlice("directories.Custom_hardware")
+        for _, chDir := range chDirs {
+            res.Add(paths.New(chDir))
+        }
+    }
+
 	return res
 }