Recover from interrupted discovery/monitor installation

[per1234]

Describe the request

Reinstall pluggable discovery and monitor tools of the builtin package if the tool executable is not found.

Describe the current behavior

Arduino CLI automatically installs pluggable discovery and pluggable monitor tools of the builtin package when missing or updatable. These tools allow Arduino CLI to find the "ports" used for communication with Arduino boards.

It can happen that the installation process is interrupted or interfered with.

🙁 If a problem during the installation results in the creation of the tool's release installation folder without the tool executable, Arduino CLI can no longer do discovery. It will not automatically recover from this condition and the user must manually delete the incomplete installation in order to restore Arduino CLI to functionality.

To reproduce

$ arduino-cli version
arduino-cli.exe  Version: git-snapshot Commit: 2c2a5cc6 Date: 2023-04-22T13:21:59Z

$ ls "C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\builtin\\tools\\serial-discovery"
1.4.0/

$ ls "C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\builtin\\tools\\serial-discovery\\1.4.0"
LICENSE.txt  serial-discovery.exe*

$ rm "C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\builtin\\tools\\serial-discovery\1.4.0\\serial-discovery.exe"

$ arduino-cli board list
Error starting discovery: discovery builtin:serial-discovery process not started: exec: "C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\builtin\\tools\\serial-discovery\\1.4.0\\serial-discovery": file does not exist
No boards found.

Arduino CLI version

2c2a5cc

Operating system

Windows, Linux

Operating system version

Windows 11, Ubuntu 22.04

Additional context

Originally reported at https://forum.arduino.cc/t/ide-2-0-lost-all-com-ports/960650

---

This situation is more problematic for Arduino IDE users, since the error message about the missing tool is not visible to them and they only notice the symptom of the IDE not seeing their ports.

---

It might seem unlikely for this interrupted installation to occur under real world conditions, but the equivalent issue with Boards Manager producing incomplete tools installations has been reported by users periodically ever since the introduction of Boards Manager, at sufficient volume for Arduino Support to consider it worth a Help Center article:

https://support.arduino.cc/hc/en-us/articles/360018444739-Error-file-does-not-exist-no-such-file-or-directory-system-cannot-find-the-file-specified

[per1234]

This now results in a cryptic panic that gives no clue about the cause to mere mortals:

$ arduino-cli version
arduino-cli  Version: 0.25.1-rc1 Commit: 436f0bb9 Date: 2022-07-23T14:28:16Z

$ rm ~/.arduino15/packages/builtin/tools/serial-discovery/1.3.2/*

$ ./arduino-cli board list
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4a51d9]

goroutine 35 [running]:
os.(*Process).signal(0xc000174770, {0x1034aa0, 0x15fd848})
	/usr/local/go/src/os/exec_unix.go:64 +0x39
os.(*Process).Signal(...)
	/usr/local/go/src/os/exec.go:138
os.(*Process).kill(...)
	/usr/local/go/src/os/exec_posix.go:68
os.(*Process).Kill(...)
	/usr/local/go/src/os/exec.go:123
github.com/arduino/arduino-cli/executils.(*Process).Kill(...)
	/home/per/Documents/deleteme/arduino-cli/executils/process.go:120
github.com/arduino/arduino-cli/arduino/discovery.(*PluggableDiscovery).killProcess(0xc0000b5480)
	/home/per/Documents/deleteme/arduino-cli/arduino/discovery/discovery.go:268 +0xc9
github.com/arduino/arduino-cli/arduino/discovery.(*PluggableDiscovery).Quit(0xc0000b5480)
	/home/per/Documents/deleteme/arduino-cli/arduino/discovery/discovery.go:381 +0xde
github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).remove(0xc000070a20, {0xc000017dfe, 0x18})
	/home/per/Documents/deleteme/arduino-cli/arduino/discovery/discoverymanager/discoverymanager.go:82 +0xba
github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).RunAll.func1(0xc0000b5480)
	/home/per/Documents/deleteme/arduino-cli/arduino/discovery/discoverymanager/discoverymanager.go:131 +0x71
github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).parallelize.func1(0x1)
	/home/per/Documents/deleteme/arduino-cli/arduino/discovery/discoverymanager/discoverymanager.go:101 +0x68
created by github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).parallelize
	/home/per/Documents/deleteme/arduino-cli/arduino/discovery/discoverymanager/discoverymanager.go:99 +0x1c5

The demo was done using Linux, but the problem also occurs on Windows.

[cmaglie]

Is this happening when there is only the directory without the executable?

[per1234]

That is correct.

$ arduino-cli version
arduino-cli.exe  Version: 0.25.1-rc1 Commit: 436f0bb9 Date: 2022-07-22T15:47:42Z

$ arduino-cli board list
Port   Protocol Type              Board Name                FQBN             Core       
COM1   serial   Serial Port       Unknown
COM254 serial   Serial Port (USB) Unknown
COM3   serial   Serial Port       Unknown
COM52  serial   Serial Port (USB) Unknown
COM7   serial   Serial Port (USB) Arduino Mega or Mega 2560 arduino:avr:mega arduino:avr

$ rm ~/AppData/Local/Arduino15/packages/builtin/tools/serial-discovery/1.3.2/serial-discovery.exe

$ arduino-cli board list
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x8 pc=0xdd27cd]

goroutine 40 [running]:
os.(*Process).signal(0xc000150770, {0x196b990, 0x1f386e0})
        C:/Program Files/Go/src/os/exec_windows.go:49 +0x4d
os.(*Process).Signal(...)
        C:/Program Files/Go/src/os/exec.go:138
os.(*Process).kill(...)
        C:/Program Files/Go/src/os/exec_posix.go:68
os.(*Process).Kill(...)
        C:/Program Files/Go/src/os/exec.go:123
github.com/arduino/arduino-cli/executils.(*Process).Kill(...)
        E:/electronics/git-nobackup/arduino-cli/executils/process.go:120
github.com/arduino/arduino-cli/arduino/discovery.(*PluggableDiscovery).killProcess(0xc000985c00)
        E:/electronics/git-nobackup/arduino-cli/arduino/discovery/discovery.go:268 +0xc9
github.com/arduino/arduino-cli/arduino/discovery.(*PluggableDiscovery).Quit(0xc000985c00)
        E:/electronics/git-nobackup/arduino-cli/arduino/discovery/discovery.go:381 +0xde
github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).remove(0xc0003782e0, {0x1716256, 0x18})
        E:/electronics/git-nobackup/arduino-cli/arduino/discovery/discoverymanager/discoverymanager.go:82 +0xba
github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).RunAll.func1(0xc000985c00)
        E:/electronics/git-nobackup/arduino-cli/arduino/discovery/discoverymanager/discoverymanager.go:131 +0x71
github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).parallelize.func1(0x2e0031002e0032)
        E:/electronics/git-nobackup/arduino-cli/arduino/discovery/discoverymanager/discoverymanager.go:101 +0x68
created by github.com/arduino/arduino-cli/arduino/discovery/discoverymanager.(*DiscoveryManager).parallelize
        E:/electronics/git-nobackup/arduino-cli/arduino/discovery/discoverymanager/discoverymanager.go:99 +0x1c5

[per1234]

#1814 fixed the panic I reported in my previous comment, but it did not resolve the feature request for Arduino CLI to recover from a situation where a discovery executable goes missing.

$ arduino-cli version
arduino-cli  Version: nightly-20220808 Commit: d0b2556 Date: 2022-08-08T01:35:05Z

$ rm ~/.arduino15/packages/builtin/tools/serial-discovery/1.3.2/*

$ arduino-cli board list
Error detecting boards: Error starting board discoveries: [discovery builtin:serial-discovery process not started: fork/exec /home/per/.arduino15/packages/builtin/tools/serial-discovery/1.3.2/serial-discovery: no such file or directory]
No boards found.

[per1234]

I'm reopening this issue because, although a nice improvement in this area, #2107 is not a complete solution.

The most common conditions where the user's encounter this issue are that only the executable file is missing (most often due to being quarantined by the user's security software). #2107 detects the broken installation state only when the tool release installation folder is empty (per the issue's original simplified demo, which cleared the entire folder), but we are including a LICENSE.txt file in the discovery and monitor tools installations. This means that if only the essential executable goes missing, the directory is not empty and the recovery mechanism is not activated:

$ arduino-cli version
arduino-cli.exe  Version: git-snapshot Commit: 2c2a5cc6 Date: 2023-04-22T13:21:59Z

$ ls "C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\builtin\\tools\\serial-discovery"
1.4.0/

$ ls "C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\builtin\\tools\\serial-discovery\\1.4.0"
LICENSE.txt  serial-discovery.exe*

$ rm "C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\builtin\\tools\\serial-discovery\1.4.0\\serial-discovery.exe"

$ arduino-cli board list
Error starting discovery: discovery builtin:serial-discovery process not started: exec: "C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\builtin\\tools\\serial-discovery\\1.4.0\\serial-discovery": file does not exist
No boards found.

[silvanocerza]

May I suggest installing in a temporary directory and then only after everything has been extraced copy it over to the final installation path?

If even a single file copy fails abort the whole process and remove the ones that have been copied.

Also I's suggest blocking the possibility of interrupting the execution to prevent it being interrupted during steps that would leave files behind.

[silvanocerza]

Also to check if the installation is valid I'd create a checksum of the whole folder when it's first installed.
Then in IsInstalled() compare it with the current content checksum to verify its integrity, if they differ it's not installed.

[cmaglie]

May I suggest installing in a temporary directory and then only after everything has been extraced copy it over to the final installation path?

The issue here is that the copy is successful, but the antivirus will move the file into quarantine after the installation has been completed.

Also to check if the installation is valid I'd create a checksum of the whole folder when it's first installed.
Then in IsInstalled() compare it with the current content checksum to verify its integrity, if they differ it's not installed.

This is an expensive operation to do at every run of the CLI....

[silvanocerza]

The issue here is that the copy is successful, but the antivirus will move the file into quarantine after the installation has been completed.

We could add an after install check to verify the integrity and warn the user that installation wasn't successfull cause of the antivirus maybe.

Even using the same checksum strategy, assuming the file is removed immediately after the copy.

This is an expensive operation to do at every run of the CLI....

Fair enough, less expensive and frustrating than a bad UX for me though.

Maybe do it only when the CLI is trying to run a certain tool, and act accordingly.

[cmaglie]

Given the very simple format of the discoveries, I guess that just checking if the executable is there is enough.

[cmaglie]

Fair enough, less expensive and frustrating than a bad UX for me though.

UX wise you are adding a fixed delay to 100% of the users to solve an issue that 0.01% are experiencing...

[silvanocerza]

Given the very simple format of the discoveries, I guess that just checking if the executable is there is enough.

Yeah, if the issue is present only for discoveries it would make sense to check only for those.

UX wise you are adding a fixed delay to 100% of the users to solve an issue that 0.01% are experiencing...

True that, I have no metrics either way. 😉