Sort Library Manager list by relevance

[kittaakos]

Describe the problem

When I search for SD I expect to see SD to show up in the top of the list so I can install it without a lot of scrolling through the list of results. But instead it is far down in the results.

To reproduce

Open the IDE2 and search for SD. If you're lucky you will find it, but IDE2 limits the search results to 50 hits (UPDATE: fixed by #1361).

Expected behavior

When I search for SD, I see the lib to install near the top of the list of results.

More generally: Arduino published libs are boosted in the search result. (UPDATE: fixed by #1541)

Update: or a match is boosted if it's an exact match.

Arduino IDE version

HEAD of the main

Operating system

macOS

Operating system version

12.3.1

Additional context

It works from the CLI but the output is too long to include it on GH.

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[kittaakos]

This issue is not yet solved, @ubidefeo.

There was a related PR (#1541), bu that boosted the Arduino libs only. This issue is about boosting by (exact) match

👇

More generally: Arduino published libs are boosted in the search result. (UPDATE: fixed by #1541)

Update: or a match is boosted if it's an exact match.

[davegarthsimpson]

@umbynos if I remember correctly the exact match prioritisation we decided to do CLI side, maybe this task needs reassigning or a new card/scope?

[umbynos]

Done in arduino/arduino-cli#1975 will be available in the next release

[per1234]

It is not fixed. Arduino IDE's Library Manager search was not affected by arduino/arduino-cli#1975 (I believe the IDE uses LibraryList instead of LibrarySearch):

[kittaakos]

It is not fixed. Arduino IDE's Library Manager search was not affected by arduino/arduino-cli#1975 (I believe the IDE uses LibraryList instead of LibrarySearch):

IDE2 runs the lib search gRPC equivalent:

arduino-ide/arduino-ide-extension/src/node/library-service-impl.ts

Lines 77 to 81 in fe3fbb1

	const resp = await new Promise<LibrarySearchResponse>((resolve, reject) =>
	client.librarySearch(req, (err, resp) =>
	!!err ? reject(err) : resolve(resp)
	)
	);

The CLI works as expected:

a.kitta@Akoss-MacBook-Pro arduino-ide % ./arduino-ide-extension/build/arduino-cli version                      
arduino-cli  Version: 0.29.0 Commit: 76251df9 Date: 2022-11-17T09:21:47Z
a.kitta@Akoss-MacBook-Pro arduino-ide % ./arduino-ide-extension/build/arduino-cli lib search "sd" --format json
{
  "libraries": [
    {
      "name": "SD",
      "releases": {
        "1.0.0": {
          "author": "Arduino, SparkFun",
          "version": "1.0.0",
          "maintainer": "Arduino \[email protected]\u003e",
          "sentence": "Enables reading and writing on SD cards. For all Arduino boards.",
          "paragraph": "Once an SD memory card is connected to the SPI interfare of the Arduino board you are enabled to create files and read/write on them. You can also move through directories on the SD card.",
          "website": "https://www.arduino.cc/en/Reference/SD",
          "category": "Data Storage",
          "architectures": [
            "*"
          ],
          "types": [
            "Arduino"
          ],

[...]

The problem is that IDE2 does additional sorting, such as

• deprecated last,
• boost results with Arduino,
• etc

IDE2 overrules the order of the search results provided by the CLI.

[kittaakos]

There is a Platform property to indicate if a platform has been deprecated. There is no such thing for the LibraryRelease. However, I have noticed that the lib search can produce results where the types array includes multiple items. These libraries are Robot IR Remote, TFT, and WiFi. These results have the following types array: ["Arduino", "Retired"].

@per1234, is this the standard procedure to deprecate a library? Should IDE2 lower the score for a deprecated library result? Or should IDE2 always take the 0^th index from the types array and ignore the rest? What do you think? Thank you!

Example:

./arduino-cli lib search "Robot IR Remote" --format json

{
  "libraries": [
    {
      "name": "Robot IR Remote",
      "releases": {
        "1.0.0": {
          "author": "Arduino",
          "version": "1.0.0",
          "maintainer": "Arduino \u003c[email protected]\u003e",
          "sentence": "Allows controlling the Arduino Robot via an IR remote control. For Arduino Robot only.",
          "website": "https://github.com/shirriff/Arduino-IRremote",
          "category": "Device Control",
          "architectures": [
            "avr"
          ],
          "types": [
            "Arduino",
            "Retired"
          ],
          "resources": {
            "url": "https://downloads.arduino.cc/libraries/github.com/arduino-libraries/Robot_IR_Remote-1.0.0.zip",
            "archive_filename": "Robot_IR_Remote-1.0.0.zip",
            "checksum": "SHA-256:0b75fb1f0fcb37de415cedd4c4b77663151ae20071687d058379954bd4e35a3e",
            "size": 22941,
            "cache_path": "libraries"
          }
        },
        "1.0.1": {
          "author": "Arduino",
          "version": "1.0.1",
          "maintainer": "Arduino \u003c[email protected]\u003e",
          "sentence": "Allows controlling the Arduino Robot via an IR remote control. For Arduino Robot only.",
          "website": "https://github.com/shirriff/Arduino-IRremote",
          "category": "Device Control",
          "architectures": [
            "avr"
          ],
          "types": [
            "Arduino",
            "Retired"
          ],
          "resources": {
            "url": "https://downloads.arduino.cc/libraries/github.com/arduino-libraries/Robot_IR_Remote-1.0.1.zip",
            "archive_filename": "Robot_IR_Remote-1.0.1.zip",
            "checksum": "SHA-256:593849f857ff48a31191fcdaaf8e42debc37b96985e6eaeb4d0813ea9ef9a0a4",
            "size": 23018,
            "cache_path": "libraries"
          }
        },
        "1.0.2": {
          "author": "Arduino",
          "version": "1.0.2",
          "maintainer": "Arduino \u003c[email protected]\u003e",
          "sentence": "Allows controlling the Arduino Robot via an IR remote control. For Arduino Robot only.",
          "website": "https://github.com/shirriff/Arduino-IRremote",
          "category": "Device Control",
          "architectures": [
            "avr"
          ],
          "types": [
            "Arduino",
            "Retired"
          ],
          "resources": {
            "url": "https://downloads.arduino.cc/libraries/github.com/arduino-libraries/Robot_IR_Remote-1.0.2.zip",
            "archive_filename": "Robot_IR_Remote-1.0.2.zip",
            "checksum": "SHA-256:884b3e2559e2188633238839c53ddb13343aa1aef161769da89f3ec4d3747537",
            "size": 23022,
            "cache_path": "libraries"
          }
        },
        "2.0.0": {
          "author": "Arduino",
          "version": "2.0.0",
          "maintainer": "Arduino \u003c[email protected]\u003e",
          "sentence": "Allows controlling the Arduino Robot via an IR remote control. For Arduino Robot only.",
          "website": "https://github.com/arduino-libraries/RobotIRremote",
          "category": "Device Control",
          "architectures": [
            "avr"
          ],
          "types": [
            "Arduino",
            "Retired"
          ],
          "resources": {
            "url": "https://downloads.arduino.cc/libraries/github.com/arduino-libraries/Robot_IR_Remote-2.0.0.zip",
            "archive_filename": "Robot_IR_Remote-2.0.0.zip",
            "checksum": "SHA-256:13e0125e8b08fa634d5c339a328b2ecf07b5646e926db02f5db605be7bb9c520",
            "size": 23065,
            "cache_path": "libraries"
          }
        }
      },
      "latest": {
        "author": "Arduino",
        "version": "2.0.0",
        "maintainer": "Arduino \u003c[email protected]\u003e",
        "sentence": "Allows controlling the Arduino Robot via an IR remote control. For Arduino Robot only.",
        "website": "https://github.com/arduino-libraries/RobotIRremote",
        "category": "Device Control",
        "architectures": [
          "avr"
        ],
        "types": [
          "Arduino",
          "Retired"
        ],
        "resources": {
          "url": "https://downloads.arduino.cc/libraries/github.com/arduino-libraries/Robot_IR_Remote-2.0.0.zip",
          "archive_filename": "Robot_IR_Remote-2.0.0.zip",
          "checksum": "SHA-256:13e0125e8b08fa634d5c339a328b2ecf07b5646e926db02f5db605be7bb9c520",
          "size": 23065,
          "cache_path": "libraries"
        }
      }
    }
  ],
  "status": 1
}

[per1234]

Hi @kittaakos. Sorry for the slow response:

is this the standard procedure to deprecate a library?

Even though it has not been done consistently unfortunately, it is the standard procedure for official libraries.

We haven't set any policy regarding requests to add "Retired" to the types field of 3rd party (i.e., without Arduino in the types field) libraries, but the https://github.com/arduino/library-registry and https://github.com/arduino/libraries-repository-engine systems would support doing that so I think it is good for Arduino IDE to also support it.