New library handling scheme - invitation to criticisms

[lmihalkovic]

There are a number of known issues/annoyances with how libraries are handled in the IDE. I am not going to list them all, however here is a shortlist of a few other issues describing some:

• lib manager overrides core libraries included with 3rd party boards #4064
• No way to remove library from the library manager dialog. #2878
• Manager libraries install a libarry into two ways ! #4179 (basically the idea that some libs are present in 2 locations)
• Support multiple version of a given library #4065 (I still can't agree that there is not a simple way to address all the hard to use objections

I am currently working on #4206: I treat a single XXXXX.INZ just like the equivalent XXXXX folder is currently used. During compilation, my builder (I am sure the same behavior can be added to arduino-builder) takes care of expanding the content into the target build folder per the normal build process. When working with a INZ file, the IDE also normally takes care of the 'save-as' step if the user tries to modify any of the source files.
The benefits of this approach are in my mind that

• sketches can now be exchanged directly by mail without having to expand their contents (double-click on a INZ and it opens transparently)
• sketches could be 'versioned' by transparently supporting XXXXX-nn-nn-nn.INZ in the file name (especially in Create)
• desktop users could directly upload any sketch as a single INZ into Create without have to switch IDE (same for downloading any INZ directly from the new community sharing platform).

As I was making good progress, it suddenly downed on me that this approach could be extended to libraries: the idea is to keep libraries inside a single container for as long as possible (i am voluntarily vague to avoid steering feedback in any particular direction). Does anyone have any thoughts about possible limitations of this scheme?! (writing the code is not a limitation, it is just a step).
My experience as a USER is still limited, so I would really appreciate any criticisms about the usability, workflow or use cases...

@mbanzi this might interest you as a simple way to reinforce the ties inside the Arduino LLC product line compared to external tools

[PaulStoffregen]

During compilation, my builder (I am sure the same behavior can be added to arduino-builder) takes care of expanding....

Have you published your builder's code?

[shiftleftplusone]

I personally would appreciate very much to be able to choose the installation folder for additional libs, and in case there are duplicate ones, erase the duplicates (or all) by provided options.

Currently the custom lib installation folder used by default is a big annoying mess as I need this for different purposes.

Also there should be an uninstall option in the IDE to uninstall either lib.

[matthijskooijman]

Having a single-file library format might be useful. There is currently the .zip format, which is used for distribution, but which is unpacked when installed. I'm not entirely sure if there are real advantages of keeping it packed, which is essentially what you're proposing, right? Can you say why you think adding this would be good?

One downside of keeping things packed is that the files need to be unpacked for compilation (gcc doesn't like zip files), but the unpacking can probably be cached in the build dir. Another downside is that modifying a library becomes more tricky, but that can also be an upside - it becomes easier to verify that a library was unmodified (though to fully do that, this needs some out-of-band checksum mechanism like used for third-party cores).

That's just some thoughts I had while thinking about this, they're by no means comprehensive.

@VogonJ, your suggestions seem unrelated to the proposal in this issue, so please take the time to find a more appropriate issue to add your comments to (or, if there is none, create a new one).

[shiftleftplusone]

I disagree.
The TO says:
"There are a number of known issues/annoyances with how libraries are handled in the IDE."
And that was just what I have been relating to.

[lmihalkovic]

@matthijskooijman @VogonJ for having started this one, and at the risk of breaking the rules, i am indeed inviting anyone to vent about libraries at large and criticize the scheme I very vaguely described. this is in the hope that the final solution will address as many use cases as possible. Thank you both for your help.

@matthijskooijman his comment is valuable to me as it basically says that version numbers are an integral part of a the identity of a library.

I dont view unpacking as a major tradeoff, because it is already done (in lib mgr) which means i could even consider blindly cut/paste the code... but you are right that i would have to consider some caching (which is also already done for the sketch source code, so again some possible lazzzzy cut/clean/paste).

Right again that this is another verification opportunity with oob SHA1s

However i look at it, I find myself going back to how homebrew deals with this ( basically what i described in #4065 ).

[lmihalkovic]

Thank you for the feedback