Reduce footprint when using libraries

[mikaelpatel]

All library object files are included into the link even if not used (i.e. referenced). This can be avoided by adding the object files to the core library (archive file). I believe that the root cause is that main is in the core library.

[matthijskooijman]

I think that the gc-sections option shoujld make sure that anything that is not used (and doesn't have the used attribute like ISRs) are optimized out again by the linker, so going through an archive file shouldn't gain much. Or did you test this and is the difference bigger than I'm suggesting now?

Having said that, .a linkage for libraries could still be useful to get ISRs only linked in when they are used. This is previously requested in #2800.

[mikaelpatel]

I think I was a bit short in describing the issue. Sorry about that.

I have started to refactor the Cosa core and structure into more libraries. A good example of the issue occurred when refactoring the LCD device drivers for the HD44780. This library contains an implementation of the Cosa LCD interface for the 160X LCD with a number of adapters (parallel, serial, SPI, TWI, etc). https://github.com/mikaelpatel/Cosa/tree/master/libraries/HD44780

When compiling the example and test sketch (https://github.com/mikaelpatel/Cosa/blob/master/examples/LCD/CosaHD44780/CosaHD44780.ino) with the Cosa command line build the size is 5214 bytes. With the Arduino IDE 1.6.3 the size is 10,716 bytes. All of the LCD adapters are linked instead of a single.

The difference between the build is mainly that the command line build puts all the object files into the library (archive) file.

[matthijskooijman]

I just tried and indeed, I also see a 5k vs 10k difference. I haven't looked very closely yet at what causes the difference exactly, but here's a diff between the two builds (left is small cosa, right is big Arduino):

https://gist.github.com/matthijskooijman/da0257575fd30fc2117f

[matthijskooijman]

Looking more closely, it seems the cosa build has no ISR vector 17 and 24 (SPI and TWI), but the Arduino build has them. I looked at where they were defined and was surprised to find them in the Cosa core, not in a library. This would suggest they would only get pulled in when their respective .cpp files are actually referenced, which isn't the case here.

Remember that linking is a two-step process:

• First, a list of symbols to include in the build is constructed. All symbols in all .o files on the commandline are included. Furthermore, any .o files inside .a files are included if any of the symbols inside are needed by a previously selected symbol.
• Second, any symbols that are not used (that is, not referenced directly or indirectly by main or symbols marked as "used", such as ISRs) are removed from the link again.

What happens here, is that all symbols in the HD44780 library are included in the first step, because they are specified directly on the commandline. This pulls in the SPI.o and TWI.o from the core.a file as well.

Now, in the second step, the SPI and TWI ISRs, which are marked as used, keep all of the SPI and TWI stuff in the link, greatly expanding the build size.

An obvious fix (and possibly the only one, apart from splitting the HD44780 library in more separate libraries) for this is indeed to compile the libraries through .a files. If we implement this, in the first step only the actual .cpp files that are referenced from the sketch will be included in the build. So only if the sketch references a driver that references SPI or TWI, the code for that will be included in the first step.

So, implementing #2800 would indeed fix this problem. I'm all for it :-D

[mikaelpatel]

@matthijskooijman

Thanks for your effort. It looks like you took "a dive in at the deep end of the pool".

The simplest solution would actually be to simply add all object files to the core library file (as the Cosa command line build). A more elegant solution would be to allow a library to marked as an archive and generate a library (archive) file for the link.

The solution that you also suggested (splitting the library) is actually what I have done for some other libraries.

The major pain is the Cosa Canvas Font library which generates binaries in the range of 70+ Kbyte as all fonts get included (when I attempted to use the libraries directory).

Thanks again for you time and effort.

[matthijskooijman]

The simplest solution would actually be to simply add all object files to the core library file (as the Cosa command line build). A more elegant solution would be to allow a library to marked as an archive and generate a library (archive) file for the link.

Whatever we do, I think this must be made opt-in, or at least controllable. I'm afraid some libraries will break, for example if they define an ISR by itself in a .cpp or .c file (preventing it from being pulled into the link, leading to hard to debug problems...).

As for the fonts library - not sure how that works? The above analysis really needs an ISR in the picture to trigger the problem, and I don't suppose your referencing the fonts from inside an ISR. So that suggest something else is preventing -gc-sections from removing the unused fonts?

[ffissore]

A fix was proposed in #3697 which was migrated to arduino/arduino-builder#10 and fixed since arduino-builder beta13