Added support for advanced sketch "size" command

[cmaglie]

This PR adds a command-line recipe that the builder can use to obtain advanced statistics on sketch memory usage. This is especially useful for non-trivial memory layouts where the classic reg-exp based approach doesn't work well.

The proposed solution consists of a new recipe to the platform.txt:

recipe.advanced_size.pattern=...command line to run...

that, when present, is used instead of the previous recipe.size.pattern=.... The old size.pattern may be left for backward compatibility.

The command run in the advanced_size recipe should output the memory usage statistics in JSON format with the following syntax:

{
    "output" : "Your sketch use 2200 bytes of program memory out of 8192 (22%)\nThe static RAM used is 200 bytes (of 2048 max)\n",
    "severity" : "info",
    "sections" : [
        { "name" : "text", "size" : 2200, "maxsize" : 8192 },
        { "name" : "data", "size" : 200, "maxsize" : 2048 }
    ]
}

The meaning of the fields is the following:

• output is a preformatted text that is displayed as-is in console.
• severity indicates the warning level of the output messages, it can be info, warning or error. Warnings and errors are displayed in red (or in a different color than normal output). Errors will make the build/upload fail.
• sections are a machine-readable representation of the memory sections and their usage level. It's an array of objects, each one representing a memory section. This array is used by the arduino-cli to report memory usage when the --format json output flag is set.

Example:

the following change in the platforms.txt:

diff --git a/platform.txt b/platform.txt
index 7c6c46c..5e5e8bb 100644
--- a/platform.txt
+++ b/platform.txt
@@ -116,6 +116,10 @@ recipe.output.tmp_file={build.project_name}.{build.preferred_out_format}
 recipe.output.save_file={build.project_name}.{build.variant}.{build.preferred_out_format}
 
 ## Compute size
+recipe.advanced_size.pattern=echo '{"output" : "Your sketch use 2200 bytes of program memory out of 8192 (22%)\nThe static RAM used is 200 bytes (of 2048 max)", "severity" : "info", "sections" : [ { "name" : "text", "size" : 2200, "maxsize" : 8192 }, { "name" : "data", "size" : 200, "maxsize" : 2048 } ] }'
+
+# the following are for backward compatibility for IDEs that doesn't support "advanced" size output
 recipe.size.pattern="{compiler.path}{compiler.size.cmd}" -A "{build.path}/{build.project_name}.elf"
 recipe.size.regex=^(?:\.text|\.data|)\s+([0-9]+).*
 recipe.size.regex.data=^(?:\.data|\.bss)\s+([0-9]+).*

will produce the following output in the build:

$ ./arduino-cli compile -b arduino-dev:samd:mkr1000 ~/Arduino/Blink 
Your sketch use 2200 bytes of program memory out of 8192 (22%)
The static RAM used is 200 bytes (of 2048 max)

$ ./arduino-cli compile -b arduino-dev:samd:mkr1000 ~/Arduino/Blink --format json
{
  "compiler_out": "Your sketch use 2200 bytes of program memory out of 8192 (22%)\nThe static RAM used is 200 bytes (of 2048 max)\n",
  "compiler_err": "",
  "builder_result": {
    "build_path": "/tmp/arduino-sketch-002050EAA7EFB9A4FC451CDFBC0FA2D3",
    "executable_sections_size": [
      {
        "name": "text",
        "size": 2200,
        "maxSize": 8192
      },
      {
        "name": "data",
        "size": 200,
        "maxSize": 2048
      }
    ]
  },
  "success": true
}
$

When the severity is set to error the build/upload is interrupted and an exception is returned to the calling process. In this case an extra exception message must be provided through the error field, for example:

{
    "output" : "Your sketch use 12200 bytes of program memory out of 8192 (122%)",
    "severity" : "error",
    "error": "sketch is too big!",
    "sections" : [
        { "name" : "text", "size" : 12200, "maxsize" : 8192 }
    ]
}

This will produce the output:

$ ./arduino-cli compile -b arduino-dev:samd:mkr1000 ~/Arduino/Blink
Your sketch use 12200 bytes of program memory out of 8192 (122%)
The static RAM used is 200 bytes (of 2048 max)

Error during build: sketch is too big!
$

Fix #480
/cc @PaulStoffregen

[PaulStoffregen]

Please consider a recipe which just copies the advanced tool's stdout without any parsing. No JSON - just copy all the stdout text directly to the user interface.

The usage case is more complex memory models, where a region of memory may be used in multiple ways. For example, if a memory region is 512K and can be segments into 4 different usages (eg, Cortex-M7 ITCM code, DTCM data, DMA-optimized data, and stack), this JSON approach would require using 4 lines which each show the same 512K region, giving users the mistaken impression that 4X as much memory exists.

[cmaglie]

Please consider a recipe which just copies the advanced tool's stdout without any parsing. No JSON - just copy all the stdout text directly to the user interface.

The output field of the JSON object is already copied verbatim:

{
         THE FOLLOWING STRING IS DISPLAYED AS-IS TO THE USER vvv
    "output" : "Your sketch use 2200 bytes of program memory out of 8192 (22%)\nThe static RAM used is 200 bytes (of 2048 max)\n",

    "severity" : "info",
    "sections" : [
        { "name" : "text", "size" : 2200, "maxsize" : 8192 },
        { "name" : "data", "size" : 200, "maxsize" : 2048 }
    ]
}

you can also omit the sections part entirely, it's not even used to check if the sketch size exceeds the available memory.
BTW having the size report in a machine-readable format helps to create CI automations that detects and reports changes in memory usage, that's why I'd like to keep it.

For eample this one arduino/ArduinoCore-mbed#256 (comment) leverage the --format json flag of the CLI to get the current size of the sketch without parsing human-readable text (that may change over time).

Anyway, if you really need it I can change this PR to handle your use case, something like this:

• if the output of the sizer command is a valid JSON: it works as specified in this PR
• otherwise the output is copied as-is to the user, and if the sizer command exits with:
  • exit code 0: (success) the output is displayed normally
  • exit code 1: (warning) the output is displayed in red as a warning, but the compile/upload will proceed anyway
  • exit code 2: (error) the output is displayed in red and the compile/upload stops

[fstasi]

closes #1514

[PaulStoffregen]

Sorry, forgot about this until just recently... now that I'm finally working on updating Teensy software for arduino-cli.

JSON format is fine. I can update my sizer utility to output JSON, and my Java patches to read JSON.

Perhaps the JSON format should offer a way to send copies of the verbatim text in non-English languages?

[cmaglie]

Perhaps the JSON format should offer a way to send copies of the verbatim text in non-English languages?

It's the tool that should create the "output" text in the currently selected language.
One thing that cli may do is to set an environment var LANG with the currently user-selected language in case the user overridden the default value via preferences, but nothing more, I would not add i18n into the json.

[PaulStoffregen]

LANG environment variable is fine.

[PaulStoffregen]

I'm going to update my sizer code and start work on a Java patch to receive JSON. Going with this format. Hope it doesn't end up changing (much).

[cmaglie]

I'm going to merge this PR as-is, only need to add docs...

[per1234]

This is a very cool enhancement. Thanks Cristian!