Compilation using source override fails if sketch file content longer than maximum gRPC message length

[per1234]

Describe the problem

The cc.arduino.cli.commands.v1.ArduinoCoreService.Compile request message of Arduino CLI's gRPC interface has a sourceOverride field:

This map (source file -> new content) let the builder use the provided content instead of reading the corresponding file on disk. This is useful for IDE that have unsaved changes in memory. The path must be relative to the sketch directory. Only files from the sketch are allowed.

The design of the field is such that the entire content of the sketch file must be sent in a single message.

🐛 The compilation will fail spuriously if the field is used with a sketch file with content larger than the default maximum gRPC message length.

To reproduce

It will probably be most convenient to use Arduino IDE to reproduce the fault:

1. Click the following link to download the demo sketch:
   BigFileSketch.zip
2. Extract the downloaded ZIP file.
3. Open the sketch from the extracted folder in Arduino IDE.
4. Select File > Preferences... (or Arduino IDE > Settings... for macOS users) from the Arduino IDE menus.
   The "Preferences" dialog will open.
5. Uncheck the box next to "Auto save" in the "Preferences" dialog.
6. Click the "OK" button.
   The "Preferences" dialog will close.
7. Add a blank line to the top of the sketch.
   ⓘ This is an arbitrary innocuous change in order to put the editor into the "dirty" state, which will cause Arduino IDE to send the contents of the sketch through the sourceOverride field of the cc.arduino.cli.commands.v1.ArduinoCoreService.Compile request message.
8. Select Sketch > Verify/Compile from the Arduino IDE menus.

🐛 Compilation fails:

grpc: received message larger than max (4226107 vs. 4194304)

Compilation error: grpc: received message larger than max (4226107 vs. 4194304)

Expected behavior

The API is designed in a way that allows the source override feature to be used even with large sketch files.

Arduino CLI version

1.0.4

Operating system

Windows

Operating system version

Windows 11

Additional context

Originally reported at https://forum.arduino.cc/t/compilation-error-grpc-received-message-larger-than-max-4322345-vs-4194304/1304422

---

A silly contrived sketch was used in the demo for the sake of simplicity. For an example of a real world sketch that also produces the fault, see:

https://forum.arduino.cc/t/compilation-error-grpc-received-message-larger-than-max-4322345-vs-4194304/1304422/5

Workaround

Save the sketch before compiling.

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the nightly build
• My report contains all necessary details

[cmaglie]

Since we had only 1 issue report about this problem IMHO increasing the limit from 4MB to 16MB, should solve the issue for 99.99999% of our users. I'm preparing a PR for this.

On the other hand, since we could not increase this limit indefinitely, the final fix could be implementing a dialog box in the Arduino IDE asking the user to save the file before compiling only in the remote case that the unsaved tabs total size is >16MB.