Arduino IDE needs to detect out of RAM condition

[penguin359]

The Arduino IDE will happily compile and upload a Sketch even if RAM is over-committed. It should detect usage and stop if not enough free RAM. It appears that at least 10% of RAM should be free for heap and stack to grow in, otherwise the sketch becomes unstable.

[RecreationalVehicle]

the simplest is to use something like this in your code,
int freeRam() {
extern int __heap_start, *__brkval;
int v;
return (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
}
to have the IDE do this would require a profiler that would take a while to evaluate.

[penguin359]

Actually, to just detect the initialized data and bss, which constitutes most of RAM, it's quite simple to do. Take a look at my patch for it at penguin359@db78876

If a user doesn't make heavy use of malloc() which I doubt many Arduino users do, that will handle detecting most problems and it just counts the bytes allocates in .data and .bss. In fact, the avr-size command has a -C option that does just that and provides very nice feedback. The address of heap_start is merely the first byte beyond the end of .bss which is known at the end of a compile before the program is even uploaded to a board.

[penguin359]

I need to forward port my patch to Arduino 1.5.x which I hope to do in the next couple days.

[lestofante]

Loren in the messages before your writed the command to know the ram usage
at compile time. It is unrealistic if you use malloc, but still very usefull
Il giorno 10/apr/2013 11:20, "Federico Fissore" [email protected]
ha scritto:

I'm not aware of ways to detect ram usage at compile time.

That freeRam function is correct, however to use it to warn the user in
the way described above you should:

1. upload the sketch with the freeram function (included in some magic
   way) that prints ram usage at startup
2. run the sketch on the target board and read the ram usage
3. if ram usage is too high, warn the user and stop
4. if the ram usage is normal, upload the sketch without the freeram
   function.

BTW even if we can implement this procedure (that is unlike to happen) we
don't have any way to detect ram overflows that can happen at runtime, for
example if a sketch allocates a big string to handle buffers coming from
external sources (like networks or whatever).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1356#issuecomment-16163534
.

[ffissore]

Yeah, sorry for the email notification: I was writing my comment and I hit the button before noticing that new comments were available.

Apart the procedure, the last part is still valid, it was:

BTW even if we can implement this procedure (that is unlike to happen) we don't have any way to detect ram overflows that can happen at runtime, for example if a sketch allocates a big string to handle buffers coming from external sources (like networks or whatever). It's a malloc, but it happens without the user coding it

[penguin359]

I don't expect runtime RAM usage overflows to be detected. From what I've witnessed, there is very little in the Arduino environment that relies on malloc() and stack usage is only a small percentage of overall RAM usage (unless a user decides to make judicious use of recursive functions or something). I've been using "avr-size -C" in my own personal build scripts for Arduino for about six months now. A re-occuring theme I've discovered is that if pre-allocated data (that's .data + .bss) exceeds much more than 85% of the RAM, than sketches become unstable. What is certain is that if .data + .bss equal or is greater than RAM, you will most certainly have a non-functional sketch and my patch will detect that and stop the upload from happening with a pretty clear warning. When I first ran into this problem, I was over 100% of RAM with all the many debug print statements I had added because I wasn't using the F() macro for Flash Strings. This would have saved me a lot of head-ache, but no, it won't solve all RAM usage problems, but it will solve quite a few.

[lestofante]

True, this is not a silver bullet, but one more check is better than
nothing.
Many times on the forum we have to suggest to encapsulate string into f()
because ram overflow.
At run time we can still Catch it with some trick, but then what should we
do? Because this imho run time ram/stack overflow should be handled as
different issue

Il giorno 10/apr/2013 11:28, "Federico Fissore" [email protected]
ha scritto:

Yeah, sorry for the email notification: I was writing my comment and I
hit the button before noticing that new comments were available.

Apart the procedure, the last part is still valid, it was:

BTW even if we can implement this procedure (that is unlike to happen) we
don't have any way to detect ram overflows that can happen at runtime, for
example if a sketch allocates a big string to handle buffers coming from
external sources (like networks or whatever). It's a malloc, but it happens
without the user coding it

—
Reply to this email directly or view it on GitHub.

[penguin359]

In standard Arduino, a lot of the standard hardware is initialized globally in the program. For example, HardwareSerial and it's buffers. When HardwareSerial.cpp is compiled into an Arduino Sketch, it will occupy space in the .data and .bss regions of RAM including the 64-byte receive and 64-byte transmit buffers. These bytes will show up as part of the RAM usage based on my patch since they are initialized as part of the C runtime, and don't use malloc(). I believe this also applied with other hardware like TWI and SPI is pulled in so looking at the ELF image and adding up the .data and .bss sections should provide a very reasonable estimate of total RAM usage.

[matthijskooijman]

A patch like this would also have saved me half a day of debugging, so I'm all for this! It's not perfect, but it helps (at least as a way to make new users aware of memory limits).

However, I have a few suggestions wrt the implementation:

• I would fail to upload when memory > 100%, but only show a warning when the memory is almost full. If a user knows what he is doing wrt memory usage, he can still continue to work as before, but this helps to make people aware of memory problems when the see stability problems.
• I would lower the "almost full" limit to 75% or so, since I've seen problems with >80% in my bigger sketches. If this just results in a warning, a lower limit helps to make new users aware of possible memory problems earlier.
• I would separate this error and warning from the Sketch too big message. Perhaps "Not enough memory" for the error and "Low available memory, stability problems may occur" for the warning?
• The "Memory usage" message could make people believe that that is the full memory usage. Perhaps "Minimal memory usage" (or "Static memory usage" or "Lower bound on memory usage") might be better?

[penguin359]

I have rebased my patch against ide-1.5.x and sent a new pull request #1377 This is based against the very latest version from GitHub so it should appear pretty clean. I have not yet addressed @matthijskooijman issue's yet, but I will look into doing that next. I've tried to keep complete compatibility with existing configurations that don't report RAM usage or where no maximum is known. It should fall cleanly back to it's old behavior when that happens.

[penguin359]

I have updated my pull request to include all the suggestions by @matthijskooijman. I only fail builds on >100% used RAM and added a configurable warning defaulting to 75% for memory usage. I have also separated out the various message from the code size errors and clarified that it's the minimum memory usage.

[matthijskooijman]

Hmm, I might be thickheaded, but I can't seem to find your updated commits? This issue seems to refer only to the old commit and the pullreq (#1377) also only has the old commits? Or am I looking in the wrong place?

[matthijskooijman]

Ah, never mind that. You added commits instead of updating the existing ones. Github confused me by saying they were added "10 days ago", which seems untrue :-)

I'll have a look at the new commit tomorrow, gotta run now.

For the finall version, you might want to merge the commits again, btw

[matthijskooijman]

Commit looks good! I added a small nitpick as a comment to it.

[penguin359]

I'm not rebasing since the commits are public knowledge now, but they all seem to be showing up in the pull request. Not sure why Github claims 10 days ago for all my recent commits, but my branch is fully merged/based-on the latest from the official Arduino ide-1.5.x branch so it should be good there. The usage of maximum is based on the existing message for Binary Sketch Size. I'd like to change both for consistency, but I hate changing existing code neednessly as it affects all other translations.

[penguin359]

Issue fixed when pull request #1377 was merged.