memory usage of core/libraries

[pgScorpio]

Basic Infos

Hardware

Hardware: ESP8266
Core Version: 2.3.0

Description

Who is watching the memory usage of the core and/or libraries ?
Did we notice that we are using more and more resources, just to do nothing ?

Just over a year ago, in (version 1.5.8, IDE 1.6.1) we used 174,034 bytes of program storage space for an empty sketch. Now it is 221,995 bytes, so that's a 27% increase...
Also we seem to have lost 24K of program storage space:
now it says: Maximum is 499,696 bytes. (which is 488K - 16 bytes ??)
and it used to be: Maximum is 524,288 bytes. (which is the full 512K)
Adding these up we come to a 33% increase of used flash space.

Since v1.5.8 didn't show ram usage I can't compare these the same way, but at least one of my older sketches doesn't run when compiled with 2.3.0 because of insufficient heap space, so I suspect the same trend in ram usage...

Settings in IDE

Module: Generic ESP8266 Module
Flash Size: 512K (no SPIFS)
CPU Frequency: 80Mhz
Flash Mode: dio
Flash Frequency: 40Mhz
Upload Using: SERIAL

Sketch

void setup()
{
}

void loop()
{
}

Debug Messages

v1.5.8:
Sketch uses 174,034 bytes (33%) of program storage space. Maximum is 524,288 bytes.

v2.3.0:
Sketch uses 221,995 bytes (44%) of program storage space. Maximum is 499,696 bytes.
Global variables use 31,568 bytes (38%) of dynamic memory, leaving 50,352 bytes for local variables. Maximum is 81,920 bytes.

[reaper7]

For this reason I still have core 1.6.4 (with sdk 1.3) for some projects
related to #2253

my example sketch compiled with different core:

"Mem type"	1.6.4 (sdk 1.3.0)	2.2.0 (sdk 1.5.3)	2.3.0 (sdk 1.5.4)
FLASH USAGE AFTER COMPILATION	236 966	238 115	287 921
RAM USAGE AFTER COMPILATION	34 956	33 612	35 436
FREE HEAP(at the setup end)	41424	43312	40536

at this rate of growth, in the near future, only SDK will be able to accommodate in memory :(

[igrr]

I do watch the changes in memory usage, but usually i don't get any feedback, see e.g. the biggest change following 2.3.0: #1752 (comment).
Also I'm only watching DRAM and IRAM usage, flash usage is normally not part of the equation because most modules are at least 1M nowadays. Owners of 512k modules may not agree with me on this, but if I have to choose between keeping flash usage down and fixing a bug with floating point format strings being unsupported, I choose fixing a bug.

Re. 524,288 vs 499,696 bytes — using the whole 524,288 bytes was incorrect for 512k flash, because there are four sectors at the end of flash used for EEPROM, phy init data, and WiFi config. This is why the "total" number was changed to the more accurate 499,696.

@reaper7 we didn't have 1.3 version in this project, perhaps you meant some 1.5.6-something?

[reaper7]

igrr - of course :) 1.6.4 (with sdk 1.3)

[pgScorpio]

Iggr I really think you are doing a great job here. And I agree that fixing bugs is the most important.
But still I can't understand why we are using this amount of resources for an empty sketch.
It probably has to do with the "arduino" way of doing things, because there is apparently a lot of code included in the build that is not used by the sketch, unfortunately also using unneccesary dataspace.

P.S. you are also right about the reserved flash space, but still, 4 sectors would be 16k not 24k.
And the extra 16 bytes is for the block header i suppose? Just trying to understand things...

[middelink]

The thing which worries me more is that suddenly we end up with our own implementations of functions which are already available in rom. E.g.
400018a4 A ets_memset
400018b4 A ets_memcpy
400018d4 A ets_memcmp
40002a88 A ets_strcpy
40002a98 A ets_strncpy
40002ab8 A ets_strncmp
40002ac8 A ets_strlen
40002ad8 a ets_strstr
40002ae8 A ets_bzero

Is there a fundamental reason we can't use them any more? Granted, they prob do not account for that 8KiB flash increase ^^

Secondly, would it be possible to add a define so I can opt out on the floating point stuff? And therefore fall back on the rom ets_printf and friends?

[CircuitSerialKiller]

I'm concerned about this too. From 2.3.0 to the latest build on GiHub as of today, I noticed a bump up of over ~2KB of heap shown by system_get_free_heap_size()

[Sod-Almighty]

I'm also very curious about the RAM usage. I'm using an ESP2866 microcontroller, and an empty sketch uses almost 32k of RAM:

Sketch uses 221995 bytes (21%) of program storage space. Maximum is 1044464 bytes.
Global variables use 31568 bytes (38%) of dynamic memory, leaving 50352 bytes for local variables. Maximum is 81920 bytes.

Why does an empty sketch use almost 40% of my RAM? That's insane. Isn't there some way to customise this? Maybe remove some of the boilerplate that isn't being used?

[devyte]

You forget that there is a full wifi and TCP/IP stack implementation behind the scenes, which requires certain resources to be ready-to-go. It would be interesting to strip out the whole network thing to see what the RAM usage looks like with what's left, but I have no idea how to do that. Also, keep in mind that the ESP firmware is built around the espressif SDK, which may also requires some resources, and there is no access to that code (it's proprietary).

…

On Jun 6, 2017 9:22 PM, "Sod-Almighty" ***@***.***> wrote: I'm also very curious about the RAM usage. I'm using an ESP2866 microcontroller, and an empty sketch uses almost 32k of RAM: Sketch uses 221995 bytes (21%) of program storage space. Maximum is 1044464 bytes. Global variables use 31568 bytes (38%) of dynamic memory, leaving 50352 bytes for local variables. Maximum is 81920 bytes. Why does an *empty sketch* use almost 40% of my RAM? That's insane. Isn't there some way to customise this? Maybe remove some of the boilerplate that isn't being used? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2254 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQC6Biu22lZBFkFI5TiY-h4ZM_u3D_Qnks5sBftcgaJpZM4JIqgK> .

[Sod-Almighty]

But if I don't #include the wifi headers, surely the wifi code shouldn't be added to the executable? For a device so RAM-limited, I'd imagine the compiler would at least be smart enough to omit libraries that weren't being used.

And furthermore, we're not even talking program size here - just RAM usage. Until I call Wifi.setup() or whatever, it shouldn't use much RAM at all. If it kept globals to a minimum - pointers instead of stack objects, for example - then RAM should only be a factor when I'm actually using the library.

[devyte]

The "arduino" core libs are just a thin wrapper around the proprietary espressif sdk calls. The actual init and a whole bunch of low level code, including a bunch of global structures, are inside the sdk, which are pre-built. Also, there's lwip, which has a bunch of global structs as well. And also, there is no OS here, which means you can't assume existence of certain services for your app. The compiler/linker do the usual optimizations, but there isn't much that they can do about some things in a build env like this one. Like I said, it would be a cool academic experiment to make a firmware from scratch, or at least without any networking, and see what you get for heap.

…

On Jun 7, 2017 11:05 AM, "Sod-Almighty" ***@***.***> wrote: But if I don't #include the wifi headers, surely the wifi code shouldn't be added to the executable? For a device so RAM-limited, I'd imagine the compiler would at least be smart enough to omit libraries that weren't being used. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2254 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQC6BnH2CKI9ADyHQWnDEAMMPAFpx4Obks5sBrwYgaJpZM4JIqgK> .

[Sod-Almighty]

How difficult would it be to avoid using the sdk at all? I mean, is there a simple way I can write a small piece of C code, upload it to the ESP8266, and have it run directly without all the sdk-loader stuff? As in, the first instruction at address zero in memory - the first instruction the CPU executes - is the first instruction I wrote.

Obviously I wouldn't get any wifi support or owt, but as an intellectual exercise it might be interesting. I'm just wondering how difficult it would be to achieve this. I have some experience in C (and several other languages), but I suspect I'd have to use assembler for this?

How would I go about it, do you think?

[davisonja]

That's not overly hard. Check out the eboot source in the arduino git. Its what resides at flash 0. Keep in mind, tho, that the flash is external to the chip, and is called by the on chip start up code - even not using the sdk the esp 'rom' still runs first.

…

On 8/06/2017 11:02 am, "Sod-Almighty" ***@***.***> wrote: How difficult would it be to avoid using the sdk at all? I mean, is there a simple way I can write a small piece of C code, upload it to the ESP8266, and have it run directly without all the sdk-loader stuff? As in, the first instruction at address zero in memory - the first instruction the CPU executes - is the first instruction I wrote. Obviously I wouldn't get any wifi support or owt, but as an intellectual exercise it might be interesting. I'm just wondering how difficult it would be to achieve this. I have some experience in C (and several other languages), but I suspect I'd have to use assembler for this? How would I go about it, do you think? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2254 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAN_A5sxlqUwq9d3sXky-VCOCS0UNf7Jks5sBywagaJpZM4JIqgK> .

[Sod-Almighty]

Thanks, I'll take a look.

How would I actually get the code into the chip, though? The Arduino IDE compiles it using the sdk and stuff...

[davisonja]

You'll end up building eboot 'by hand' so that's the build step. However, the ide build process is controlled/defined in platform.txt with boards.txt - ultimately all commands you can run 'by hand' (upload of binary is a distinct step). I'm not close enough to a system to be more specific, but that's where the specifics are 😊 J,

…

On 8/06/2017 11:15 am, "Sod-Almighty" ***@***.***> wrote: Thanks, I'll take a look. How would I actually get the code into the chip, though? The Arduino IDE compiles it using the sdk and stuff... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2254 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAN_AwLNsyGNBd3YBiY_2e3eulFFHEeEks5sBy8VgaJpZM4JIqgK> .

[devyte]

With 4MB modules at under usd$3 today, sketch size is not an issue. I believe even Espressif doesn't formally support 512KB devices anymore with their latest AT firmware.
About free heap, I think an optimization effort is worth doing. I've seen many fixed-sized buffers around in objects, and all strings should be checked for F/PROGMEM. Any extra mem we get is worth it.
But first, repo cleanup, and a new release.
Closing per #3655 .

[MohdShanawaz]

I am using arduino 1.8.5 and I am getting this error message. How to resolve it.

Sketch uses 5018 bytes (16%) of program storage space. Maximum is 30720 bytes.
Global variables use 574 bytes (28%) of dynamic memory, leaving 1474 bytes for local variables. Maximum is 2048 bytes.

Program is File > Examples > Modbus Master > RS-485_Halfduplex

Please reply urgently.

[Sod-Almighty]

That is not an error message. That is normal operation.

[MohdShanawaz]

Thanks

[aldolo69]

is there a way to use the nowifisdk with arduino core to save ram? i'm porting a zxspectrum emulator and and the available ram is not enough.