CONFIG_LWIP_MAX_SOCKETS=10 is too small

[BbIKTOP]

Well, I'm trying to make a HAP device with the Espressif hap, Espressif arduino and Espressif idf.
The problem I experienced is:

1. Default CONFIG_LWIP_MAX_SOCKETS is set to 10
2. No lwip sources provided with arduino core
3. Homekit opens a persistent connection to each client. So, in average family, there are 10-15 devices connected simultaneously. I tested with my own 6 and ran into this problem.
4. I also use WebServer (which built on WiFiServer), and by default it consumes, as far as I remember, 5 sockets. But no matter, even without the WiFiServer it is a problem, because 7 connected HAP clients surely ain't enough for real purposes. So, in my case, with the default configuration and 6 connected hap clients, any access to the WiFiServer leads to lwip_accept fails with errno 23.

It's a bit strange, that Espressif products don't work together. Even more, the main esp32's advantage is the networking, but 10 sockets is just a joke in this sense.

Proposed fixes:

1. Include lwip sources, so any user could configure the required sockets amount and range (not default 1-16, but more reasonable 1-20 or maybe 1-30 as lwip has no such limit)
   It would probably lead to the need to add the whole IDF sources, but I'm not sure

2. Include different precompiled liblwip.a versions, like with 10 (current value), 16 (current menuconfig max value) and 20 or even 30 CONFIG_LWIP_MAX_SOCKETS

Maybe you could propose better, I don't know, sorry.

[BbIKTOP]

Found also, that httpd_server performs an explicit check of this setting, so, it should be recompiled as well. Also, set the limit to 20, everything is working fine so far. It’s definitely needs to be changed

[me-no-dev]

@SuGlider please check memory usage if that value changes to 16 as part of your memory investigations :)

[me-no-dev]

@david-cermak mind giving your two cents? :)

[BbIKTOP]

@SuGlider please check memory usage if that value changes to 16 as part of your memory investigations :)

already did:

After some settlement period, with 20 sockets:

Memory usage: : Total heap: 322999, free heap: 84043, total himem 4191995, free himem 3092611

and with default 10 sockets

Memory usage: : Total heap: 323747, free heap: 86575, total himem 4191947, free himem 3092395

So, the difference is about 2kb per 10 sockets. Sorry, didn't test it with 16, but I suppose the result will be quite close to 20 ;)

[SuGlider]

Ok. So this is the Heap with nothing running on the ESP32.

What about when the application opens and uses all those sockets?

[BbIKTOP]

This is with esp-hap and webserver and 6 hap clients connected

[BbIKTOP]

With 16 sockets after 6 minutes:

Memory usage: : Total heap: 323819, free heap: 86675, total himem 4191971, free himem 3092503

So, I suppose 20 is ok, but would require custom idf with redefined max value in the esp-idf/components/lwip/Kconfig

    config LWIP_MAX_SOCKETS
        int "Max number of open sockets"
        range 1 16
        default 16
        help
            Sockets take up a certain amount of memory, and allowing fewer
            sockets to be open at the same time conserves memory. Specify
            the maximum amount of sockets here. The valid value is from 1
            to 16.

As the least thing to do, it would be good to set it to at least 16. 20 would be better, for sure, but I understand it is too much complicated to implement )))) So, please, make it at least 16.

Btw, 14 minutes uptime, 16 sockets max, the same app:

Memory usage: : Total heap: 323783, free heap: 86611, total himem 4191947, free himem 3092395

[SuGlider]

Is this running on a ESP32WROVER with PSRAM, right?

Although it seems that himem allocation changes almost nothing, would it change heap allocation in a plain ESP32WROOM (no PSRAM)?

Can you test it as well, please?

[BbIKTOP]

I have A LOT of psram allocations for my objects, like peripheral drivers, events, etc, it's really difficult to change everywhere, and I suppose it won't affect the result. As I have a pool allocator for psram and my objects use almost 1mb of it, I'd need to rewrite it to the standard alloc/free. I could try, if you insist, but really, do you think it matters? Arduino and idf both don't use psram, so its just me who does.

[SuGlider]

Ok. Most probably nothing will really change. Some IDF 4.4 drivers allow to use PSRAM for buffers and queues, I but think this is disabled for Arduino.

All right, thus, your suggestion/request is to increase MAX SOCKETS in IDF KConfig for LWIP in Arduino for a next release, correct?

Arduino uses a set of IDF components precompiled into a set of libxxx.a which are built from IDF sdkconfig and KConfigs once a new Arduino version is released.

We could consider setting the number of sockets to 20 in LWIP KConfig and make it default in the precompiled library and its header files.

At this time we have people complaining about high heap allocation in new Arduino Core version 2.0.0. I'll have to check how to make room for all those changes and yet free some Heap to users.

An alternative option for you would be build your own library version with all the resources you need and replace the libxxx.a + header files set in your Arduino environment. Not sure if this is something you would like to do...

[BbIKTOP]

Exactly. At least to 16, as it won't require any changes in the idf cfg. But 20 would be better, of course, it's a network device, isn't it?

[SuGlider]

Ok. I'll include this request on the Heap allocation study I'm currently working.

Anyway, I guess that currently you are using a modified liblwip.a with a number sockets you need for your project, right?

[BbIKTOP]

Yes, among with httpd libs, as httpd checks for CONFIG_LWIP_MAX_SOCKETS -3 and failed to start if this isn't satisfied: https://github.com/espressif/esp-idf/blob/3e370c4296247b349aa3b9a0076c05b9946d47dc/components/esp_http_server/src/httpd_main.c#L410

[SuGlider]

@BbIKTOP

I created a repository with a new Arduino Core 2.0.0 that can be tested.
It is part of modifications related to heap and binary allocation (#5630), but I changed it to support 16 sockets as requested.
https://github.com/espressif/arduino-esp32/tree/mem-optimized

In order to install it as a separated board for testing, please follow the instructions from
https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html
The only difference here would be to clone mem-optimized branch instead.

git clone -b mem-optimized https://github.com/espressif/arduino-esp32 esp32

In case it works fine for a number of users, we will commit it for the next ESP32 Arduino release.
Please test it with your project and let me know.

[BbIKTOP]

Thank you, will try and report back.

Btw, as you already do the "memory optimizations", what would you think about unnecessary memory fragmentations caused by things like this, or this, or this?

Also, do you think, it's a good idea to (re)allocate here or here? Dunno, why don't just reduce it's size and make a single class member buffer? Or move it to the stack? And make an option to keep it in the spi memory, where available? I tried to use the arduino's webserver, but found it eats memory. I suppose, it is due to the fragmentation, because sometimes memory returns, sometimes not... I wanted to fix it, but just switched to the idf instead, which was a way simpler. But maybe it will be a good idea to fix that as well...

(sorry for edits, still sleeping lol)

[SuGlider]

Please try the new Arduino Core branch that I sent.

I think that you will see that it consumes a lot less RAM.

Let me know.

[BbIKTOP]

Well, sorry for delay, was a bit busy working. esps are quite expensive these days, need to earn some money.
have installed it with pio:

platform = https://github.com/platformio/platform-espressif32.git#feature/arduino-upstream
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#mem-optimized

Everything looks like it's working so far. have about +20 free kbs (wow) on heap (comparing to old one). Will leave it for a few days to observe. So far so good, thank you!

[SuGlider]

Arduino Core V2.0.1RC1 has been released today.

It has all changes regarding MAX_SOCKETS, lower binary size and more heap left for application.

[BbIKTOP]

How could it be installed? Which branch?

[me-no-dev]

either use github master or 2.0.1-RC1 through the board managed (development link)

[ViktorRemennik-Rogers]

I don't have this "board manager", so will try the master branch. Thank you!

[looxonline]

Found also, that httpd_server performs an explicit check of this setting, so, it should be recompiled as well. Also, set the limit to 20, everything is working fine so far. It’s definitely needs to be changed

I know that I am resurrecting the dead here but there is scant info about this limit online and this is the only "thread" I have found about pushing beyond the 16 socket max listed in the IDF config so here goes...

We are developing a product that needs to maintain multiple, simultaneous MQTT connections. The current max socket limit of 16 is a bit of a bummer. 20 or even 30 would completely change the game for us. We are using the IDF without the arduino framework on top of it.

I see that you successfully were able to push 20. Did you simply set the config to 20 in the IDF (in spite of the written limit of 16) and compile or were there other changes that you needed to make? We are using the ESP32-S3 with external 8MB PSRAM so we should have enough mem to handle more. I know that lwip itself will allow more and I have searched through the code to look for sanity checks against the stated 16 max and cannot find any so I am wondering if we can just set the config value to more than 16 connections and run with it.

Thanks in advance if you find the time to respond.

[Flole998]

In the upstream esp-idf they increased the maximum limit from 16 to 253, so once commit 48295d31ca90d693356b632750630efd2f1b34eb ends up being used here, this could be bumped from 16 to 20 (or possibly even higher). The bad part is, that this can not easily be made user-configurable, so the consequences of increasing this need to be considered (additional memory is used).

[hitecSmartHome]

In IDF 51.03.05 the max socket is still 16.

[phatpaul]

Looks like there's hope: espressif/esp-idf#14454

espressif/esp-idf@48295d3