MQTT poor performance

[FStefanni]

Board

esp32 dev module

Device Description

esp32 devkit c v4

Hardware Configuration

• I2C sensors
• I2C display SSD1306

Version

v2.0.2

IDE Name

VS Code + arduino-cli

Operating System

Linux Debian Testing

Flash frequency

default

PSRAM enabled

no

Upload speed

default

Description

Hi,

I try to summarize in the following.

The program

• ESP32 configuration:
  • ESP32 attached to WiFi
  • ESP32 attached to a MQTT broker
• ESP32 during its loop:
  • At periodic timeout
    • Show into the display a different information page (i.e. cycle through available pages)
    • Available pages are:
      • Sensors data
      • WiFi information (ssid, rssi, ip, mode)
  • If a mqtt message arrives
    • Publish a message with sensors data

The issue

The reply to the MQTT message can be very slow, till 11 seconds.
Also, the response to a simple network ping is sometimes very slow (more than 1 second).

The workaround

I have found that the calling of the WiFi methods to retrieve the information to display are the origin of the slowdown.
So, if I avoid to call them when I display the page (i.e. I read them only once during the setup() and then I show the "preloaded" values), everything works fine (immediate MQTT reply, ping avg near to 300ms).
The methods are:

• WiFi.RSSI()
• WiFi.localIP()
• WiFi.SSID()
• WiFi.getMode()
• Maybe also the "AP alter ego methods" have the same issue (not tested):
  • WiFi.softAPSSID()
  • WiFi.softAPIP()

I have seen that the Arduino ESP32 is not a plain wrapper to ESP-IDF methods, but it also adds some logic, so I do not know whether the issue is in the Arduino layer or the ESP-IDF methods.

FYI, also the Arduino ESP32 version 2.0.0 has the same issue, and the same workaround works as well.

Regards.

Sketch

Not possible to share.

Debug Message

No message available

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[VojtechBartoska]

Hello @FStefanni, are you able to test this on v2.0.3-RC1 as well? Thanks

[FStefanni]

Hi,

I can, if someone tells me how to do it: it is not available in arduino ide for install.

Regards.

[VojtechBartoska]

Take a look here: https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html

You need to change your link in Preferences in Arduino IDE to development release link.

[FStefanni]

Hi,

thank you for the hints.
I checked, and yes, there is still the same behavior, maybe it happens slightly less frequent, but it still happen to have a delay around 11s/13s.

NOTE: something strange happens during compiling with 2.0.3-RC1: the first compilation fails, but the second succeeds... maybe something to be adjusted into my project? Nevertheless, as the second consecutive compiling goes fine, I used that to do the check.

Regards

[SuGlider]

I suggest to turn on debug output using Core Debug Level: "Verbose" on the Arduino IDE Menu->Tools->Core Debug Level
It may help to see some messages that may help to debug the issue.

[podaen]

Same here #6664

[SuGlider]

Please refer to these two commentaries:

#6664 (comment)
#6664 (comment)

[FStefanni]

Hi,

regarding issue #6664, I believe in my case it is something different, since it was polling the WiFi status which was degrading the WiFi performance (please refer to the workaround section of this issue).

To fix (or understand) my issue, we should look the implementation of the involved methods... and this requires someone who knows that code -- but unfortunately it is not me.

Regards

[podaen]

Without code it's harder to understand what you are doing. Did you tried the event for the wifi status?

For the MQTT I did already managed to send fast messages to the Hive broker (not tested from the broker). However this was in SDK 1.0.6, using pubsubclient-master.h. And the message size was not longer than 40bytes.

[SuGlider]

@FStefanni

The reply to the MQTT message can be very slow, till 11 seconds.

Can you please provide more information. What is the topology? Which server? Is there any simple sketch that we can test and see the issue happening?

Also, the response to a simple network ping is sometimes very slow (more than 1 second).

Ping is something relative to all the elements in the network as well as the current traffic at the time ping is executed.
We need more information about topology and connections.

[FStefanni]

Hi,

• Topology:
  • A local wifi access point, which exposes also a MQTT broker. For example, a Raspberry with Mosquitto
  • The ESP32 connects to the WiFi, and to the Broker.
  • My laptop also connects to the WiFi and to the Broker, to receive/send MQTT data from/to the ESP32

Not tested (I'll check in the next days if I have time), but I believe you can take any MQTT simple sketch,
and in the loop() call a method as the following:

void wifiLoop()
{
    WiFi.RSSI();
    WiFi.localIP();
    WiFi.SSID();
    WiFi.getMode();
}

I have seen performance issues also with the I2C (#6553), -- my sketch in the loop() also read a I2C sensor data and prints the WiFi informations of the above snippet and sensor data to a I2C display -- but I do not believe it matters for this issue (I2C issues exists in 2.0.2 but not in 2.0.0, while this WiFi issue exists in 2.0.0 and 2.0.2).

Regards

[FStefanni]

Hi,

as a small update: I have tested with 2.0.3 and the issue is still there:

• the ping can take more than 1 second (up to 3 seconds)
• MQTT messages can take up to 4 seconds

Regards.

[VojtechBartoska]

Changing the title as it's about MQTT and not Wifi.

[guydvir2]

Few days ago, I posted this question in Arduino stackExchange- still now anwer.
Ithink we share same problem (MQTT slow response).

I'll be happy to help and get help
//////////////////////////////////////////////////////////////////////////////////////

Problem description: Every ~20+ published messages (using terminal), MCU receives that message in a very noticeable delay (mostly ~10 sec, few time it got up to 1 min). Upon receive (printed in Serial monitor), MCU is working as expected.

Hardware: ESP32 DEVKIT MCU, PubSubClient.h for MQTT and WiFi.h for WiFi.

Important remark: Code is same for ESP8266 and ESP32.This behavior happens only on ESP32.

Question: Does ESP32 has a know issue regarding receiving MQTT messages? is there a workaround? or perhaps- bug in code?

What did I do to try isolating the problem:

1. Tried on 3 different ESP32s (2 DEVkit MCU, and 1 with relays on board) - same behavior - it is not an specific MCU hardware problem.

2. Used another PC to publish MQTT messages (via terminal. MAC and Linux, and a 3rd party MQTT App)- no change. It has nothing to do with publishing platform.

3. To rule out it may be a MQTT broker's fault (Local RPI3 connected using LAN cable)- Subscribing to that topic in order to see if the delay repeats . MQTT server receive pubs on time. Broker responds on time (while waiting to see when publish is received in Serial monitor).

4. adding a flashing LED every 200ms in loop(), to verify that MCU is not get stuck in a process. loop() is looping as expected (see in code below).

5. Spitting heap free memory to Serial monitor, in order to see if there is some memory issue, degradation. Nothing suspicious.

6. Code simplification- used code segments from ESP32 wifi and MQTT pubsub's example, while disabling every functionality of original code(with and without). Delay still happens sporadically.

OUTPUT1- sending "0" and "1" almost simultaneously, almost 15 sec delay

13:31:21.516 -> Message arrived [myHome/test] 0 <--- right after this, "1" was sent
13:31:22.510 -> 278616 <---- free heap size (send every 1000 ms. non-blocking)
13:31:23.570 -> 278616
13:31:24.630 -> 278616
13:31:25.657 -> 278616
13:31:26.718 -> 278616
13:31:27.778 -> 278616
13:31:28.805 -> 278616
13:31:29.866 -> 278616
13:31:30.926 -> 278616
13:31:31.954 -> 278616
13:31:33.014 -> 278616
13:31:34.075 -> 278616
13:31:35.102 -> 278616
13:31:35.964 -> Message arrived [myHome/test] 1 <--- received 15 sec later

OUTPUT-2: Code

#include <WiFi.h>
#include <PubSubClient.h>

WiFiClient espClient;
PubSubClient client(espClient);


const char *mqtt_server = "192.168.2.100";


void setup_wifi()
{

        delay(10);
        // We start by connecting to a WiFi network
        Serial.println();
        Serial.print("Connecting to ");
        // Serial.println(ssid);

        WiFi.mode(WIFI_STA);
        WiFi.begin("iot", "GdS");

        while (WiFi.status() != WL_CONNECTED)
        {
                delay(500);
                Serial.print(".");
        }

        Serial.println("");
        Serial.println("WiFi connected");
        Serial.println("IP address: ");
        Serial.println(WiFi.localIP());
}

void callback(char *topic, byte *payload, unsigned int length)
{
        Serial.print("Message arrived [");
        Serial.print(topic);
        Serial.print("] ");
        for (int i = 0; i < length; i++)
        {
                Serial.print((char)payload[i]);
        }
        Serial.println();

        // Switch on the LED if an 1 was received as first character
}

void reconnect()
{
        // Loop until we're reconnected
        while (!client.connected())
        {
                Serial.print("Attempting MQTT connection...");
                // Create a random client ID
                String clientId = "ESP8266Client-";
                clientId += String(random(0xffff), HEX);
                // Attempt to connect
                if (client.connect(clientId.c_str()),"guy","kupelu9e")
                {
                        Serial.println("connected");
                        // Once connected, publish an announcement...
                        client.publish("myHome/log", "hello world");
                        // ... and resubscribe
                        client.subscribe("myHome/test");
                }
                else
                {
                        Serial.print("failed, rc=");
                        Serial.print(client.state());
                        Serial.println(" try again in 5 seconds");
                        // Wait 5 seconds before retrying
                        delay(5000);
                }
        }
}

void setup()
{
        Serial.begin(115200);
        setup_wifi();
        client.setServer(mqtt_server, 1883);
        client.setCallback(callback);
        reconnect();
}
void loop()
{
        loop_buttons();
        if (!client.connected())
        {
                reconnect();
        }
        client.loop();
        static unsigned long lastentry = 0;
        static unsigned long lastentry2 = 0;
        if (millis() - lastentry > 200)
        {
                pinMode(2, OUTPUT);
                digitalWrite(2, !digitalRead(2));
                lastentry = millis();
        }
        delay(50);
        if (millis() - lastentry2 > 1000)
        {
                Serial.println(ESP.getFreeHeap());
                lastentry2 = millis();
        }
}

[TD-er]

2 thoughts:

• Do not call delay(50) in the loop, but do call the MQTTclient's loop() more often.
• Do not try to print the entire message to serial. As soon as the serial port buffer is full, printing may be blocking until the (rather small) buffer is empty and you're done printing.

[podaen]

What TD-er says is absolutly right. You also might use the WiFi event to check your internet connection on your ESP and ad some actions in sertain case's. For the MQTT I would do a time check so it doesn't keep hanging in the loop infinitely if it can not connect.

[guydvir2]

Appreciate your answers!

Putting delay(50) was just a check to see on opposite occurs (looping too fast), and printing to serial, was just for indication when message has entered cb, besides - both were added later, in order to debug.
Do you think it is the cause for such a long delay?

[SuGlider]

To add a bit of suggestions....

• Whenever possible, use FreeRTOS and create Tasks for specific parts or your firmware.
• The software may use ISR (pin function attachment) for Buttons and so on, it may also deal with debouncing.
• The sketch can use a Blocking Serial Print inside a Task and it can be done to do not affect the rest of your firmware.
• The sketch can also set a callback for WiFi events in a separated task and get the Networks events at the time it happens.
• If a Task depends on the execution of some other task, the sketch can use FreeRTOS IPC, or some signaling such as xEventGroupSync() or Queues or any other way -- https://www.freertos.org/Inter-Task-Communication.html
• It is important to think about an Architecture for your Application and then break it into Tasks, States, Communitation Flows and so on.

[guydvir2]

@SuGlider
Thank you. 🙏
I'm guessing it is a good practice tips (which I'll have to read about) but writing code without all those good tips wrtting above, is THIS the reason for such latency ?

[SuGlider]

Not sure... I think it really needs investigation.
Maybe #7183 is also related to this issue. ---- Just saw that you have open it...

[SuGlider]

There is a change I've made to WiFi configuration that will be available for Arduino Core 2.0.5.
This change is about the number of simultaneous sockets connections, that was just 1 and now it will be 4.
It may be related to the MQTT performance issue... MQTT may need an extra check regarding WiFi IDF sdkconfig setup.

[SuGlider]

@guydvir2 - Please check this: #7183 (comment) --- nevermind, you are the issuer!
The PR regarding the number of sockets is #7044

[Parsaabasi]

Hello,

I close this since this report contains the release we no longer support. Please try the new versions and in case the issue persists, feel free to reopen it.

Thanks