Serial communication breaking at around 300 characters, regression on 2.0.10

[osomdev]

Board

Adafruit QT Py ESP32-C3

Device Description

Adafruit QT Py ESP32-C3 - https://www.adafruit.com/product/5405

Hardware Configuration

Nothing extra, just board.

Version

v2.0.11

IDE Name

Arduino IDE

Operating System

ubuntu 22.04

Flash frequency

80Mhz

PSRAM enabled

yes

Upload speed

921600

Description

When communicating with device over Serial with more than 256 bytes the data is truncated (most commonly round 300-350 bytes but it is not static number) despite setting buffers to 1024 bytes (setRxBufferSize, setTxBufferSize).

It was working fine in the past but stopped working after upgrade of Arduino IDE.

I've been able to narrow this down to ESP 2.0.10 version:

• using 2.0.5 - works fine
• using 2.0.6 - works fine
• using 2.0.9 - works fine
• using 2.0.10 - communication truncated
• using 2.0.11 - communication truncated

Steps to reproduce:

1. Create sketch which reads data from serial, setup it with bigger buffers
2. Run sketch and feed it with 512 bytes of data
3. Data is truncated and not received by sketch while 512 bytes should comfortably fit into 1024 bytes buffer.

Using serial monitor is sufficient to replicate this behavior.

Test string used (nothing special - just plain text):

YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY

There are some similar reports (https://github.com/espressif/arduino-esp32/issues?q=is%3Aissue+is%3Aopen+setRxBufferSize+) but for versions older than 2.0.10 and other hardware.
I suspect there was new regression introduced in 2.0.10.

Video showing the unexpected behavior (pasted 512 YYY... and it was not received fully):
https://github.com/espressif/arduino-esp32/assets/5213149/e5b66c90-dff8-4acd-ac21-9830809e25c0

Sketch

void setup() {
  // tried to use 0 to force-reset buffers - did not help, leaving for reference
  // Serial.setRxBufferSize(0);
  Serial.setRxBufferSize(1024);
  // Serial.setTxBufferSize(0);
  Serial.setTxBufferSize(1024);
  //Serial.setTimeout(1);
  Serial.begin(115200);
}

#define MAX_INPUT_BUFFER_SIZE 4096

uint32_t inputBufferPos = 0;
uint8_t inputBuffer[MAX_INPUT_BUFFER_SIZE + 1];

void loop() {
  uint32_t readInLoopCount = 0;
  while (Serial.available()) {
    uint8_t readByte = Serial.read();
    inputBuffer[inputBufferPos] = readByte;
    inputBufferPos += 1;
    readInLoopCount += 1;

    if (readByte == '\n') {
      inputBuffer[inputBufferPos] = 0;
      // Serial.print((char*)inputBuffer);
      Serial.printf("GOT: %d\n", strlen((char*)inputBuffer));
      inputBufferPos = 0;
    }
    if (inputBufferPos >= MAX_INPUT_BUFFER_SIZE) {
      Serial.println("ERR INPUT TOO LONG");
      inputBufferPos = 0;
    }
  }

  if (readInLoopCount > 0) {
    Serial.printf("GOT IN LOOP: %d\n", readInLoopCount);
  }
}

Debug Message

Nothing useful, just output from my test run that I've recorded in video:

GOT: 73
GOT IN LOOP: 73
GOT: 5
GOT IN LOOP: 5
GOT: 346
GOT IN LOOP: 346
GOT: 5
GOT IN LOOP: 5
GOT: 335
GOT IN LOOP: 335
GOT IN LOOP: 353
GOT IN LOOP: 351
GOT: 709
GOT IN LOOP: 5
GOT IN LOOP: 342
GOT IN LOOP: 326
GOT: 996
GOT IN LOOP: 328
GOT: 5
GOT IN LOOP: 5

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[SuGlider]

@osomdev - Thanks for reporting.
Issue confimed. It happens with ESP32-C3 USB CDC + 2.0.10+
No issue with 2.0.9.

[SuGlider]

@me-no-dev @VojtechBartoska - This is a regression of USB Hardware CDC in v2.0.10+
No change visible in the USBCDC.cpp of in Lib Builder arduino_tinyusb component that justifies such regression.

It can be related or to some change done in IDF 4.4.5.

[me-no-dev]

@SuGlider C3 does not use tinyusb. This is about the internal HW CDC+JTAG

[me-no-dev]

Calling Serial.setRxBufferSize(1024); AFTER Serial.begin(115200); helps. Buffer is still 256 bytes long after begin

[me-no-dev]

To summarize:

• Expecting large amounts of data and reading byte-by-byte is never a good option. This reads from the buffers significantly slower than eg: readStringUntil('\n') or just reading as many as there are available(). Reading slow causes the buffers to overfill and not accept more data.
• Recent changes in the code that manages the RX/TX buffers caused any previous buffer to be freed and default 256 bytes one to be allocated. This should probably be changed, but the solution above will work until then (and in the future).

[SuGlider]

@me-no-dev / @osomdev - a PR to fix it and improve the way how TX/RX buffers are set has been released. PR #8559

HWCDC::setTxBufferSize() and/or HWCDC::setRxBufferSize() can be called at any time, after or before HWCDC::begin(), even more than one time or in any order.