Skip to content

Response Code -3 in HTTPClient.POST #5674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
EdoMon93 opened this issue Jan 26, 2019 · 4 comments
Closed

Response Code -3 in HTTPClient.POST #5674

EdoMon93 opened this issue Jan 26, 2019 · 4 comments

Comments

@EdoMon93
Copy link

EdoMon93 commented Jan 26, 2019 •
edited by devyte
Loading

I'm having trouble sending post using httpclient.
I think it might be related to free heap and fragmentation but have not been able to find any viable solutions or workarounds.
I'm using v2.5.0-beta2 and ArduinoJSON v5.13.4 (this might not be relevant).

So in the code I have arrays with a bunch of data which im trying to send to a backend. Due to the fact that the arrays are of considerable size, im splitting up posts to aprox less then 2kB (It should not be necessary but im running out of heap). I do this by calling this function several times, the code surrounding it retries several times in case of error:

int8_t sendSens(uint16_t arrayStart, uint16_t arrayEnd) {

	if (WiFi.status() == WL_CONNECTED) { //Check WiFi connection status
		int httpCode = 0;
		DEBUG_PRINT(F("Free Heap before jsonBuffer: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		uint16_t maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		const size_t bufferSize = 2 * JSON_ARRAY_SIZE(arrayEnd - arrayStart + 1)
				+ JSON_OBJECT_SIZE(4);
		DEBUG_PRINT(F("bufferSize: "));
		DEBUG_PRINTLN(bufferSize);
		if (maxFreeBlock < bufferSize) {
			return MAX_BLOCK_SIZE_TOO_SMALL; //TOO FRAGMENTED
		}
		DynamicJsonBuffer jsonBuffer(bufferSize);
		JsonObject& root = jsonBuffer.createObject();
		root["cmd"] = "insert";
		root["device"] = DEVICE;
		JsonArray& sens = root.createNestedArray("sens");
		for (int i = arrayStart; i < arrayEnd; i++) {
			sens.add(dataSens[i]);
		}
		JsonArray& sensTime = root.createNestedArray("sensTime");
		for (int i = arrayStart; i < arrayEnd; i++) {
			sensTime.add(dataSensTime[i]);
		}
#ifdef DEBUG_MODE
		root.printTo(Serial);
#endif
		size_t len = root.measureLength() + 1;
		DEBUG_PRINT(F("\njson Length: "));
		DEBUG_PRINTLN(len);
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINT(F("Max Block: "));
		DEBUG_PRINTLN(maxFreeBlock);
		if (maxFreeBlock < len) {
			return MAX_BLOCK_SIZE_TOO_SMALL; //TOO FRAGMENTED
		}
		DEBUG_PRINT(F("Free Heap before postRequest malloc: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		char *postRequest = (char *) malloc(len);
		root.printTo(postRequest, len);
		DEBUG_PRINT(F("postRequest: "));
		DEBUG_PRINTLN(postRequest);
		DEBUG_PRINT(F("Free Heap before jsonBuffer.clear: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		jsonBuffer.clear();
		ESP.wdtFeed();
		DEBUG_PRINT(F("Free Heap before http.begin: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		WiFiClient client;
		HTTPClient http;
		http.setTimeout(2000);
		http.begin(F("http://mywebpage.com/dataReciever.php")); //Specify request destination
		http.addHeader(F("Content-Type"), F("application/json")); //Specify content-type header
		DEBUG_PRINT(F("Free Heap before http.POST: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		httpCode = http.POST(postRequest); //Send the request
		DEBUG_PRINT(F("Free Heap after http.POST: "));
		DEBUG_PRINT(ESP.getFreeHeap());
		DEBUG_PRINT(F(" Fragmentation: "));
		DEBUG_PRINT(ESP.getHeapFragmentation());
		DEBUG_PRINT(F("% MaxBlock: "));
		maxFreeBlock = ESP.getMaxFreeBlockSize();
		DEBUG_PRINTLN(maxFreeBlock);
		DEBUG_PRINTLN(F("JSON Data Sent"));
		DEBUG_PRINT(F("Response: "));
		DEBUG_PRINTLN(httpCode);
		ESP.wdtFeed();
		if (httpCode == 200) {
			DEBUG_PRINT(F("Response Payload: "));
			DEBUG_PRINTLN(http.getString());
			DEBUG_PRINT(F("Free Heap before HTTP.END: "));
			DEBUG_PRINT(ESP.getFreeHeap());
			DEBUG_PRINT(F(" Fragmentation: "));
			DEBUG_PRINT(ESP.getHeapFragmentation());
			DEBUG_PRINT(F("% MaxBlock: "));
			maxFreeBlock = ESP.getMaxFreeBlockSize();
			DEBUG_PRINTLN(maxFreeBlock);
			http.end();  //Close connection
			DEBUG_PRINT(F("Free Heap before FREE POST REQ: "));
			DEBUG_PRINT(ESP.getFreeHeap());
			DEBUG_PRINT(F(" Fragmentation: "));
			DEBUG_PRINT(ESP.getHeapFragmentation());
			DEBUG_PRINT(F("% MaxBlock: "));
			maxFreeBlock = ESP.getMaxFreeBlockSize();
			DEBUG_PRINTLN(maxFreeBlock);
			free(postRequest);
			DEBUG_PRINT(F("Free Heap before RETURN: "));
			DEBUG_PRINT(ESP.getFreeHeap());
			DEBUG_PRINT(F(" Fragmentation: "));
			DEBUG_PRINT(ESP.getHeapFragmentation());
			DEBUG_PRINT(F("% MaxBlock: "));
			maxFreeBlock = ESP.getMaxFreeBlockSize();
			DEBUG_PRINTLN(maxFreeBlock);
			return JSON_POST_OK;
		}
		DEBUG_PRINT(F("Error to String: "));
		DEBUG_PRINTLN(http.errorToString(httpCode));
		DEBUG_PRINT(F("FAILED: Response Payload: "));
		DEBUG_PRINTLN(http.getString());
		http.end();  //Close connection
		free(postRequest);
		return SERVER_BAD_RESPONSE;
	} else {
		DEBUG_PRINTLN(F("WiFi not connected during temp report"));
		return NOT_CONNECTED_TO_WIFI;
	}
}

Sorry for the long post, most of it is debug output...

Next is the output:

Send sensors from 0 to 99
Free Heap before jsonBuffer: 5776 Fragmentation: 8% MaxBlock: 5344
bufferSize: 2512
{"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
json Length: 1750
Max Block: 2816
Free Heap before postRequest malloc: 3248 Fragmentation: 13% MaxBlock: 2816
postRequest: {"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
Free Heap before jsonBuffer.clear: 1488 Fragmentation: 26% MaxBlock: 1056
Free Heap before http.begin: 4016 Fragmentation: 32% MaxBlock: 2528
Free Heap before http.POST: 3800 Fragmentation: 28% MaxBlock: 2528
Free Heap after http.POST: 3368 Fragmentation: 38% MaxBlock: 1768
JSON Data Sent
Response: -3
Error to String: send payload failed
FAILED: Response Payload: 

Send sensors from 0 to 99
Free Heap before jsonBuffer: 5448 Fragmentation: 32% MaxBlock: 3272
bufferSize: 2512
{"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
json Length: 1750
Max Block: 1768
Free Heap before postRequest malloc: 2920 Fragmentation: 34% MaxBlock: 1768
postRequest: {"cmd":"insert","device":2,"sens":[59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59135,59391,59391,59135,59135,59391,59391,59135,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59391,59135,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548367506,1548367755,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367769,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367770,1548367785,1548367785,1548367786,1548367786,1548367786,1548367786,1548368058,1548368058,1548368058,1548368058,1548368059,1548368060,1548368071,1548368071,1548368100,1548368100,1548368101,1548368101,1548368154,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368158,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159]}
Free Heap before jsonBuffer.clear: 1160 Fragmentation: 31% MaxBlock: 744
Free Heap before http.begin: 3688 Fragmentation: 11% MaxBlock: 3272
Free Heap before http.POST: 3472 Fragmentation: 6% MaxBlock: 3272
Free Heap after http.POST: 2840 Fragmentation: 28% MaxBlock: 1880
JSON Data Sent
Response: 200
Response Payload: 100 OK
Free Heap before HTTP.END: 3544 Fragmentation: 25% MaxBlock: 2544
Free Heap before FREE POST REQ: 3600 Fragmentation: 6% MaxBlock: 3384
Free Heap before RETURN: 5360 Fragmentation: 29% MaxBlock: 3384

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5584 Fragmentation: 32% MaxBlock: 3384
bufferSize: 2512
{"cmd":"insert","device":2,"sens":[59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368162,1548368162,1548368162,1548368162,1548368162,1548368162]}
json Length: 1750
Max Block: 1768
Free Heap before postRequest malloc: 3056 Fragmentation: 35% MaxBlock: 1768
postRequest: {"cmd":"insert","device":2,"sens":[59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59391,59391,59391,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59135,59391,59135,59135,59135,59135,59135,59135,59135],"sensTime":[1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368159,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368160,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368161,1548368162,1548368162,1548368162,1548368162,1548368162,1548368162]}
Free Heap before jsonBuffer.clear: 1296 Fragmentation: 30% MaxBlock: 856
Free Heap before http.begin: 3824 Fragmentation: 12% MaxBlock: 3384
Free Heap before http.POST: 3608 Fragmentation: 7% MaxBlock: 3384
Free Heap after http.POST: 3176 Fragmentation: 32% MaxBlock: 1768
JSON Data Sent
Response: -3
Error to String: send payload failed
FAILED: Response Payload: 

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512

Send sensors from 100 to 199
Free Heap before jsonBuffer: 5256 Fragmentation: 46% MaxBlock: 1768
bufferSize: 2512
WIFI turned OFF
Free Heap: 4992

As you can see from the output, the exact same size of data succeeds only on the second try and eventually aborts due to lack of minimum free heap required for the json parser buffer.
The response of my http.post seems to arbitrarily fragment my memory unnecessarily, I'm not sure if it requires a minimum of heap to operate "normally"...
Until now I had been solving this issue by reducing the packet size, but now it just seems weird for my esp not to being able to send more than 1.5kB of data on one post, splitting it into smaller pieces would take forever to send.

Anyone has this problem? Is this a bug in HTTPClient? Am I asking too much for my esp?
General recommendations are welcome :)

Originally posted by @EdoMon93 in #1872 (comment)
@devyte
Copy link
Collaborator

devyte commented Jan 26, 2019 •
edited
Loading

Trying to do HTTP operations with 5KB free heap, and then doing the whole json thing, is too much. Your problem is obviously oom and fragmentation, your logs clearly show that. You're not even leaving enough for the tcp/ip buffers and packets.
Out of sheer curiosity, why are you so out of heap? That hints at bad design. Why such a huge array with data? Why not read some info, send some info, read some info, send some info? If for whatever reason that isn't possible, why not read to a file, then stream the file out?
Some ideas:

  • ArduinoJson may not be the right choice for you, you could try a streaming lib that converts to string on the fly.
  • You could also measure stack usage at that point, and move some buffers to the stack. This is dangerous and must be done with a lot of care, but it's viable

Whatever else, it's not a problem with the core, but with the approach taken in your app.

Closing.

@devyte devyte closed this as completed Jan 26, 2019
@EdoMon93
Copy link
Author

EdoMon93 commented Jan 28, 2019 •
edited
Loading

@devyte thanks for the feedback, I guess I am asking too much from my esp.

Out of sheer curiosity, why are you so out of heap? That hints at bad design. Why such a huge array with data? Why not read some info, send some info, read some info, send some info? If for whatever reason that isn't possible, why not read to a file, then stream the file out?

  • Most of the time, the device has no wifi available so it needs to store data from several days of automomous operation.
  • It never sleeps because its doing stuff in the background so I don't really need persistance on the data (assuming no wdt resets).
  • Data in the arrays is written several thousands times a day so to avoid eating up my flash I thought using only ram would be essential for a long-lasting device.

I considered an SD card, however, I've had bad experiences with sd and vibrations I so decided to avoid that headache.

Might have to integrate an external ram to store my data, which I was trying to avoid.

Hope this satisfies your curiosity, throw me a bone if you think of a more appropiate solution or for further discussion.

Any rules of thumb for available space needed for HTTP operations?

@devyte
Copy link
Collaborator

devyte commented Jan 29, 2019

I may be misunderstanding your app, but off the top of my head:

  • keep a small-ish buffer in mem, say 8KB
  • Once the buffer fills up, write it to a file
  • Keep adding files until you need to dump after several days.
  • Maybe limit the number of files to the last N (say N= 16). Once you have N files and need to dump, reuse the oldest file (overwrite, or truncate to 0).
  • when it's time to dump, stream all files in reverse order, then add whatever you had in the buffer.
  • clear everything once dump is done

SPIFFS has wear leveling, so it's ok to write repeatedly, as long as you don't write to the entire filesystem continuously (that would make wear leveling useless).

If you go with an SD card, make sure it's a brand that is known to implement hw wear leveling (not all brands implement it). I believe at least SanDisk does. Most cheaper cards don't.

@EdoMon93
Copy link
Author

EdoMon93 commented Jan 29, 2019 •
edited
Loading

I'm not really convinced on using SPIFFS for storing my data, reliability being my main concern, but I'll give it a try, seems pretty standard.

SPIFFS has wear leveling, so it's ok to write repeatedly.

That is awesome

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@devyte @EdoMon93