-
Notifications
You must be signed in to change notification settings - Fork 13.3k
Found a way to upload data VERY fast by wifi #1853
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Yes, doing it this way certainly works. There is a pull request (#1570) which attempts to speed up WiFiClient transfers so that ESP8266WebServer and other libraries can benefit from this. It's currently blocked by lack of testing though. |
One question would be, why isn't the setNoDelay on by default? |
The nagle algorithm is enabled by default for all systems and all tcp stacks. This is done to protect the network from inadvertant sending of very small packets, with nagle algorithm only the first packet is small and the following ones are batched up thus improving the overall network performance. It makes sense to keep this behavior as is and allow application developers to override it if they have a good reason to do so. |
Most TCP stacks do not send only one packet and wait for an ACK. They send a few packets at once. I don't think network congestion is an issue with this MCU, it dosn't have the capacity to saturate a network. |
Vote for keeping current behavior and document the tcpNoDelay option (as it is done also with other languages, like java...). |
Documenting that feature would certainly help a lot. Speaking of which, is there any place with ALL the current esp8266 specific Arduino API? |
I would add it here as a kind of "hint": https://github.com/esp8266/Arduino/blob/master/doc/libraries.md#wifiesp8266wifi-library |
I've tried to implement the suggestions (client.setNoDelay(1); and 1760 packet), but am still getting slow uploads...approximately 82k bits/sec, or about 1/10 of the OP's rate. IE 11 is a little bit faster than Firefox 47 (22 sec vs 30 sec to serve a 307kB image from SPIFFS). I'd greatly appreciate some help (or is this the best that can be done from SPIFFS?)
|
Which version are you using? Current git master or one of the releases? |
Try to upload an empty file, like all 0 or something, to see where the
bottleneck is.
|
With the latest git version you should get better performance without using |
Thanks for the quick replies. I was using v. 2.2.0... will update to 2.3.0, try |
What i have described above is not available in 2.3.0. You need to get latest git version for that (check repository Readme for instructions). |
Ah, so. Will do... |
Yipee!!!! It works! Super fast! Less than a second to serve the 307kB file to IE and Firefox. Thank you very much!!!! |
Sorry, And second question is (more dedicated to igrr): |
Yes it's true. |
I forgot the exact speed, but it only took a few seconds to transfer a What exactly are you trying to do? Transfer files? If so, where are they On 8/17/2016 6:33 PM, Pe3ucTop wrote:
|
Thanks for yours answers, |
NoDelay can only work with packets below 1460 bytes. You send as much as you can (2 x 1460 the first time, then 1460 each) and wait for ack. When ack is received, you send another 1460 bytes. |
This information has been very useful to me, so thank you all for posting it all. I wanted to share my experience and see if anyone can comment on a few questions which I had. I tried compiling the SD Web Server example using the 2.3.0 tools available in the board manager and was able to test the download speeds - they were pretty abysmal at ~ 300kbps (call it 30kb/s). This was also true for uploads. After finding the information in this thread, I tried pulling the latest ESP package from git and dropping it in - without modifying any of the code in the SD Web Server example, again. This time, I noticed that the files would start transferring (the first 32kb) and then stop, the debug would print "sent less than expected!" I reverted to the 2.3.0 version, and applied the changes mentioned above to the library (HTTPWebServer) -- that is setNoDelay(1) and changed the HTTP packet size to 1760 from 1460). This resulted in throughput speeds of approx. 1.8mbps (225kb/s). My questions are:
Thank you! |
Hello everyone,
This is errtest.php server side file for receiving the data:
Other notes:
Yes you can speed up SPI using SD.begin(SS, CLOCK_SPEED) (for example 50000000 instead of 8Mhz default, this way you are reading at maximum sd card speed ). This will decrease total reading time from 4s to 1.8s reading 1MB of data. Total uploading time also decreases, but it is of course slower than using SPIFFS. |
Server side File Accept code in php is the same as above, but instead of ussing SPIFFS, reading data from SD example:
|
Hello to all, Sorry, I can't comment on Arduino and it's library part. I'm programming ESP8266 with C and NONOS_SDK . |
in our case server is apache + php |
Hi Kakarotas, I've just tried your SPIFFS example above in IDE 1.6.12 and SDK 2.3.0 on a NodeMCU 12E and it fails with the attached error. If I cast the write:
Then it compiles ok but I don't have the server side to check if it's functional. I'm a bit new to all of this - is the cast a correct approach? I'm actually trying to debug a problem in my own code where the client.write won't write the full amount of code (I needed to break it into around 3K chunks... which is very slow). I was originally using an array stored in ROM (with PROGMEM) but realised that using SPIFFS seems like the right approach to store large files in the flash. Original:
Now:
This new approach seems to just send 1 byte (checked with wireshark). How can I send the whole file without breaking it up into chunks and having the delays associated with .write? Any advise would be appreciated! Error running your example:
|
With the above code, you are sending a single byte with the value of "spiFile" file handle, not the content of the file. |
Thanks Martin. To be honest, I've just copied the solution from kakarotas: client.write(webFile); If that's not correct, can you advise how to fix it? Thanks |
I've posted a fix, but in the forum : http://www.esp8266.com/viewtopic.php?p=57668#p57668 |
Thanks Martin. I've replied there too. I thought each additional client.write causes delays, and it didn't seem to be needed by kakarotas. I'll give it a go but i think it'll be slow, and certainly not giving me high KB of tranfer per second? |
For all those who may have problems like mph070770. You have to instal git version and delete, move or rename (I preffer rename) the following folder (which contains boards info and 2.3. version files):
|
There might be a bug somewhere in the core library. I have a gif-file that is about 200kb. If I try to fetch it, it usually works just fine, but if I keep refreshing the page, it gives me errors once in a while. It looks like the entire file is not sent and gets stuck at about 199kb. You can replicate it with this simple code: Just make sure you have a file called "test.gif" in your SPIFFS.
|
I am struggling to read a plain text SPIFFS file. The file is 100 lines long with each line being 1kb. It reads the first 30 lines within a couple of seconds but then each subsequent line takes 5 seconds or more. Sample sketch below, anyone know what the problem is?
|
The original issue seems to be fixed in latest git per relevant comments above, and per the referenced PR. There is a known issue with concurrent requests (e.g.: running a webserver and refreshing the browser repeatedly), which is going to be addressed with lwip2. |
so its not possible 5 Mbps download spped from esp8266 sd server? |
@haxord7 raw esp-upload bandwidth exceeds 5Mbits/s. |
Hi friends, I try to Upload SPIFFS large size .txt file [500KB] in HTTP web server POST request method. But my data didn't upload my server after use the given code. this is my code
} Serial monitor continuously show only file name and file size otherwise server posting or connection error details not show I don't know will be next please help me. |
There are timing requirements to respect on the ESP, and your code doesn't. |
This comment has been minimized.
This comment has been minimized.
Hi...I'm saving to SPIFFS content edited in a text area.. through web sockets..but I find the limit being 2K bytes aprox... |
I am posting this here because I never found the solution online, and I've been looking for days. Before this 'hack', the transfer speed on Windows it was about 6.7 KBps, and on Linux, about 36KBps
Now the speed is very high, something like many mbps, at least 187 KBps, limited by my SD reading time.
So, here is to to send data really fast, on any host:
First, you have to disable the Nagle 'feature', like so:
WiFiClient client; //stuff client.setNoDelay(1);
Then, you need to set the packet size HIGHER than the MTU, which is 1460 bytes. I got good results sending 1760 at a time, but as I said, the bottleneck is the SD card (I am uploading files by ftp). If the size is lower or equal than the MTU, you won't get good results. I think the trick is forcing it to send two packets.
Can someone please try to reproduce my findings? And if it works, maybe this should be documented somewhere (more than here).
Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.
The text was updated successfully, but these errors were encountered: