Add Generic OTA for NonSSH boards #333

me-no-dev · 2015-12-17T18:06:56Z

This follows arduino/Arduino#4107
I have done the mods required, but have also removed some of your code (the one that sends a request to your server) and have needed to structure the code here and there to be able to read it (spaces and tabs are a nightmare).
My Fork is here: https://github.com/me-no-dev/arduino-eclipse-plugin
Feel free to pull what you need if you are interested in supporting this feature.

jantje · 2015-12-17T19:09:24Z

Thanks for bringing this to my attention.
I'm sorry to have to tell you that I am not tempted to pull your code. Let me explain why.
Due to your different taste of spaces and tabs there is lots of non-functional change I will need to undo.
Due to the fact you removed functionality I will need to undo.
Due to the fact I don't even know what this is about.
Due to the fact I'm currently rewriting the plugin.
There is lots of unnecessary work involved which -at this point in time- I'm not willing to take on.
I hope you understand
Best regards
Jantje

me-no-dev · 2015-12-17T23:54:29Z

sure no problem for me :)
and by the way it's not taste for spaces or tabs, but rather the resulting look when they are mixed together. I had really hard time following the logic with what I was seeing (i guess different IDE).
I would not mind to do a branch that is only changes specific, but I also do not agree with you not giving an option to not send tracking requests (as anonymous as they are). I hope you can understand that point of view :)

jantje · 2016-05-06T13:10:15Z

With #439 being fixed and having a sponsor I looked at the ESP8266 and ATO.
I understand how it worked with the ATO menu item. Is the 2.2.x solution hardcoded?

me-no-dev · 2016-05-06T13:32:47Z

nothing is hardcoded :) the command for upload comes from here
port, password and other settings come from mDNS TXT properties
here is most of the important changes (plus some unrelated).

jantje · 2016-05-06T13:58:04Z

Thanks for the input.
I see that you switch to another command based on the fact the upload port comes from the network.

tools.esptool.cmd=esptool
tools.esptool.cmd.windows=esptool.exe
tools.esptool.path={runtime.platform.path}/tools/esptool
tools.esptool.network_cmd=python
tools.esptool.network_cmd.windows=python.exe

would it not be better to switch the tool based upon the fact it is a network upload?
I'm thinking of the following in boards.txt

nodemcuv2.upload.tool=esptool
nodemcuv2.upload.network.tool=python

And in platform.txt

tools.esptool.cmd=esptool
tools.esptool.cmd.windows=esptool.exe
tools.esptool.path={runtime.platform.path}/tools/esptool
tools.python.cmd=python
tools.python.cmd.windows=python.exe

tools.esptool.upload.protocol=esp
tools.esptool.upload.params.verbose=-vv
tools.esptool.upload.params.quiet=
tools.esptool.upload.pattern="{path}/{cmd}" {upload.verbose} -cd {upload.resetmethod} -cb {upload.speed} -cp "{serial.port}" -ca 0x00000 -cf "{build.path}/{build.project_name}.bin"

tools.python.upload.pattern="{cmd}" "{runtime.platform.path}/tools/espota.py" -i "{serial.port}" -p "{network.port}" "--auth={network.password}" -f "{build.path}/{build.project_name}.bin"

me-no-dev · 2016-05-06T14:04:15Z

all changes are in sync with the ArduinoIDE and it is checking for network_pattern. if not available it will run the regular upload pattern (the one used for serial devices).
So network_pattern is not actually required, but used when needed.

me-no-dev · 2016-05-06T14:16:58Z

Thinking about it I could have gone either way... I guess this is what first came to mind

me-no-dev · 2016-05-06T14:28:12Z

btw I can help develop the other two tools used in conjunction with ESP8266/ESP32.
https://github.com/esp8266/arduino-esp8266fs-plugin
https://github.com/me-no-dev/EspExceptionDecoder

jantje · 2016-05-06T14:46:15Z

@me-no-dev
If you do not try to remove functionality (like the anonymous http requests) and stick to the project guide lines as described in the read.me, your changes are welcomed.

me-no-dev · 2016-05-06T14:56:06Z

I will re-fork once you have OTA working and I see my projects compiling fine. From then on PRs will be fine. BTW what IDE are you using to work on this project? You have lots of mixed tabs/spaces and that drives my editors and eyes mad...
Maybe we can agree on a setting to disable the anonymous requests? Have them on by default... it's fine by me

jantje · 2016-05-06T14:57:30Z

For those interested with #439 fixed it is possible to upload via OTA when using 2.1
With this code on Arduino
ArduinoOTA.setHostname("charger");

Select OTA and as port type "charger.local" (only this; no IP address or anything else)
You get an dialogue box because the plugin tries to reset "charger.local" but it works.

Edit: only anonymous works this way. No passwords!!

jantje · 2016-05-06T14:59:16Z

@me-no-dev

BTW what IDE are you using to work on this project?

Eclipse. See here on how to set it up https://github.com/jantje/arduino-eclipse-plugin/blob/master/readme.md

jantje · 2016-05-06T18:42:30Z

Fixed #444 to avoid resetting the com port when working with the workaround.
Unfortunately in 2.1 there is following line in boards.txt

nodemcu.upload.wait_for_upload_port=true

If you remove this line from boards.txt for you boards there should no longer be a reset of the com port.
Now 2.1 works as good as it gets I'm moving to 2.2

master

jantje · 2016-05-06T22:25:41Z

This worked for me here but I did not do enough testing on other boards so I made the branch working and put it there.
Is it possible to give it a go from the working branch?

jantje · 2016-05-07T20:20:30Z

I just tried this in my production environment and it didn't work.
The problem is that network.port and network.password are not defined

me-no-dev · 2016-05-07T20:38:30Z

what did you try? If you are cool with it, I have setup a clean fork and eclipse and can get to implement this and PR it

jantje · 2016-05-07T22:05:28Z

I just finished the part that didn't work. Checking in right now.
I need to add the auth part.
We have a discussion on the dev list about the formatting. Seems eclipse default for java is indent 4 and tab 8, mix tab and spaces.

Also auth is extracted but not jet used.

me-no-dev · 2016-05-07T22:54:29Z

yeah... why that did not work? I'm at the same point otherwise. Its just not parsing them in the final command?

IEnvironmentVariable var = new EnvironmentVariable(Const.ENV_KEY_SERIAL_PORT, this.myAdderss);
contribEnv.addVariable(var, configurationDescription);
var = new EnvironmentVariable(Const.ENV_KEY_NETWORK_PORT, this.myPort);
contribEnv.addVariable(var, configurationDescription);
var = new EnvironmentVariable(Const.ENV_KEY_NETWORK_PASSWORD, this.myPassword);
contribEnv.addVariable(var, configurationDescription);
//values above are correct but the result from below does not parse them
String command = envManager.getVariable("A.TOOLS." + this.myUploadTool.toUpperCase() + ".UPLOAD.NETWORK_PATTERN", configurationDescription, true).getValue();

And this same code worked in the previous version

jantje · 2016-05-08T11:00:10Z

I just checked on my system.
I can do ATO and serial upload in 2.1 and 2.2.
I used the strings provided in the combo box. Variants failed on me.
The only difference I see between 2.1 and 2.2 is that 2.2 does not include -vv so you have to be patient to see the result.
Note that you need the working thread or download site http://eclipse.baeyens.it/nightly_jantje

jantje · 2016-05-08T11:01:02Z

authentication is not there yet. I need a think on how to do it.

me-no-dev · 2016-05-08T11:12:14Z

here is my branch with OTA implemented the same way that it's done in ArduinoIDE. I do face the problem I outlined above though... the rest is working

jantje · 2016-05-08T11:56:22Z

My implementation is completely different from what you have done.
I do not use a special uploader.
I do not parse data at upload time.
I parse data at configuration time.
I use redirection in the environment variables.

#for esp8266 network upload
tools.esp8266ATO={tools.esptool.network_cmd}
tools.esp8266ATO.upload.pattern={tools.esptool.upload.network_pattern}
esp8266.network.upload.tool=esp8266ATO

So if the plugin identifies the com port as a "network" it will use the tool in
[boardid].network.upload.tool
if it exists

me-no-dev · 2016-05-08T12:00:29Z

is what you have done platform dependent? Does it require changes in platform? And BTW ATO -> OTA (Over The Air).

jantje · 2016-05-08T12:16:50Z

is what you have done platform dependent?

Not that I'm aware off.
Which OS do you use?

ATO -> OTA

Thanks fixing this.

me-no-dev · 2016-05-08T12:19:06Z

final working implementation
I did not ask about host platform, but rather target platform (ESP8266 in this case). I ask this because of what you pasted above.

me-no-dev · 2016-05-08T12:20:43Z

Other than that I am on a Mac and I required this to get all of my serial devices to show

jantje · 2016-05-08T12:25:27Z

for the serial ports to show on your system.
It is a regular expression so I'll make the part you removed optional.
I'm not on mac so I have to rely on others for that.

I also do not understand why you are trying OTA to get to work as it is working in the working thread

jantje · 2016-05-08T12:29:26Z

does this regex work for you
"^cu\\.(.*serial|usb)?.*"

me-no-dev · 2016-05-08T12:36:28Z

no. serial ports can have many different names on a Mac

ls /dev/cu.*
/dev/cu.Bluetooth-Incoming-Port /dev/cu.H-C-2010-06-01-DevB     /dev/cu.SLAB_USBtoUART          /dev/cu.Scooterino-DevB         /dev/cu.wchusbserial1440

All those are valid serial ports and I have not even plugged all of my different boards

me-no-dev · 2016-05-08T12:37:56Z

I also do not understand why you are trying OTA to get to work as it is working in the working thread

what do you mean by that? I just replicated what the other uploaders do.

jantje · 2016-05-08T12:41:23Z

no. serial ports can have many different names on a Mac

Let me find out why the additional filter was added. It is clear it is to limiting for your case.

jantje · 2016-05-08T12:43:46Z

I also do not understand why you are trying OTA to get to work as it is working in the working thread

what do you mean by that? I just replicated what the other uploaders do.

I mean that I have implemented OTA for ESP8266 and it works fine.
1 implementation is enough. So why spend time on another implementation?

me-no-dev · 2016-05-08T12:44:39Z

"^cu.(.serial|usb)?." will cover the genuine Arduino boards, but will not cover any board that uses silicon labs chip, bluetooth and some more. /dev/cu.SLAB_USBtoUART is my NodeMCU Amica board (ESP8266)

me-no-dev · 2016-05-08T12:47:00Z

you did not answer if you have implemented it just for ESP8266? It sounds to me that that is the case, where what I propose is platform independent and mirror to the functionality in the mainstream Arduino IDE (also my contribution). I have not submitted a PR, it's totally up to you which way you want to go with it.

jantje · 2016-05-08T12:55:07Z

I have implemented an addition to the boards.txt to change the upload tool based on the "upload port" being a network name.

In other words:
If the plugin identifies the com port as a "network" it will use the tool in
[platformid].network.upload.tool
if it exists

This is as I proposed here
#333 (comment)

To make this work with ESP2866 boards.txt I made specific changes.
Like

#for esp8266 network upload
tools.esp8266ATO={tools.esptool.network_cmd}
tools.esp8266ATO.upload.pattern={tools.esptool.upload.network_pattern}
esp8266.network.upload.tool=esp8266ATO

here https://github.com/jantje/arduino-eclipse-plugin/blob/working/it.baeyens.arduino.core/config/pre_processing_platform_default.txt#L11

Edit: changed boardid to platformid

jantje · 2016-05-08T12:57:27Z

As to the serial
This is the change that introduced the extra filter
f806fc0#diff-754b17547d5b9866369625d5ba75b38e

me-no-dev · 2016-05-08T12:57:48Z

exactly :) your way requires a change to the plugin to support other boards, while mine requires nothing :) it just works (I have 4 different platforms/boards that use this method)

jantje · 2016-05-08T12:59:25Z

(I have 4 different platforms/boards that use this method)

I'm sorry but it is a crappy method

me-no-dev · 2016-05-08T13:10:01Z

those are the different usb-uarts that I can plug. You can use this list to compile a better regex

/dev/cu.SLAB_USBtoUART
/dev/cu.usbmodem1A1231
/dev/cu.wchusbserial1440
/dev/cu.usbserial-A9Q1PBJJ

How is my method crappy if it requires less to work on more boards? Let's forget that I have more than those boards OTA enabled. Are you gonna make a new change when the ESP32 board comes out? And expect the users to update their plugin just to support one command on a new board? How is that a less crappy method?

jantje · 2016-05-08T13:20:51Z

It is crappy because the boards.txt specification has the capability to change the tool and as such change the complete upload procedure.
This is what is used in 2.1
2.2 changed to not use the boards.txt supported "tool feature" but add code to change all the underlying command calls.
So instead of using the supported method new code has to be added for doing something that is already supported.
That I call crappy.

me-no-dev · 2016-05-08T13:25:41Z

You do your thing :) I'll go with crappy because I OTA to more boards and I need that functionality working.

jantje · 2016-05-08T13:27:59Z

just to be clear. I will not accept a pull request based on changing the underlying commands.

me-no-dev · 2016-05-08T13:31:12Z

Not sure what you mean by "changing the underlying commands", but I was not going to submit a PR and I think that was made perfectly clear by this conversation so far :) I expect you to implement your way, and me to do it my way in my fork in a separate branch.
Can we close this now?

jantje · 2016-05-08T13:53:55Z

Can we close this now?

I keep issues open till they are delivered in a release.

wimjongman · 2016-05-08T16:42:46Z

Why do you have that extra dot before serial in

"cu.(.serial|usb)?."

—

Sent from TypeApp

On May 8, 2016, 14:29, at 14:29, jantje [email protected] wrote:

does this regex work for you
"^cu.(.serial|usb)?."

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#333 (comment)

jantje added status: waiting for sponsor @ patreon importance: improvement request domain: upload importance: board specific labels Dec 17, 2015

jantje mentioned this issue Apr 22, 2016

V3 Upload menu option is not taken into account (for example with esp8266 ) #439

Closed

jantje added Help wanted If you want to become a active contributor, start looking at this issue. and removed status: waiting for sponsor @ patreon labels May 6, 2016

jantje pushed a commit that referenced this issue May 6, 2016

#333 This should work but I'd like to do some testing before making it

97aefec

master

jantje removed the Help wanted If you want to become a active contributor, start looking at this issue. label May 7, 2016

jantje self-assigned this May 7, 2016

jantje pushed a commit that referenced this issue May 7, 2016

#333 network port is now extracted from bonjour

2a51a76

Also auth is extracted but not jet used.

jantje added the status: fixed in 3.1 label May 19, 2016

jantje closed this as completed Aug 1, 2016

Add Generic OTA for NonSSH boards #333

Add Generic OTA for NonSSH boards #333

Comments

me-no-dev commented Dec 17, 2015

jantje commented Dec 17, 2015

me-no-dev commented Dec 17, 2015

jantje commented May 6, 2016

me-no-dev commented May 6, 2016

jantje commented May 6, 2016 • edited Loading

me-no-dev commented May 6, 2016

me-no-dev commented May 6, 2016

me-no-dev commented May 6, 2016

jantje commented May 6, 2016

me-no-dev commented May 6, 2016

jantje commented May 6, 2016 • edited Loading

jantje commented May 6, 2016

jantje commented May 6, 2016

jantje commented May 6, 2016

jantje commented May 7, 2016

me-no-dev commented May 7, 2016

jantje commented May 7, 2016

me-no-dev commented May 7, 2016 • edited Loading

jantje commented May 8, 2016

jantje commented May 8, 2016

me-no-dev commented May 8, 2016

jantje commented May 8, 2016

me-no-dev commented May 8, 2016

jantje commented May 8, 2016

me-no-dev commented May 8, 2016

me-no-dev commented May 8, 2016

jantje commented May 8, 2016

jantje commented May 8, 2016 • edited Loading

me-no-dev commented May 8, 2016

me-no-dev commented May 8, 2016

jantje commented May 8, 2016

jantje commented May 8, 2016

me-no-dev commented May 8, 2016

me-no-dev commented May 8, 2016

jantje commented May 8, 2016 • edited Loading

jantje commented May 8, 2016

me-no-dev commented May 8, 2016

jantje commented May 8, 2016

me-no-dev commented May 8, 2016

jantje commented May 8, 2016 • edited Loading

me-no-dev commented May 8, 2016

jantje commented May 8, 2016

me-no-dev commented May 8, 2016

jantje commented May 8, 2016

wimjongman commented May 8, 2016

jantje commented May 6, 2016 •

edited

Loading

jantje commented May 6, 2016 •

edited

Loading

me-no-dev commented May 7, 2016 •

edited

Loading

jantje commented May 8, 2016 •

edited

Loading

jantje commented May 8, 2016 •

edited

Loading

jantje commented May 8, 2016 •

edited

Loading