Arduino v1.6.3 Board Manager Package

[toddtreece]

I have been working on adding support for Adafruit's boards to the new Arduino IDE Board Manager, and I packaged up your core and added it to our proxy server. The proxy server injects new boards into the Board Manager's package_list.json file and serves the modified version back to the IDE. More info can be found here: https://blog.adafruit.com/2015/04/06/adding-custom-boards-to-the-arduino-v1-6-board-manager/

Not sure if that's helpful, but it might be easier than rebuilding the IDE for new releases. Let me know if you have any questions, or if you would like me to remove it from the proxy.

[igrr]

This is certainly very cool, and people have been asking for this earlier (#13).

I would like to understand the packaging process a bit better. What needs to be done to generate the distributions and the packag_list.json file from this git repository? If this is some manual process, then perhaps we can write a script to automate it and integrate with GitHub releases. If you have already written such a script, perhaps you could add it here by means of a pull request?

[toddtreece]

@igrr check out the JSON files in the boards and packages directories here: https://github.com/adafruit/adafruit-arduino-proxy

The proxy adds the ESP8266 board JSON to the ESP8266 package JSON, and then adds the package to the package list that it grabs from the arduino.cc server. I did it this way so people wouldn't need to modify anything other than the proxy preferences in a Arduino v1.6.3 IDE fresh install.

I packaged esptool as separate archives for each platform, and created a package for the ESP8266 core. Check out the links to the archives in those JSON files if you would like to download and inspect them. They should be fairly easy to automate if you would like to.

[igrr]

Ah, right, thanks.
I think at the moment package.json is still missing the xtensa toolchains, is that true?
Also for esptool I would use github releases for download links:
https://github.com/igrr/esptool-ck/releases
as it is already packaged for each platform. This should also make it easier to upgrade esptool (which I plan to do soon anyway).
I'll make these changes and will submit them to https://github.com/adafruit/adafruit-arduino-proxy as a pull request.

[toddtreece]

@igrr sounds good. i didn't notice that repo.

i'll deploy to our proxy once you have submitted the pull req. you can test the changes by running the proxy locally and pointing your IDE at localhost port 5050. install instructions are in the readme. thanks!

[toddtreece]

@igrr a couple of other things:

• new releases can be added to boards/esp8266.json by converting it to an array and adding a new entry. new versions of esptool can be added in packages/esp8266.json, and then referenced in the board JSON as a dependency.
• the folder structure for the tools and boards is very picky, and might need to be modified to work with the board manager. check out my archives for both the core and esptool if you have problems.

[gerardwr]

Hi,

Sounds like an excellent way of distribution to me.

I just tried this procedure on my Mac OSX 10.10.2. Here are my findings:

• The Board Manager installed support for the ESP8266 OK.
• A test sketch compiled OK.

But esptool is NOT listed in the Programmer selection list, so uploading the sketch is not possible.

Using the Board Manager the ESP support can also be removed, nice feature when upgrading to a new release.

[toddtreece]

@gerardwr try a restart of the IDE. Sometimes the Board Manager is a bit flakey with installing tools. I was able to compile and upload sketches.

[gerardwr]

@toddtreece : Restarted the IDE as suggested, and that solved it. Sketch is uploading and running OK now.

Thanks, excellent work!

[toddtreece]

@gerardwr great!

[nerdralph]

Didn't work for me:
C:\Users\XXuser\AppData\Roaming\Arduino15\packages\esp8266\tools\esptool\0.4.2/xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ -D__ets__ -DICACHE_FLASH -IC:\Users\XXuser\AppData\Roaming\Arduino15\packages\esp8266\tools\esptool\0.4.2/sdk/include -c -Os -mlongcalls -mtext-section-literals -fno-exceptions -fno-rtti -std=c++11 -MMD -DF_CPU=80000000L -DARDUINO=10603 -DARDUINO_ESP8266_ESP01 -DARDUINO_ARCH_ESP8266 -IC:\Users\XXuser\AppData\Roaming\Arduino15\packages\esp8266\hardware\esp8266\1.6.1\cores\esp8266 -IC:\Users\XXuser\AppData\Roaming\Arduino15\packages\esp8266\hardware\esp8266\1.6.1\variants\esp01 C:\Users\XXuser\AppData\Local\Temp\build3560924801977961826.tmp\BlinkLED.cpp -o C:\Users\XXuser\AppData\Local\Temp\build3560924801977961826.tmp\BlinkLED.cpp.o
Error compiling.

This is under Win7E/64. I have detailed compile output enabled, and only get what you see above.
Arduino 1.6.1 with the esp8266 core installed using Jeroen Beemster's 0.21 setup.exe works fine for me. I've also tested with the latest build of esptool-ck using the 921.6kbps upload.

[toddtreece]

@nerdralph This is for v1.6.3+ of the official Arduino IDE from arduino.cc. The support for the board manager stuff wasn't added until 1.6.2 (which was buggy), so 1.6.3 is the minimum requirement.

[nerdralph]

@toddtreece My test was with 1.6.3 (the .zip package if it makes any difference); you can tell from the '-DARDUINO=10603' build flag.

[rogerclarkmelbourne]

Guys

I dont know if its worth spending too much effort on this feature

IMHO the boards manager feature if quite problematic, because it puts the compiler in the roaming data folder which may be synced to Microsoft's cloud sync service if you have it enabled

I have already reported an issue to the IDE dev team, as its in breach of MS guidelines to put binaries in this folder, and it also causes issues for Linux systems and anyone on a multi user system e.g. universities etc.

The old system by unzipping into the Hardware folder is still more reliable for all users, as it works on all versions back as far as around 1.5.5 right up to 1.6.3

IMHO. Forcing people onto 1.6.3 when the whole IDE is in such a state of flux is not necessarily the best idea

[toddtreece]

@nerdralph I'll have to test that out. I misread the error messages. I was reading the core version.

@rogerclarkmelbourne That's probably true, but it seems like moving that folder shouldn't be that big of a deal in the next release. We are trying to find ways to support Adafruit boards in the latest IDE, which people will download, and some of our changes currently require overwriting files in the latest version of the IDE when using the .zip method, which can also be problematic. This is not meant to force users or the ESP8266 project to use 1.6.3, it's just more of a "here's something we are trying".

[nerdralph]

Whether it is by way of the boards manager or some other method, an easy way to add esp8266 support to the Arduino IDE would be welcome.
Earlier today the version tag on the Arduino git was bumped to 1.6.4, so it could be released in a few days. Currently, getting the latest esp8266/Arduino core is a pain. I went with Jeroen's update package, although Roger's or Sandeep's might have worked too. I cloned esptool-ck and built it (needed for 921,600 upload speed, and hopefully my auto-reset using break will be included soon). Then the boards.txt and core files (at least core_esp8266_main.cpp) needs to be updated to get the upload baud rate selection menu and the CPU speed selection. I still won't have the serial monitor patch to disable DTR and RTS (not a big deal since I rarely use serial monitor).
I think the git repo should exclude fork/clone of the Arduino IDE, or at most include it as a git submodule. Mods to the IDE (like DTR and RTS control in the serial class) should be pushed to the official Arduino IDE repo.

[rogerclarkmelbourne]

I can't devote enough time to keep my repo up to date, especially as the Arduino team seem to release fairly major changes in every new release.

IMHO, Sandeep's version is the best one to use, as hopefully the way he has set up his repo cores file as a sub module of the main repo, should mean it automatically track the core changes.

[nerdralph]

Sandeep's repo looks pretty fresh, though the release file (0.0.2) is stale and broken (at least on Win7E). Downloading the latest version of the repo master isn't much better. The new boards.txt is in it so the faster upload speeds show up, but I still get the same "error compiling" that I got using the board manager with the Adafruit proxy.
The "submit issues" is a circular link back to the repo. I'm not sure if that's a mistake or a signal that he doesn't want any bug reports.

[rogerclarkmelbourne]

Ralph,

If you get the "Error compiling" message, you need to copy libiconv-2.dll into the root of your Arduino install.(look in AVR tools folder, it used to be in there somewhere with the other helper exe's)

I was getting this issue in 1.6.1 but it seemed to have gone away in 1.6.3, but I could not track down why the error was not in 1.6.3, because I thought that the issue was that Ivan's version of the compiler doesn't statically link the helper DLL's like libiconv-2 into gcc (and g++ etc)

But I thought that possibly 1.6.3 had changed some internal environment vars that allowed the Xtensa compiler to find what it was looking for, as I was pretty sure my copy of 1.6.3 was clean and I'd not copied libiconv-2

I did look at recompiling the Xtensa compiler, in order to statically link the helpers, but I haven't managed to build a complete set of compiler files yet, let alone work out how to change the automake templates etc to guarantee static linking.

To build the compiler you have to install MinGW and also install GitMysys and make some config changes and also install wget if its missing.

Then you have to change get the build script from Gist and modify it
etc etc etc

And then it only seems to make 80% of the compiler files for me :-( :-(

Actually, the build scripts do specify that the compiler should be statically linked, there is the flag --enable-static used liberally, but either it doesn't get passed though to make file, or the issue isn't to do with static linking at all

I'm beginning to wonder if the process that Ivan uses to build the compiler is the best option. As I've seen other methods listed and they may or may not work better.

Either way, compiling a compiler never seems to be an easy process !

[nerdralph]

Roger,
Compiling gcc-xtensa is more complicated than the regular gcc, which can be tricky to figure out by itself. I've started by compiling a standard gcc release from source (just got a gcc 4.9.2 build working yesterday) before I try gcc-xtensa. Next I'm trying to figure out what branches of gcc have the patches:
https://gcc.gnu.org/ml/gcc-cvs/2015-03/msg00078.html
If it is in the 5.0 RC snapshot, then that would be great, since there's big improvements to the optimizer since 4.9 (the new IPA and improvements to LTO).

[marvinroger]

xtensa-lx106-elf-g++.exe stopped working for me on compilation with the Error compiling message on Arduino 1.6.3 - Windows 8.1 (x64). As @rogerclarkmelbourne suggested, copying libiconv-2.dll into the root of the Arduino install did help, so the issue have not gone away with 1.6.3.

[rogerclarkmelbourne]

@marvinroger

Thanks for confirmation that this issue is still in 1.6.3

We have isolated the issue to be a build setting when the compiler is compiled and linked such that lib iconv is dynamically linked to the MinGW libivonv-2 dll

However none of has any experience in compiling a compiler from source code, hence why this is not getting resolved quickly

[nerdralph]

I'm learning more about the process, with the intention of building from
the official gcc sources. Max's patches are in gcc-5.0RC, so thats what
I'm planning on building.
https://gcc.gnu.org/ml/gcc-cvs/2015-03/msg00078.html
I've figured out gcc dependency on as from binutils and the gmp/mpfr/mpc
library dependency. I figure I'm a few days away from having a working
xtensa-gcc5 build. Then I still need to figure out how to integrate the
expressif libs. Longer term I want to compare libc implementations like
uClibc, musl, and newlib.
On Apr 15, 2015 5:59 PM, "Roger Clark" [email protected] wrote:

@marvinroger https://github.com/marvinroger

Thanks for confirmation that this issue is still in 1.6.3

We have isolated the issue to be a build setting when the compiler is
compiled and linked such that lib iconv is dynamically linked to the MinGW
libivonv-2 dll

However none of has any experience in compiling a compiler from source
code, hence why this is not getting resolved quickly

Reply to this email directly or view it on GitHub
#57 (comment).

[nerdralph]

I have been able to build binutils with target=xtensa-elf, but no luck with GCC so far. I've tried a bunch of target options including xtensa-lx106-elf, and none have worked.
Looking at the GCC source, XSHAL_ABI has to be set to XTHAL_ABI_CALL0, but I haven't found any configure target options that do that. I don't want to admit defeat, but relucantly I emailed Max to ask how to enable the call0 ABI. Hopefully I'll get a reply from him.

[grantnlee]

Any ideas here to get up and running?

Windows 7 64bit
Fresh install of IDE 1.6.3
Downloaded and extracted Sandeep's latest package
Copied his "esp8266com" folder to here: C:\Users\poloj1\Documents\Arduino\hardware\esp8266com
Copied the libiconv2.dll to this Arduino root folder here: C:\Program Files (x86)\Arduino

The IDE loads and under Tools|Board I can select select "Generic ESP8266 Module", But I still get the Error Compiling message when I try to compile an empty program or the Blink example....

Thoughts?

Error message:
Arduino: 1.6.3 (Windows 7), Board: "Generic ESP8266 Module, 80 MHz, 115200"

C:\Users\poloj1\Documents\Arduino\hardware\esp8266com/esp8266/tools/windows/xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ -D__ets__ -DICACHE_FLASH -IC:\Users\poloj1\Documents\Arduino\hardware\esp8266com/esp8266/sdk//include -c -Os -mlongcalls -mtext-section-literals -fno-exceptions -fno-rtti -std=c++11 -MMD -DF_CPU=80000000L -DARDUINO=10603 -DARDUINO_ESP8266_ESP01 -DARDUINO_ARCH_ESP8266 -DESP8266 -IC:\Users\poloj1\Documents\Arduino\hardware\esp8266com\esp8266\cores\esp8266 -IC:\Users\poloj1\Documents\Arduino\hardware\esp8266com\esp8266\variants\generic C:\Users\poloj1\AppData\Local\Temp\build1371331012900594506.tmp\Blink.cpp -o C:\Users\poloj1\AppData\Local\Temp\build1371331012900594506.tmp\Blink.cpp.o

Error compiling.

[luc-github]

that is interresting - I have same configuration 1.6.3/W7 64 and it works perfectly but did not copied libiconv2.dll

but on my 8.1 64bits I have exactly same behaviour as yours https://github.com/sandeepmistry/esp8266-Arduino/issues/1

EDIT:
seems a missing DLL I compared my 2 systems and on one of them which work I have GIT for windows with a bunch of GNU tools/dll. I have downloaded/installed http://git-scm.com/download/win on not working system and it is fine now

[luc-github]

found it : libiconv-2.dll need to be in path not libiconv2.dll (difference is the dash)

[Makuna]

I freshly reinstalled Arduino IDE, copied over the esp2866com, but I don't have a libiconv-2.dll on my Win81(x64) machine and I am hitting the compile error problem.

Where is this dll supposed to be?

[luc-github]

in the path so as mentioned above _C:\Program Files (x86)\Arduino_ is good place to copy it

[Makuna]

luc-github pointed me a git tool install (http://git-scm.com/download/win) that gave me the dll, so I could then copy it to the root of Arduino, which solves the problem.

Most Arduino users will not have this DLL on their system.

[grantnlee]

@luc-github it took a while, but I eventually saw that same typo, that there is a similar dll missing the dash. But even then, I found that I don't have the correct dll on my machine. Used search and came up empty. So I went and pulled the full 1.6.1 package from esp8266/Arduino and was able to extract that full zip into a directory and can now compile fine. So I have both installed... I should try moving the dll with the dash into the 1.6.3 version to see if it works there... Will report back.

But for now I am on to trying to get a program pushed to an esp8266... Hoping for the best so I can start writing code. I have both the esp01 and the esp12. TBD which is easier to write to...

[grantnlee]

Had to try it out. I copied the libiconv-2.dll file that I had downloaded along with the full 1.6.1 package on esp8266/Arduino into the root of the 1.6.3 deployment that I described in a previous post... The 1.6.3 version now also compiles without any errors.

[grantnlee]

Yay! Blink is blinking! Used the example program from the IDE and replaced output 13 (Arduino pinout) with output 1 (NODEMCU / ESP8266 pinout).

The IDE downloads to the NODEMCU device in the exact same manner as an Arduino. I am glad to have started with the $15 NODEMCU ESP-12 device to validate that all is working. As some point I will move on to the $3 standard ESP-12 boards to get the super inexpensive (nearly disposable) devices working...

Time to play...

[rogerclarkmelbourne]

It looks like Arduino IDE 1.6.3 doesn't use this dll any more. The older versions did. They must be using a different AVR compiler that is statically linked with lib iconv, rather than requiring the external dll file

[Makuna]

I had the same problem with 1.6.1 earlier today though.
Some of the git tools come with it, and they can modify the path so it is accessible.

    Sent by Outlook for Android

On Fri, Apr 17, 2015 at 9:04 PM -0700, "Roger Clark" [email protected] wrote:
It looks like Arduino IDE 1.6.3 doesn't use this dll any more. The older versions did. They must be using a different AVR compiler that is statically linked with lib iconv, rather than requiring the external dll file

---

Reply to this email directly or view it on GitHub:
#57 (comment)

[sej7278]

i'm not a fan of the boards manager either, its breaking all sorts of best-practices on linux/osx (as well as windows that @rogerclarkmelbourne mentioned above) to have binaries and certainly compilers in $HOME directories, plus it adds a lot of content to a $HOME directory backup, especially on a multi-user system.

that said, i do think maintaining a fork of the whole IDE is a bit over-the-top, and a "core" like https://github.com/sandeepmistry/esp8266-Arduino makes more sense - the same way Teensy works, just bung it in the hardware/ directory.

[igrr]

Board manager package has been released.