Remove IDE code from the repository

[igrr]

This ticket should track long due repository overhaul. Since the Boards manager was introduced, it doesn't make sense to maintain the whole IDE fork in this repository. We only need the files under hardware/esp8266com/esp8266, plus some scripts to build/bundle/upload boards manager package.
ESP8266FS tool should go into a separate repository.

This cleanup will allow us to remove tags inherited from the IDE (e.g. 1.6.5) and start maintaining a sane version numbering scheme for the core alone (currently versions look like 1.6.5-1160-gef26c5f which is a shame).

There are three ways to do the cleanup.

1. Remove IDE files, move core files to new locations, and commit everything. Pros: easy to do, commit history is mostly preserved (git log --follow), rebasing of pending pull requests is straightforward. Cons: this will generate a huge diff, and repository download will get much larger.
2. Move core files to new locations and purge all commits inherited from the IDE, along with corresponding files. Pros: commit history for the esp8266 core preserved (same as with option 1), we get clean history and repository size is reduced considerably. Cons: this will mess up history, so any pending changes from pull requests will need to be applied manually. Also if some contributor is working on a large piece of code he will be rather disappointed by not being able to merge.
3. Rename repository into esp8266/old-Arduino, create new repository called esp8266/Arduino with all the core files. Pros: clean repo, small download size, history may be preserved (same as with option 2). Cons: github doesn't allow to transfer issues from one repository to another, so issues will be lost (and isn't that lovely? :) )

Personally I'm inclined to go with option 2. However this may upset people who are currently working on some pieces of code. So I'm leaving this ticket open until next week to gather comments, perhaps there is another better option which I'm missing here.

[Links2004]

I think git subtree shut do the best jop for this.

• split the ESP and SDK / tools out.
• keep the history for the new 2 gits
• remove the "Arduino history"

so we not loose any information's

in best case we can setup the gits in a way that we can check them out in to the Arduino installation or user directory for easy testing and development, without the need for triggering the build process or creating many symlinks.

may creating an new "https://github.com/esp8266/Arduino" where we add all components as sub modules as global versions history and matching, with Tags for each release and branches for stable, staging and master for development.

for more info of git subtree see:
http://stackoverflow.com/questions/359424/detach-subdirectory-into-separate-git-repository/17864475#17864475
https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt

[grahamehorner]

@igrr option 2 works for me; source for the Arduino IDE and the ESP8266 board package should IMHO be separated as suggested.

The board package should have tools and core library only; also we should consider libraries that build on top of core having a separate package and downloadable via Arduino IDE menu -> Sketch -> Include Library -> Manage Libraries ... like to board is downloadable via Tools -> Board -> Boards Manager ...

This would mean as the board package and core libraries become stable; the cadence will be reduced until additional core ESP8266 SDK feature become available; and this would not impact on any cadence of the libraries that may/or may not be as robust as the core or require change to accommodate feature enhancements.

[igrr]

Another comment regarding option 2 proposed in my first message. Since history will be rewritten using git filter-branch, things as simple as git pull will not work for anyone after the change. Perhaps adding a temporary notice to README.md describing this change and giving instructions on how to proceed will partially alleviate the pain for people who are using this project's git.

Edit: basically, most people who use git but do not contribute code to this project will have to do git reset --hard original/refs/heads/master or something similar.

[igrr]

First draft is here: https://github.com/igrr/esp8266-arduino-core
Repository size is down from 1.4G to 6M.
Things left to do:

• Add back README, LICENSE, etc.
• Add back docs/
• Add back SD and OneWire libraries which were in libraries/ directory
• Add travis CI script (old one needs major changes)
• Add bash script to build boards manager package
• Update scripts which build HTML version of docs
• Write bootstrapping script(s) which will download tools (compiler, esptool, mkspiffs) into tools directory for development workflow
• Set up branches/tags equivalent to stable/staging versions
• Back up esp8266/Arduino repo (esp8266/Arduino-old?)
• Write up a notice on the esp8266.com forum
• Push new repo to esp8266/Arduino

(Edited according to the comments below)

[grahamehorner]

@igrr good work, took less than 30seconds to clone the repo :D 💃 👍

[pgollor]

@igrr maybe this https://github.com/pgollor/esp8266-arduino-autobuilder could help for the new travis CI script???

[Links2004]

@igrr looks good
some notes:

• ./docs shut be ported may integrated in doc?
• ./libraries/OneWire has ESP changes
• ./libraries/Adafruit_ILI9341 has some but can be better pulled from: https://github.com/Links2004/Adafruit_ILI9341

what is you proposal regarding external libraries?
hope I have some time on weekend to wrote a script for the handling.

[igrr]

@pgollor Thanks for the link, will consider using arduino-builder for CI. The reason I wanted to use Arduino app itself was that we can also test downloading and installation of boards manager package this way. Or perhaps we can do both? Will think about this.

@Links2004 thanks for noting docs and OneWire! Will move those as well. Regarding Adafruit_ILI9341, i think it would be better off maintained in your repository.
Overall my opinion regarding libraries which are a) not essential (like SPI) and b) not tightly coupled with the core (like WiFi), is pretty much what @grahamehorner mentioned above — make them installable using Library Manager. See this FAQ for details. I think Adafruit_ILI9341 might be installed via Library manager. Or perhaps it's possible to upstream esp8266-specific changes? Adafruit seems to be supporting esp8266 quite well, so there's a good chance, IMO.

[igrr]

I've written a script to do all the changes on Friday without a lot of manual steps:
https://gist.github.com/igrr/6e61e11f729efa733edd
I've also written a python script to download tools (gcc, esptool, mkspiffs) when the repository is used inside hardware folder. Will upload later.

[pgollor]

@igrr You are right. It is better to use the Arduino App instead of the arduino-builder.
The Arduino app supports accessing the boards manager over the console to download other hardware stuff:
https://github.com/arduino/Arduino/blob/master/build/shared/manpage.adoc

[igrr]

Updated https://github.com/igrr/esp8266-arduino-core to include missing libraries, docs, and readme. Readme needs links to be fixed, but that may be done afterwards.

[igrr]

Mostly done, just need to add scripts to generate boards manager package.
This repository may now be used inside Arduino hardware directory.

[asetyde]

Json board manager method is the best at my opinion .

Good work

Sent from my iPhone

On 30 ott 2015, at 00:43, Ivan Grokhotkov [email protected] wrote:

Mostly done, just need to add scripts to generate boards manager package.
This repository may now be used inside Arduino hardware directory.

—
Reply to this email directly or view it on GitHub.

[pgollor]

@igrr At first: Great work :-).
With which tool do you generate the stable and staging releases?
It is better to modify your scripts instead of creating a new one, isn't it?

What we are need?

• bash script to compressing the current git code as zip and this to a tag or release?
• Python script to modify the json template file which adds the current tools version and so on

If we have fully automatic scripts we can use a CI system to generate a nightly build board manager package or so :)

I hope you are understanding my thoughts. ;-)
If you are want, I can try to create the two scripts from above.

[igrr]

Building a board manager package release used to mean the following:

1. Fix platform.txt using sed (this is required because of differences in how IDE handles tool paths for packages installed using boards manager, and packages present in hardware folder)
2. Compress all the necessary files into a zip package, calculate hash of that.
3. Create json file with correct version, link to the zip package, etc.
4. Generate HTML version of Markdown docs using jekyll
5. Upload HTML docs, badges, zip package, and json file to the server (arduino.esp8266.com).
6. Set up symlinks on server to point staging/stable build to the new package.
7. Call some webhooks to announce there is a new version available.

Steps 1—3 were handled by build/build_boards_manager_package.sh file from the old repository. I have mostly updated it to be compatible with the new directory layout yesterday.
Steps 4—7 were handled by another script which i didn't publish. I will look into updating this, perhaps we can switch to Github releases for package distribution.

Regarding nightly builds, this should be supported by Arduino IDE starting with 1.6.6:
arduino/Arduino#3449

[pgollor]

Thanks for your summary. At which point can I help?
It is helpful if we both do not do the same thing twice. :-)

I think it is better to use github releases instead of uploading it to arduino.esp8266.com because it is easier to get older releases and seeing the outline from all releases.

[pgollor]

For the documentation we can use GitHub Pages? GitHub Pages supports jekyll too.

[igrr]

The trick is that we need to maintain multiple copies of documentation, one for every version.
GitHub Pages might be a good approach because in that case it's easier to add collaborators, without having to worry about giving access to server/SSH keys/etc.
I've pushed some commits which cover points 1—3.
I can push my jekyll templates a bit later, if you figure out how to use them along with Github Releases, that would be cool.

[Links2004]

GitHub Pages are basically a branch inside the git.

example:
https://github.com/links2004/Arduino/tree/gh-pages
linked to:
http://links2004.github.io/Arduino/

[pgollor]

I will wait for your jekyll templates and after that i will do some tests with automatically creating the documentation and adding to github pages.
But the earliest date I can do this is on sunday.

[pgollor]

Github pages isn't useful to host different documentation versions. We could use gh-pages to publish the latest version.

Solutions for older releases:

• Link the Supported Hardware page and Change log page in the reference.md file.
  And in readme.md we could add a link like:
  https://github.com/esp8266/Arduino/blob/a6a0b87b215e04c0bdea02b6b413ea34975fb51d/doc/reference.md
• Using a jenkins build job on a jenkins server and create boards manager package and build documentation and everyone can access the build artifacts. Travis CI doesn't support archive build artifacts, does it?
• Add documentation as html into the boards manager package zip file and release this zip @ the Arduino repository.
• Leave documentation at arduino.esp8266.com

Do you have better ideas?

What do you think about stable and staging git tags for better versioning?

[igrr]

I think gh-pages may work perfectly fine for us. Each time we do a release, we use jekyll to build html docs and add these docs to gh-pages. Each version goes into a separate directory. On gh-pages we can maintain a menu with a list of versions pointing to these directories.

Regarding stable/staging versions, my idea is to adopt the following scheme.

• tag every released version
• mark intermediate ('staging') releases with a suffix (i.e. 2.1-dev1, 2.1-dev2, 2.1-rc1, 2.1-rc2)
• for the final release, drop the suffix (2.1)
• versions which are intermediate are added to 'staging' boards manager package json. Final releases are added to 'stable' boards manager package json.

This way we maintain a single branch, and mark releases as intermediate ('staging') or final ('stable') depending on where in development cycle we find ourselves.

@pgollor I've pushed my jekyll template to https://github.com/igrr/esp8266-arduino-docs.
Here is a (rough) bash script which i used to copy all the things in place and run jekyll: https://gist.github.com/igrr/aac32668df5c38019461.

[pgollor]

My thought was, that gh-pages will increase with every new version and thereby the hole repository will increase with redundant data. But I think that is the easiest way. :-)

@igrr Thanks for it. I will do some test with your scripts.

I have some more questions:

• Should we added the hole package manager script collection into package dir?
• What version is the repository now? 2.0? stable? staging?

[Links2004]

for the local development you not need to fetch the gh-pages branch. so the size will be not a problem.
git clone -b master --single-branch https://github.com/esp8266/Arduino.git will do the jop.
The most GUIs for git have this feature integrate too.

[Links2004]

I configured doxygen to generate a class and function overview, may its interesting to integrate this in the release management.
example:
http://links2004.github.io/Arduino/

https://gist.github.com/Links2004/e645e58248395cc6b63c

[pgollor]

I did some tests with gh-pages and here are the results:

• added a new bash script into the repository: https://github.com/pgollor/Arduino/blob/master/package/esp8266-arudino-doc.bash which handles the jekyll stuff
• add doc to gh-pages http://pgollor.github.io/Arduino/
• create bash script which do some stuff automatically and call esp8266-arudino-doc.bash https://gist.github.com/pgollor/96bc746d37a4dacaaeb3
• change jekll files a bit https://github.com/pgollor/esp8266-arduino-docs

TODO:

• Change script to add the package.zip as release file into github repository
• Host boards manager json file at gh-pages
• add static and stable link in main index html file on http://pgollor.github.io/Arduino/

[igrr]

Awsome! Looks good to me.
I have to disagree on the final three points though. My view is that package.zip should be distributed through github releases — because that's what it is. Regarding json file, we still need to fix the way it is generated. Right now we generate one new file per release, which is not what Arduino IDE expects. It expects that we would append every new release into the same package.json file. I've mostly done this using jq tool, need to tidy up and commit.
Regarding your previous question, we are currently at 1.6.5+something, and the aim is to do a 2.0.0 release when we sort out all these packaging issues.

[pgollor]

That was my fault. I wrote it not correctly, but i mean the same as you. :) The package file should be distributed via github releases. ;)

Okay, my time is very limited at this week. So should I do the last steps at the end of this week, or could you do the final steps?

[igrr]

If you can do the pull request both for shell scripts and Jekyll templates,
I'll do the rest. Thank you very much.

On Mon, Nov 2, 2015, 12:00 pgollor [email protected] wrote:

That was my fault. I wrote it not correctly, but i mean the same as you.
:) The package file should be distributed via github releases. ;)

Okay, my time is very limited at this week. So should I do the last steps
at the end of this week, or can you do the final steps?

—
Reply to this email directly or view it on GitHub
#904 (comment).

[pgollor]

Pullrequests: done
The missing bash script is here: https://gist.github.com/pgollor/96bc746d37a4dacaaeb3
This script have to be called outside the repository because it changes the git branch to gh-pages and this branch doesn't contain the files from the master branch. ;-)

I think there is a way to link some files from the master branch in the gh-pages branch, but i can't remember.

Attention:
I changed the repository links in the bash scripts. So the scripts will work again only after merging the pull requests.

@igrr By the way: Thanks four your trust to adding me as calloborator to Arduino.