Removing hardware folder contents: platforms are downloaded at build time

[ffissore]

Platform github projects must be created once this pull req is merged

[ffissore]

This PR is an extension of this one other #1806

[damellis]

BTW, can we do this (are we doing this) in a way that preserves history on those files as they move to separate repositories?

[ffissore]

Indeed that's the idea: once we agree at merging this, we'll create two repositories with the relative files for the sam and avr platforms

[ffissore]

Ah sorry you meant porting their history to the other repos? Uhm I'm not sure we can do that, it's probably a snapshot of their current state and a new history

[ffissore]

Maybe this will help: http://stackoverflow.com/questions/10524578/how-to-move-a-file-from-one-git-repository-to-another-while-preserving-history

I'll try it asap

[damellis]

I think it would be a real shame to lose that history (or, at least, to make it much harder to find). It looks like there are relatively straightforward ways to do this. See, for example:

http://stackoverflow.com/questions/359424/detach-subdirectory-into-separate-git-repository/17864475#17864475
https://help.github.com/articles/splitting-a-subpath-out-into-a-new-repository

[damellis]

Cool, thanks!

[ffissore]

This is what I came up with, starting from current ide-1.5.x branch
https://github.com/ffissore/core-avr
https://github.com/ffissore/core-sam

I've used this script to produce them https://gist.github.com/ffissore/8639728

If you're fine with it, I will spread the word to the developers mailing list.

[damellis]

I've mentioned this to Cristian, but I think we should call these "platforms" or "architectures" rather than "cores". We currently use core to refer to the "cores/arduino" sub-directory of these new repositories, so calling the repository "core" seems confusing and ambiguous. What do you think?

Also, the history only seems to go back as far as the 1.5.x IDE (when these files were moved into a sub-directory). Do you have a sense of what's happening to the history before that?

[ffissore]

Regarding the name, my fault: I've named the zip files platform-* but forgot to do the same for the repos
https://github.com/ffissore/platform-avr
https://github.com/ffissore/platform-sam

Regarding the history: yes, it goes back until 2011. I think its because git applies a folder filter, thus previous folders are ignored. I'll try using multiple folders, specifying the old one, not sure if it will work

[ffissore]

Indeed specifying multiple dirs doesn't work and it won't suffice anyway since the folders have changed name from the first days (once it was only "hardware" then hardware/avr and hardware/sam then hardware/arduino/avr...)

The only other alternative I was able to set up is to have a repo with both avr and sam
https://github.com/ffissore/platforms-avr-sam
(this is the script https://gist.github.com/ffissore/8666298)
This solves the problem with repo history but still bundles the two platforms into one repo, which was something I was trying to avoid, although my opinion is now weaker.

[damellis]

Hmm, I feel like it's useful to keep the history accessible via the commit
log of the current repositories, but I realize that having everything in
one repo isn't ideal. What if you duplicate the platforms-avr-sam
repository and delete avr from one and sam from the other? You'd have a
couple of extra sub-directories at the root, but maybe that's okay? (Or you
could then move the files to the root?)

On Tue, Jan 28, 2014 at 7:04 AM, Federico Fissore
[email protected]:

Indeed specifying multiple dirs doesn't work and it won't suffice anyway
since the folders have changed name from the first days (once it was only
"hardware" then hardware/avr and hardware/sam then hardware/arduino/avr...)

The only other alternative I was able to set up is to have a repo with
both avr and sam
https://github.com/ffissore/platforms-avr-sam
(this is the script https://gist.github.com/ffissore/8666298)
This solves the problem with repo history but still bundles the two
platforms into one repo, which was something I was trying to avoid,
although my opinion is now weaker.

Reply to this email directly or view it on GitHubhttps://github.com//pull/1823#issuecomment-33472512
.

[matthijskooijman]

Just a quick notice to prevent double work: I've figured out a way to keep history, while also splitting into two repositories. The history is quite complicated, so I'm still ironing out some final details, I'll comment with more details in a few hours when I have it completely working.

[matthijskooijman]

I dug a bit into the git filter-branch manpage and found a way to make this work. By doing a two-step filtering, I managed to first sort out all the renaming stuff (geez, the hardware repository has been reorganized a lot) by separating out all platform code into an /avr and /sam directory (outside of the hardware folder for convenience, and without any extra "arduino" layer in between). As a second step, you use the regular subdirectory-filter to pick out these subfolders into two separate repositories.

For the first step, I used some index manipulation based on an example from the git filter-branch manpage, which applies the same trick to be able to apply sed and grep on the list of filenames in a commit. This first step also drops hardware/tools entirely, since I don't think that should belong in the new repo? Also, hardware/atmel/sam is dropped as well, since looking at the history, that code was a dead branch that got removed in 170870d. Finally, it seems that two versions of the sam platform co-existed at some point in the repository (merged in 2583daa). I put the "google" variant into a subdirectory of the avr repository to handle this.

Here's the script I used, and the repositories I generated :

https://gist.github.com/matthijskooijman/8673439
https://github.com/matthijskooijman/arduino-platform-avr
https://github.com/matthijskooijman/arduino-platform-sam

To double-check that I didn't leave extra files cluttering around, I ran the command git log --pretty=format: --name-only | sort |uniq|less in each of these two repositories to get the list of filenames that appear in all revisions in the entire history. No funny stuff shows up (anymore).

The resulting repositories now have history up to 2011 for sam and back to 2005 for avr (that's the first commit in the original repository).

[ffissore]

Impressive work!
Out of curiosity: what's the purpose of ( git ls-files --error-unmatch core/global.h > /dev/null 2> /dev/null && sed "s-\\(\\t\"*\\)core/-\1avr/-" || cat ) ?

[matthijskooijman]

Oh, I was planning to explain that, but I was in a bit of a hurry and forgot :-)

In the early commits, there are platform files inside core/, which I wanted to include to get the complete history. However, there is another branch of development that imports a version of the Processing IDE, which has java files inside the core/ directory. To get the former, but not the latter, I added this check to only keep the core/ directory when a file called global.h is present.

Having said and done all this, I just realized another argument for keeping both the avr and sam bits in the same repositories: It makes development easier. In particular, when a change affects both avr and sam (e.g., added API that needs to be implemented in both), you can then put them in a single pullrequest. Having two repositories increases the overhead wrt to submitting, reviewing and merging changes. Additionally, for API changes this allows adding the API to all platforms in a single commit, whereas otherwise there is additional risk of the APIs for different platforms to fall out of sync.

Perhaps a way of looking at this would be that there is the Arduino "provider" (I don't want to say "core", but perhaps "provider" isn't the best term either...) that supports the "avr" and "sam" platforms, so a single repository for the entire Arduino "provider" could make sense.

[matthijskooijman]

But of course my script would still be useful for a single repository, since it nicely collects all history for either platform in the proper subdirectory. It will need a few tweaks, of course.

[matthijskooijman]

My previous comments were all regarding the creating of the new repositories. I also have a few things to say about the actual commits in this pullrequest :-)

It seems that most of these commits belong in #1806, so I'll limit myself to talking about the latter two.

• A nitpick about the commit message "Removing hardware/arduino folders: platforms are now downloaded at build time". The platforms aren't actually downloaded at build time yet, this happens in the next commit. Also, for the final merge I'd include the urls to the new repositories in the commit message, to help anyone tracing around history in the future.
• The zip files you created contain an arduino/avr or arduino/sam directory, which again contains the contents of both repositories. It seems to me like these directory names are currently not defined anywhere inside a repository and thus require a manual step when generating the zip files. Also, it would make more sense to me when the zip files would contain exactly the repository contents. Or it would make sense when the zip file would contain a single top-level directory, whose name is not actually relevant, like some_name-version as is common with tarballs (with the added advantage that the zipfiles can be generated by or even downloaded directly from github). The latter is a bit more work in the unpack rule though, since it should unpack to some temporary location, confirm that there is only a single top-level directory and then move the contents of that directory but not the directory itself (this might be useful for other (third-party) tarballs as well, though.
• Currently, it seems the IDE does not make it easy to do development on the platform files. AFAIU every time you change something, you have to re-run "ant build" to copy the change into (in my case) linux/work/hardware to have it take effect (I tried to improve this in matthijskooijman@cfd8472 using a build option). I believe that these commits will further complicate matters, since now you'd have to create a zipfile with your changes and replace the checksum file as well, or copy the changes manually (having them overwritten on the next ant build), etc. Perhaps this is a good moment to add better support for this usecase (for example by detecting a .git directory inside linux/work/hardware/arduino/avr and if present, skip unpacking the zipfile?). Also, I'm wondering how your workspace is set up so that this problem doesn't bite you (assuming you'd have it fixed already if you were facing the same issue).

[ffissore]

Agree with the argument, not to mention copying issues filed on the IDE repo but belonging to one or both of these new repos.
Agree with the message: should have read "will be downloaded".
The zip files were created before setting up the repos, that's why they are messed up. A final release will have them properly formatted.
Main purpose of this pull req is to provide me those zip files as they are going to be included in the Arduino Tre.
Since the IDE is taking the shape of a distribution of tools rather than a single software, it made sense to me to try to separate the pieces and see if they could still be recombined. Good ideas and insights usually come up from such attempts.
Regarding the "zip and chechsum", I haven't yet taken it into account but I would follow the lead of some build systems (gentoo IIRC): the zip files would be unpacked only if the target folder was missing or its files were all older the the zip file itself. Thus, if you modify one file, it will have a more recent modification date, the zip file will be considered "old" and ignored.

[ffissore]

I suppose we could also use things like git submodule or something, but I've read sad/angry comments about it, although I haven't really read them

[ffissore]

I've force pushed this branch (yep, sorry but it wasn't merging anymore) with a proposal to the "zip and checksum" issue.
That's what you should do to setup a dev env

cd ~ #or wherever you wish to start
mkdir -p hardare/arduino
cd hardware/arduino
git clone [email protected]:matthijskooijman/arduino-platform-avr.git avr
git clone [email protected]:matthijskooijman/arduino-platform-sam.git sam
cd -
cd Arduino/build
ant -DLOCAL_HARDWARE_FOLDER=~/hardware clean run

The IDE will NOT download the zip files and will instead use the files from the cloned repos.
This all is to avoid using both git submodule and git subtree: the former for the issues specified here [0], the latter because it requires additional setup and slightly changes the way you use git

[0] http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/

[ffissore]

For the record, doing this is not mandatory. If you want to develop the core files of the IDE (WString, Print and the like) this is a suggested method. If you are an Arduino library developer, these instructions are not for you, you just don't need them.

[damellis]

Hmm, I'm sure git submodules and subtrees have their quirks, but having everyone manually check out multiple repositories seems even worse. (Also, then you lose information about which version of the platform repositories goes with which version of the main repository.) Are submodules and subtrees really that bad?

[ffissore]

Regarding which versions goes in, we could use git tags (releases) and ship them. Also consider that this multi repo setup is true only for core developers. A library dev won't need to bother.

Submodules seem bad because even an experienced user may lose track of her changes and even wipe them out, throwing her work away. Subtrees seem better but still requires the user to use git in a slightly different way: more commands specific to managing the subtrees, something git GUIs may not support. Moreover, git subtree is not enabled by default: command line linux versions need to run this [0], brew and windows look fine.

[0] http://cogumbreiro.blogspot.it/2013/05/how-to-install-git-subtree-in-ubuntu.html

[ffissore]

This PR is old and no longer maintained. The gathered knowledge will be used for future improvements. For now, it's better to close it as it's piling up the PR stack

[ffissore]

Branch moved on https://github.com/ffissore/Arduino/tree/ide-1.5.x-avr-toolchain-separate-platforms