Buildfixes: Better linux packaging and https #5438

NicoHood · 2016-10-03T22:10:37Z

This PR fixes several build bugs or adds missing features that are essential when packaging arduino for a linux distribution. More details can be found here #5379 (comment) and here #5437

The edits are small, please look at the separate commits.

The build can be called like this as an example with all new features:

ant clean build -Dlight_bundle=true -Dno_docs=true -Dlocal_sources=true

…ded sources

cmaglie · 2016-10-19T15:15:58Z

build/build.xml

+      <antcall target="untar-unzip-download-web" />
+      <antcall target="untar-unzip-download-local" />
+  </target>
+  <target name="untar-unzip-download-web" unless="local_sources">


Is this distinction really needed?

AFAIR if you copy the needed archives in the right place before launching the build, the file is not downloaded again (this is done in the untar-unzip-check that is a dependency of untar-unzip-download).

One of the biggest reasons I remember is that some paths are different for different architectures or not existant (mkdir -p). On top of that you need to always change the PKGBUILD and need to carefully check where the build script wants to have its downloaded files. This makes it harder to maintain.

This patch makes it simpler to search the download files from a local source instead.

cmaglie · 2016-10-19T15:24:02Z

build/build.xml

@@ -220,7 +220,8 @@
  </target>

  <!-- copy library folder -->
-  <target name="assemble-libraries" unless="light_bundle">
+  <target name="assemble-libraries" depends="assemble-libraries-full, assemble-libraries-light" />
+  <target name="assemble-libraries-full" unless="light_bundle">


I guees this is to keep the empty libraries folder. Is it really needed?

cmaglie · 2016-10-19T15:26:00Z

build/build.xml

    <copy todir="${target.path}/hardware">
      <fileset dir="../hardware">
        <exclude name="arduino/sam/**"/>
      </fileset>
    </copy>
  </target>
+  <target name="assemble-hardware-light" if="light_bundle">
+    <copy file="../hardware/package_index_bundled.json" todir="${target.path}/hardware" />


In this case the file ../hardware/package_index_bundled.json is wrong, it should be a json with zero platforms defined. I don't know maybe the best thing to do is to add a check to the IDE...

I had a lot of problems with this file. Currently I add a copy fo the avr core to the hardware path with a different name and patches to use the archlinux tools. However If i define this package in this json the IDE always crashs on a name not equal to "arduino". I am not sure if the file is required in this case.

If it should be empty, maybe you should add a stub with no data in it instead. However this works for now.

package_index_bundled.json is, basically, a core-package json that describe what's bundled inside the IDE hardware folder. The IDE merges the content of package_index_bundled.json with the general package_index.json that is downloaded from arduino.cc. The bundled index has priority over the downloaded index.

If you want to make a real no-bundle (or light) version of the IDE, the correct thing to do is to create an empty (== with zero platforms defined) package_index_bundled.json.

NicoHood · 2016-10-30T23:39:51Z

Any problems left with the PR? I'd also like to address some more build changes, but possibly in another PR then.

NicoHood · 2016-11-24T16:28:32Z

Another release has been tagged and still no progress in this PR. It makes it complicated to package arduino if those patches are not applied. Could you please consider to add this?

facchinm · 2016-11-24T16:34:45Z

1.6.13 has been a bugfix release, so we tried to include the minimum number of changes. We are currently merging a lot of pending PRs and we'll surely analyze this in the next days.

NicoHood · 2016-11-24T18:08:49Z

Thank you for the information.

If you consider to fix the suggested build improvements you might also want to think of adding GPG signatures: #5619

I am about to put arduino into [community] of ArchLinux, but I hesitate to do that because of the missing gpg checks and also because there was no clear statement if we are allowed to do so. I see no point why we should not, but in any case, feel free to contact me for making a clear statement about yes or no. The ArchLinux team would be happy about some more clear statements and gpg checks. Thanks.

NicoHood · 2016-11-26T17:33:20Z

I've resolved the merging conflicts. Please consider to merge this, as it is a real struggle to patch back the sources with those patches.

erlanger · 2016-11-26T21:16:57Z

@facchinm yes, could you plese include this patch in the next release? It makes it difficult for us linux users who have to wait longer for a new release (until the patches are merged manually).

Tante grazie!!

NicoHood · 2016-12-23T17:06:08Z

@facchinm any news about this? Another required fix would be to exclude arduino-builder from the build, ~~but i first wanted this pr to be reviewed and merged~~ (Edit: added to this pr, was simpler than i though). Can we see this soon please, as i need to create patches for every release?

Please run the builder on this pr as I cannot test the mac builds

NicoHood · 2016-12-23T22:34:11Z

The arduino-builder download contains a file platform.txt and platform.keys.rewrite.txt that needs to be placed inside the hardware folder. This should be part of the IDE and not the arduino-builder download. I try to exclude arduino-builder from the package and those 2 files are missing.

But ctags is also missing then. I built it myself and included it as several package, so people who exclude arduino-builder should be also aware that ctags is missing. I also do not understand why ctags is packaged with arduino-builder and not as separate download.

cmaglie · 2016-12-27T15:16:34Z

@NicoHood
I've pushed into master the first three commits about:

Https downloads
No Docs bundled (no_docs option)
Archives downloaded outside build folder (local_sources option)

can you rebase this PR on top of master so we can continue discussion on the other points?

…no-builder" This reverts commit 4f41384.

NicoHood · 2016-12-27T15:47:10Z

@cmaglie Thanks a lot! I rebased my PR.

And I've also removed the platform.txt again. A user pointed me to the information that it needs to be packaged with arduino-builder instead. It would be good if those 2 file would be inside the arduino-builder github repository though. Dont know where they are currently coming from in the zip download.

So left in my pr is the arduino-builder, hardware and libraries exclusion. I also want to exclude liblistserial and libastyle so that every dependency is built from source for arduino on archlinux. This makes it simpler and safer for us to link against shared libraries at build time.

It would be nice if you can link against those binaries in a simpler way. I am also not sure if it will work to replace those liblistserial etc because of the shared libraries used.

cmaglie · 2016-12-27T16:44:29Z

I rebased my PR.

mmh not exactly, you merged the master branch in your PR, but this leaves a lot of useless history (various upstream merge, reverts, and the commits that I have already cherry picked duplicated in your branch).

I suggest to experiment with interactive rebase if you didn't already: https://www.atlassian.com/git/tutorials/rewriting-history/git-rebase

once rebased you can force push your branch and get a clean history with only the meaningful commits.

cmaglie · 2016-12-27T17:06:24Z

And I've also removed the platform.txt again. A user pointed me to the information that it needs to be packaged with arduino-builder instead. It would be good if those 2 file would be inside the arduino-builder github repository though. Dont know where they are currently coming from in the zip download.

Those file are here:
https://github.com/arduino/arduino-builder/tree/master/src/arduino.cc/builder/hardware

cmaglie · 2016-12-29T14:09:10Z

build/build.xml

+      <param name="final_folder" value="${staging_folder}/arduino-builder-${platform}/arduino-builder" />
+      <param name="dest_folder" value="${staging_folder}/arduino-builder-${platform}" />
+    </antcall>
+    <copy file="${staging_folder}/arduino-builder-${platform}/arduino-builder" tofile="linux/work/arduino-builder" />


This is wrong because the file is always copied into tofile="linux/work/arduino-builder" while it should be something like ${staging_folder}/work/arduino-builder. BTW this is still not correct for macosx, probably the best way is to use ${staging_folder}/work/${staging_hardware_folder}/../arduino-builder for everything.

cmaglie · 2016-12-29T14:12:43Z

build/build.xml

+    <delete dir="${staging_folder}/arduino-builder-${platform}" includeemptydirs="true"/>
+    <mkdir dir="${staging_folder}/arduino-builder-${platform}"/>
+    <antcall target="untar">
+      <param name="archive_file" value="./arduino-builder-${platform}-${ARDUINO-BUILDER-VERSION}.tar.bz2" />


For linuxarm the filename is arduino-builder-linuxarm-1.3.23.tar.bz2, but the file is actually called arduino-builder-arm-1.3.23.tar.bz2. I'm going to copy arduino-builder-arm-1.3.23.tar.bz2 to arduino-builder-linuxarm-1.3.23.tar.bz2 on the download server (and change the build script on jenkins to output the archive with linuxarm for the new releases).

This allows also conditional build of arduino-builder. See arduino#5438

cmaglie · 2016-12-29T14:47:14Z

Superseded by #5778

This allows also conditional build of arduino-builder. See #5438

NicoHood added 4 commits October 4, 2016 00:02

Add option to disable doc packaging

5de6c1f

Fix light_bundle package option

b637e38

Added option local_sources for better linux packaging with predownloa…

4cee989

…ded sources

Use https in build downloads

f224a39

cmaglie self-assigned this Oct 18, 2016

cmaglie reviewed Oct 19, 2016

View reviewed changes

Merge remote-tracking branch 'upstream/master' into buildfix

58b8b3f

NicoHood mentioned this pull request Nov 26, 2016

https downloads #5437

Open

NicoHood added 2 commits December 23, 2016 22:41

Merge remote-tracking branch 'upstream/master' into buildfix

11e8091

Added build option for disabling arduino-builder

f29abf8

Added platform.txt files back to hardware folder. Fixes arduino-builder

4f41384

NicoHood added 2 commits December 27, 2016 16:40

Merge remote-tracking branch 'upstream/master' into buildfix

054bc50

Revert "Added platform.txt files back to hardware folder. Fixes ardui…

35e5f33

…no-builder" This reverts commit 4f41384.

cmaglie reviewed Dec 29, 2016

View reviewed changes

cmaglie added a commit to cmaglie/Arduino that referenced this pull request Dec 29, 2016

Factored out arduino-builder-build task in build.xml

838b52e

This allows also conditional build of arduino-builder. See arduino#5438

cmaglie mentioned this pull request Dec 29, 2016

Various improvements in build script #5778

Merged

cmaglie closed this Dec 29, 2016

cmaglie added a commit that referenced this pull request Jan 25, 2017

Factored out arduino-builder-build task in build.xml

3d0a153

This allows also conditional build of arduino-builder. See #5438

tlk mentioned this pull request Feb 20, 2020

MrBean version HuayraLinux/pkg-arduino#4

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Buildfixes: Better linux packaging and https #5438

Buildfixes: Better linux packaging and https #5438

NicoHood commented Oct 3, 2016 •

edited

Loading

cmaglie Oct 19, 2016

NicoHood Oct 20, 2016

cmaglie Oct 19, 2016

cmaglie Oct 19, 2016

NicoHood Oct 20, 2016

cmaglie Dec 27, 2016

NicoHood commented Oct 30, 2016

NicoHood commented Nov 24, 2016

facchinm commented Nov 24, 2016

NicoHood commented Nov 24, 2016

NicoHood commented Nov 26, 2016

erlanger commented Nov 26, 2016

NicoHood commented Dec 23, 2016 •

edited

Loading

NicoHood commented Dec 23, 2016 •

edited

Loading

cmaglie commented Dec 27, 2016

NicoHood commented Dec 27, 2016 •

edited

Loading

cmaglie commented Dec 27, 2016

cmaglie commented Dec 27, 2016

cmaglie Dec 29, 2016

cmaglie Dec 29, 2016

cmaglie commented Dec 29, 2016

Buildfixes: Better linux packaging and https #5438

Buildfixes: Better linux packaging and https #5438

Conversation

NicoHood commented Oct 3, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NicoHood commented Oct 30, 2016

NicoHood commented Nov 24, 2016

facchinm commented Nov 24, 2016

NicoHood commented Nov 24, 2016

NicoHood commented Nov 26, 2016

erlanger commented Nov 26, 2016

NicoHood commented Dec 23, 2016 • edited Loading

NicoHood commented Dec 23, 2016 • edited Loading

cmaglie commented Dec 27, 2016

NicoHood commented Dec 27, 2016 • edited Loading

cmaglie commented Dec 27, 2016

cmaglie commented Dec 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmaglie commented Dec 29, 2016

NicoHood commented Oct 3, 2016 •

edited

Loading

NicoHood commented Dec 23, 2016 •

edited

Loading

NicoHood commented Dec 23, 2016 •

edited

Loading

NicoHood commented Dec 27, 2016 •

edited

Loading