CI: Use conda compiler in `environment.yml` #27062

marcelotrevisani · 2019-06-26T18:50:11Z

Improvement to the conda environment to use the compiler from the virtual env instead of the system

Using the cxx-compile would be better because it will use the compiler from the virtual environment and it will not rely on the system anymore. That would be great to also avoid any kind of abi problems if someone uses some old compiler

closes #xxxx
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

…tual env instead of the system

jreback · 2019-06-26T18:57:51Z

Makefile

@@ -10,7 +10,7 @@ clean_pyc:
 	-find . -name '*.py[co]' -exec rm {} \;

 build: clean_pyc
-	python setup.py build_ext --inplace
+	python setup.py build_ext --inplace --compiler=$CC


this is not used in our CI at all, though I do use it locally, so nice to improve this generally.

We should add some documentation (in contributing.rst) that things exist & work (not 100% sure this actually works on windows; the Makefile)

That is true, unless pandas is using mingw it will not going to use it, indeed
but this part I am just enforcing it, because cython usually already uses the $CC

marcelotrevisani · 2019-06-26T19:03:55Z

#27061 should be merged first, after that I can rebase from main

marcelotrevisani · 2019-06-26T19:09:46Z

For example, it will use the compiler from the conda virtual environment

/home/trevisani/miniconda3/envs/pandas-dev/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -fPIC -DNPY_NO_DEPRECATED_API=0 -I/home/trevisani/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/numpy/core/include -I/home/trevisani/miniconda3/envs/pandas-dev/include/python3.7m -c pandas/util/move.c -o build/temp.linux-x86_64-3.7/pandas/util/move.o -Wno-unused-function
x86_64-conda_cos6-linux-gnu-gcc -pthread -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/trevisani/miniconda3/envs/pandas-dev/lib -L/home/trevisani/miniconda3/envs/pandas-dev/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/trevisani/miniconda3/envs/pandas-dev/lib -L/home/trevisani/miniconda3/envs/pandas-dev/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 build/temp.linux-x86_64-3.7/pandas/util/move.o -o /home/trevisani/pandas/pandas/pandas/util/_move.cpython-37m-x86_64-linux-gnu.so

That would be multiplatform as well, in that way it will isolate the compilation process from the system

As you can see it is using this compiler /home/trevisani/miniconda3/envs/pandas-dev/bin/x86_64-conda_cos6-linux-gnu-cc

marcelotrevisani · 2019-06-26T19:22:33Z

Still need some work, the CI is getting the compiler from the system, at least for osx

codecov · 2019-06-26T19:22:47Z

Codecov Report

Merging #27062 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #27062      +/-   ##
==========================================
- Coverage   92.04%   92.03%   -0.01%     
==========================================
  Files         180      180              
  Lines       50713    50713              
==========================================
- Hits        46680    46676       -4     
- Misses       4033     4037       +4

Flag	Coverage Δ
#multiple	`90.68% <ø> (ø)`	⬆️
#single	`41.86% <ø> (-0.09%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	`88.88% <0%> (-11.12%)`	⬇️
pandas/core/frame.py	`96.89% <0%> (-0.12%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ff50b46...d810dda. Read the comment docs.

codecov · 2019-06-26T19:22:49Z

Codecov Report

Merging #27062 into master will decrease coverage by 1.19%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #27062     +/-   ##
=========================================
- Coverage   92.98%   91.78%   -1.2%     
=========================================
  Files         184      180      -4     
  Lines       50362    50934    +572     
=========================================
- Hits        46827    46752     -75     
- Misses       3535     4182    +647

Flag	Coverage Δ
#multiple	`90.48% <ø> (-1.17%)`	⬇️
#single	`42.01% <ø> (-0.48%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	`88.88% <0%> (-11.12%)`	⬇️
pandas/util/_decorators.py	`92.13% <0%> (-3.22%)`	⬇️
pandas/core/dtypes/cast.py	`91.05% <0%> (-2.01%)`	⬇️
pandas/core/indexing.py	`93% <0%> (-1.82%)`	⬇️
pandas/core/dtypes/common.py	`96.35% <0%> (-1.44%)`	⬇️
pandas/core/series.py	`92.5% <0%> (-1.17%)`	⬇️
pandas/core/groupby/ops.py	`96% <0%> (-0.92%)`	⬇️
pandas/core/tools/datetimes.py	`85.05% <0%> (-0.74%)`	⬇️
pandas/core/internals/blocks.py	`95.01% <0%> (-0.58%)`	⬇️
pandas/core/arrays/categorical.py	`95.95% <0%> (-0.51%)`	⬇️
... and 160 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 88ccb25...27fa92e. Read the comment docs.

datapythonista

I think we should test this in windows, I guess Makefile is rarely used if ever, but we want to make sure the environment can be resolved.

I'll see if I have time to add the build of the docs from windows, and then we'll have a windows build using environment.yml, but if someone can test this before manually that would be great.

datapythonista · 2019-06-27T08:07:47Z

environment.yml

@@ -79,5 +81,4 @@ dependencies:
  - xlrd  # pandas.read_excel, DataFrame.to_excel, pandas.ExcelWriter, pandas.ExcelFile
  - xlsxwriter  # pandas.read_excel, DataFrame.to_excel, pandas.ExcelWriter, pandas.ExcelFile
  - xlwt  # pandas.read_excel, DataFrame.to_excel, pandas.ExcelWriter, pandas.ExcelFile
-  - pip:
-    - pyreadstat  # pandas.read_spss
+  - pyreadstat  # pandas.read_spss


Can you revert this change here please. After that, you should run ./scripts/generate_pip_deps_from_conda.py so requirements.txt is synchronized with the changes here.

Sure, I will rebase my branch from master and that should be reverted

datapythonista · 2019-06-27T08:12:39Z

environment.yml

@@ -14,6 +14,8 @@ dependencies:

  # building
  - cython>=0.28.2
+  - cxx-compiler
+  - zlib


can you add a short description why those are needed

Sure! I will add the explanation, actually the zlib we can remove since it is already a dependency of other packages

I also think that the conda-forge channel should be prioritized by the defaults
since not all of the packages are present on the default channel and the project is using the missing packages from conda-forge, you might see some problems in the future (unlikely) but it is better to avoid any kind of problem

But I can create another PR for that

pep8speaks · 2019-07-02T07:32:54Z

Hello @marcelotrevisani! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file pandas/__init__.py:

Line 165:16: E131 continuation line unaligned for hanging indent
Line 166:28: E131 continuation line unaligned for hanging indent
Line 168:67: W292 no newline at end of file

Comment last updated at 2019-07-02 07:38:21 UTC

…he/bin/x86_64-conda_cos6-linux-gnu-cc compiler

marcelotrevisani · 2019-07-02T08:35:48Z

I will squash my commits when it is ready to be integrated.
I just found one problem with Linux, for some reason one of the builds is still using the compiler from the system. It needs more investigation

TomAugspurger · 2019-07-08T16:14:43Z

I will squash my commits when it is ready to be integrated.

No need to squash, we do that on merge.

WillAyd · 2019-08-28T17:10:16Z

@marcelotrevisani is this still active? Can you merge master?

marcelotrevisani · 2019-08-29T11:54:21Z

@marcelotrevisani is this still active? Can you merge master?

Sure, during this weekend I will have some time and I am going to do it

WillAyd · 2019-09-05T17:05:37Z

ci/setup_env.sh

+    CCACHE=ccache
+    echo "ccache: $CCACHE"
+    export CC="ccache $CC"
+elif [ -z "$NOCACHE" ] && [ "${TRAVIS_OS_NAME}" == "osx" ]; then


Can this conditional branch be combined with the one above? Seems like mostly copy / paste save one off things so would be easier to read if consolidated

jbrockmendel · 2019-09-16T17:00:36Z

@marcelotrevisani can you rebase

jreback · 2019-10-06T23:54:01Z

@marcelotrevisani we have had a fair amount of CI churn recently, so if you can merge master we can look again

WillAyd · 2019-10-11T21:54:33Z

Nice PR but I think this has gone stale. @marcelotrevisani ping if you'd like to pick this back up

Improvement to the conda environment to use the compiler from the vir…

d810dda

…tual env instead of the system

jreback reviewed Jun 26, 2019

View reviewed changes

marcelotrevisani mentioned this pull request Jun 26, 2019

Install pyreadstat using conda instead #27061

Merged

4 tasks

datapythonista requested changes Jun 27, 2019

View reviewed changes

datapythonista added CI Continuous Integration Dependencies Required and optional dependencies labels Jun 27, 2019

datapythonista changed the title ~~Improvement to the conda environment~~ CI: Use conda environment in `environment.yml1 Jun 27, 2019

datapythonista changed the title ~~CI: Use conda environment in `environment.yml1~~ CI: Use conda compiler in environment.yml Jun 27, 2019

CI Improvements to work with conda compilers

a731708

marcelotrevisani added 4 commits July 2, 2019 08:36

Revert to use source deactivate

c1b28e5

Build improvements regarding ccache

a8eabf9

Fix __init__.py

d819a72

Fix windows build and force /home/trevisani/miniconda3/envs/test-ccac…

ad4b82c

…he/bin/x86_64-conda_cos6-linux-gnu-cc compiler

marcelotrevisani added 8 commits July 2, 2019 10:06

Removed compiler option

a322411

Fix ccache command

bb5706c

Update makefile

e8d80be

reset makefile

ef24547

readd --compiler

3313c85

add --compiler to the build extension

a4136e5

Update build command

8a70b52

Setting gcc to CC

335f8bf

Forcing to use GCC env var

83e5795

marcelotrevisani added 4 commits July 22, 2019 10:12

Merge branch 'master' into conda-env

7bc4b87

Setting gcc to be CC

27fa92e

Setting gcc

97c84f8

Reset

e789326

Merge remote-tracking branch 'origin/master' into conda-env

630415a

WillAyd requested changes Sep 5, 2019

View reviewed changes

WillAyd closed this Oct 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Use conda compiler in `environment.yml` #27062

CI: Use conda compiler in `environment.yml` #27062

marcelotrevisani commented Jun 26, 2019

jreback Jun 26, 2019

marcelotrevisani Jun 26, 2019

marcelotrevisani commented Jun 26, 2019

marcelotrevisani commented Jun 26, 2019 •

edited

Loading

marcelotrevisani commented Jun 26, 2019

codecov bot commented Jun 26, 2019

codecov bot commented Jun 26, 2019 •

edited

Loading

datapythonista left a comment

datapythonista Jun 27, 2019

marcelotrevisani Jun 28, 2019 •

edited

Loading

datapythonista Jun 27, 2019

marcelotrevisani Jun 28, 2019

marcelotrevisani Jun 28, 2019

pep8speaks commented Jul 2, 2019 •

edited

Loading

marcelotrevisani commented Jul 2, 2019

TomAugspurger commented Jul 8, 2019

WillAyd commented Aug 28, 2019

marcelotrevisani commented Aug 29, 2019

WillAyd Sep 5, 2019

jbrockmendel commented Sep 16, 2019

jreback commented Oct 6, 2019

WillAyd commented Oct 11, 2019

CI: Use conda compiler in environment.yml #27062

CI: Use conda compiler in environment.yml #27062

Conversation

marcelotrevisani commented Jun 26, 2019

jreback Jun 26, 2019

Choose a reason for hiding this comment

marcelotrevisani Jun 26, 2019

Choose a reason for hiding this comment

marcelotrevisani commented Jun 26, 2019

marcelotrevisani commented Jun 26, 2019 • edited Loading

marcelotrevisani commented Jun 26, 2019

codecov bot commented Jun 26, 2019

Codecov Report

codecov bot commented Jun 26, 2019 • edited Loading

Codecov Report

datapythonista left a comment

Choose a reason for hiding this comment

datapythonista Jun 27, 2019

Choose a reason for hiding this comment

marcelotrevisani Jun 28, 2019 • edited Loading

Choose a reason for hiding this comment

datapythonista Jun 27, 2019

Choose a reason for hiding this comment

marcelotrevisani Jun 28, 2019

Choose a reason for hiding this comment

marcelotrevisani Jun 28, 2019

Choose a reason for hiding this comment

pep8speaks commented Jul 2, 2019 • edited Loading

Comment last updated at 2019-07-02 07:38:21 UTC

marcelotrevisani commented Jul 2, 2019

TomAugspurger commented Jul 8, 2019

WillAyd commented Aug 28, 2019

marcelotrevisani commented Aug 29, 2019

WillAyd Sep 5, 2019

Choose a reason for hiding this comment

jbrockmendel commented Sep 16, 2019

jreback commented Oct 6, 2019

WillAyd commented Oct 11, 2019

CI: Use conda compiler in `environment.yml` #27062

CI: Use conda compiler in `environment.yml` #27062

marcelotrevisani commented Jun 26, 2019 •

edited

Loading

codecov bot commented Jun 26, 2019 •

edited

Loading

marcelotrevisani Jun 28, 2019 •

edited

Loading

pep8speaks commented Jul 2, 2019 •

edited

Loading