Skip to content

CI: clean environment.yml, comment that req-dev is generated #30392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Dec 25, 2019
Merged

CI: clean environment.yml, comment that req-dev is generated #30392

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
171ae7e
Organize environment.yml deps, add comment to auto-generated file
jbrockmendel Dec 21, 2019
2ca88b0
Merge branch 'master' of https://github.com/pandas-dev/pandas into ci…
jbrockmendel Dec 21, 2019
e342fe1
cleanup env
jbrockmendel Dec 21, 2019
979a598
update requirements-dev.txt
jbrockmendel Dec 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 19 additions & 10 deletions environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ dependencies:
- nbconvert>=5.4.1
- nbsphinx
- pandoc
# Dask and its dependencies

# Dask and its dependencies (that dont install with dask)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC conda install dask does the right thing without needing to specify everything else, but pip install dask does not. could we unify all this as dask[dataframe]? also i guess put it in the "downstream tests" block?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger might have ideas

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dask[dataframe] works for installing from PyPI, but there isn't a dask[dataframe] or dask-dataframe conda package.

- dask-core
- toolz>=0.7.3
- fsspec>=0.5.1
Expand All @@ -54,6 +55,8 @@ dependencies:
- pytest>=5.0.1
- pytest-cov
- pytest-xdist>=1.21

# downstream tests
- seaborn
- statsmodels

Expand All @@ -74,22 +77,28 @@ dependencies:
- scipy>=1.1

# optional for io
- beautifulsoup4>=4.6.0 # pandas.read_html
# ---------------
# pd.read_html
- beautifulsoup4>=4.6.0
- html5lib
- lxml

# pd.read_excel, DataFrame.to_excel, pd.ExcelWriter, pd.ExcelFile
- openpyxl<=3.0.1
- xlrd
- xlsxwriter
- xlwt
- odfpy

- fastparquet>=0.3.2 # pandas.read_parquet, DataFrame.to_parquet
- html5lib # pandas.read_html
- lxml # pandas.read_html
- openpyxl<=3.0.1 # pandas.read_excel, DataFrame.to_excel, pandas.ExcelWriter, pandas.ExcelFile
- pyarrow>=0.13.1 # pandas.read_parquet, DataFrame.to_parquet, pandas.read_feather, DataFrame.to_feather
- python-snappy # required by pyarrow

- pyqt>=5.9.2 # pandas.read_clipboard
- pytables>=3.4.2 # pandas.read_hdf, DataFrame.to_hdf
- python-snappy # required by pyarrow
- s3fs # pandas.read_csv... when using 's3://...' path
- sqlalchemy # pandas.read_sql, DataFrame.to_sql
- xarray # DataFrame.to_xarray
- xlrd # pandas.read_excel, DataFrame.to_excel, pandas.ExcelWriter, pandas.ExcelFile
- xlsxwriter # pandas.read_excel, DataFrame.to_excel, pandas.ExcelWriter, pandas.ExcelFile
- xlwt # pandas.read_excel, DataFrame.to_excel, pandas.ExcelWriter, pandas.ExcelFile
- odfpy # pandas.read_excel
- pyreadstat # pandas.read_spss
- pip:
- git+https://github.com/pandas-dev/pandas-sphinx-theme.git@master
15 changes: 9 additions & 6 deletions requirements-dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# This file is auto-generated from environment.yml, do not modify.
# See that file for comments about the need/usage of each depdendency.

numpy>=1.15
python-dateutil>=2.6.1
pytz
Expand Down Expand Up @@ -48,20 +51,20 @@ matplotlib>=2.2.2
numexpr>=2.6.8
scipy>=1.1
beautifulsoup4>=4.6.0
fastparquet>=0.3.2
html5lib
lxml
openpyxl<=3.0.1
xlrd
xlsxwriter
xlwt
odfpy
fastparquet>=0.3.2
pyarrow>=0.13.1
python-snappy
pyqt5>=5.9.2
tables>=3.4.2
python-snappy
s3fs
sqlalchemy
xarray
xlrd
xlsxwriter
xlwt
odfpy
pyreadstat
git+https://github.com/pandas-dev/pandas-sphinx-theme.git@master
9 changes: 7 additions & 2 deletions scripts/generate_pip_deps_from_conda.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,9 +87,14 @@ def main(conda_fname, pip_fname, compare=False):
elif isinstance(dep, dict) and len(dep) == 1 and "pip" in dep:
pip_deps += dep["pip"]
else:
raise ValueError("Unexpected dependency {}".format(dep))
raise ValueError(f"Unexpected dependency {dep}")

pip_content = "\n".join(pip_deps)
fname = os.path.split(conda_fname)[1]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for the split here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

otherwise we get a full path including username

header = (
f"# This file is auto-generated from {fname}, do not modify.\n"
"# See that file for comments about the need/usage of each depdendency.\n\n"
)
pip_content = header + "\n".join(pip_deps)
Comment on lines +93 to +97
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small personal preference, but I think this would be simpler if when defining pip_deps above, instead of pip_content = [] you write:

pip_content = [
    f"# This file is auto-generated from {fname}, do not modify.",
    "# See that file for comments about the need/usage of each depdendency.",
]

If you do that, all this can be deleted here (you'll have to define fname before).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ATM the pip_deps list has a specific meaning. id prefer to retain that

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not important, but headers could still be a list, and use "\n".join(headers + pip_deps). I think it'll still be an improvement, avoiding the \n in the strings. Just saying, happy to get this merge as is.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would actually write this as a triple quoted string.

header = f"""
...
..."""

which avoids the embedded newlines

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

triple quoted strings that arent at the module-level mess with code-folding, so i prefer to avoid them

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k that's what i would have done, but ok as is.


if compare:
with open(pip_fname) as pip_fd:
Expand Down