Skip to content

Commit f7cb4f8

Browse files
ivanovmgivanovmg
committed
Merge branch 'master' into feature/series-info
2 parents e9c5220 + d378852 commit f7cb4f8

File tree

431 files changed

+11940
-12248
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

431 files changed

+11940
-12248
lines changed

‎.github/workflows/autoupdate-pre-commit-config.yml

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name: "Update pre-commit config"
2+
3+
on:
4+
schedule:
5+
- cron: "0 7 * * 1" # At 07:00 on each Monday.
6+
workflow_dispatch:
7+
8+
jobs:
9+
update-pre-commit:
10+
if: github.repository_owner == 'pandas-dev'
11+
name: Autoupdate pre-commit config
12+
runs-on: ubuntu-latest
13+
steps:
14+
- name: Set up Python
15+
uses: actions/setup-python@v2
16+
- name: Cache multiple paths
17+
uses: actions/cache@v2
18+
with:
19+
path: |
20+
~/.cache/pre-commit
21+
~/.cache/pip
22+
key: pre-commit-autoupdate-${{ runner.os }}-build
23+
- name: Update pre-commit config packages
24+
uses: technote-space/create-pr-action@v2
25+
with:
26+
GITHUB_TOKEN: ${{ secrets.ACTION_TRIGGER_TOKEN }}
27+
EXECUTE_COMMANDS: |
28+
pip install pre-commit
29+
pre-commit autoupdate || (exit 0);
30+
pre-commit run -a || (exit 0);
31+
COMMIT_MESSAGE: "⬆️ UPGRADE: Autoupdate pre-commit config"
32+
PR_BRANCH_NAME: "pre-commit-config-update-${PR_ID}"
33+
PR_TITLE: "⬆️ UPGRADE: Autoupdate pre-commit config"

‎.pre-commit-config.yaml

+42-22
Original file line numberDiff line numberDiff line change
@@ -15,33 +15,36 @@ repos:
1515
- id: flake8
1616
name: flake8 (cython template)
1717
files: \.pxi\.in$
18-
types:
19-
- file
18+
types: [text]
2019
args: [--append-config=flake8/cython-template.cfg]
2120
- repo: https://github.com/PyCQA/isort
22-
rev: 5.6.3
21+
rev: 5.6.4
2322
hooks:
2423
- id: isort
2524
name: isort (python)
2625
- id: isort
2726
name: isort (cython)
2827
types: [cython]
2928
- repo: https://github.com/asottile/pyupgrade
30-
rev: v2.7.2
29+
rev: v2.7.3
3130
hooks:
3231
- id: pyupgrade
3332
args: [--py37-plus]
3433
- repo: https://github.com/pre-commit/pygrep-hooks
35-
rev: v1.6.0
34+
rev: v1.7.0
3635
hooks:
3736
- id: rst-backticks
37+
- id: rst-directive-colons
38+
types: [text]
39+
- id: rst-inline-touching-normal
40+
types: [text]
3841
- repo: local
3942
hooks:
4043
- id: pip_to_conda
4144
name: Generate pip dependency from conda
4245
description: This hook checks if the conda environment.yml and requirements-dev.txt are equal
4346
language: python
44-
entry: python -m scripts.generate_pip_deps_from_conda
47+
entry: python scripts/generate_pip_deps_from_conda.py
4548
files: ^(environment.yml|requirements-dev.txt)$
4649
pass_filenames: false
4750
additional_dependencies: [pyyaml]
@@ -53,19 +56,6 @@ repos:
5356
types: [rst]
5457
args: [--filename=*.rst]
5558
additional_dependencies: [flake8-rst==0.7.0, flake8==3.7.9]
56-
- id: incorrect-sphinx-directives
57-
name: Check for incorrect Sphinx directives
58-
language: pygrep
59-
entry: |
60-
(?x)
61-
# Check for cases of e.g. .. warning: instead of .. warning::
62-
\.\.\ (
63-
autosummary|contents|currentmodule|deprecated|
64-
function|image|important|include|ipython|literalinclude|
65-
math|module|note|raw|seealso|toctree|versionadded|
66-
versionchanged|warning
67-
):[^:]
68-
files: \.(py|pyx|rst)$
6959
- id: non-standard-imports
7060
name: Check for non-standard imports
7161
language: pygrep
@@ -76,9 +66,17 @@ repos:
7666
from\ pandas\.core\ import\ common|
7767
7868
# Check for imports from collections.abc instead of `from collections import abc`
79-
from\ collections\.abc\ import|
69+
from\ collections\.abc\ import
8070
81-
from\ numpy\ import\ nan
71+
- id: non-standard-numpy.random-related-imports
72+
name: Check for non-standard numpy.random-related imports excluding pandas/_testing.py
73+
language: pygrep
74+
exclude: pandas/_testing.py
75+
entry: |
76+
(?x)
77+
# Check for imports from np.random.<method> instead of `from numpy import random` or `from numpy.random import <method>`
78+
from\ numpy\ import\ random|
79+
from\ numpy.random\ import
8280
types: [python]
8381
- id: non-standard-imports-in-tests
8482
name: Check for non-standard imports in test suite
@@ -99,12 +97,34 @@ repos:
9997
language: pygrep
10098
entry: (\.\. code-block ::|\.\. ipython ::)
10199
files: \.(py|pyx|rst)$
100+
- id: unwanted-patterns-strings-to-concatenate
101+
name: Check for use of not concatenated strings
102+
language: python
103+
entry: python scripts/validate_unwanted_patterns.py --validation-type="strings_to_concatenate"
104+
files: \.(py|pyx|pxd|pxi)$
105+
- id: unwanted-patterns-strings-with-wrong-placed-whitespace
106+
name: Check for strings with wrong placed spaces
107+
language: python
108+
entry: python scripts/validate_unwanted_patterns.py --validation-type="strings_with_wrong_placed_whitespace"
109+
files: \.(py|pyx|pxd|pxi)$
110+
- id: unwanted-patterns-private-import-across-module
111+
name: Check for import of private attributes across modules
112+
language: python
113+
entry: python scripts/validate_unwanted_patterns.py --validation-type="private_import_across_module"
114+
types: [python]
115+
exclude: ^(asv_bench|pandas/tests|doc)/
116+
- id: unwanted-patterns-private-function-across-module
117+
name: Check for use of private functions across modules
118+
language: python
119+
entry: python scripts/validate_unwanted_patterns.py --validation-type="private_function_across_module"
120+
types: [python]
121+
exclude: ^(asv_bench|pandas/tests|doc)/
102122
- repo: https://github.com/asottile/yesqa
103123
rev: v1.2.2
104124
hooks:
105125
- id: yesqa
106126
- repo: https://github.com/pre-commit/pre-commit-hooks
107-
rev: v3.2.0
127+
rev: v3.3.0
108128
hooks:
109129
- id: end-of-file-fixer
110130
exclude: ^LICENSES/|\.(html|csv|txt|svg|py)$

‎Makefile

+2-2
Original file line numberDiff line numberDiff line change
@@ -30,11 +30,11 @@ check:
3030
python3 scripts/validate_unwanted_patterns.py \
3131
--validation-type="private_function_across_module" \
3232
--included-file-extensions="py" \
33-
--excluded-file-paths=pandas/tests,asv_bench/,pandas/_vendored \
33+
--excluded-file-paths=pandas/tests,asv_bench/ \
3434
pandas/
3535

3636
python3 scripts/validate_unwanted_patterns.py \
3737
--validation-type="private_import_across_module" \
3838
--included-file-extensions="py" \
39-
--excluded-file-paths=pandas/tests,asv_bench/,pandas/_vendored,doc/
39+
--excluded-file-paths=pandas/tests,asv_bench/,doc/
4040
pandas/

‎asv_bench/benchmarks/strings.py

+17-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
import numpy as np
44

5-
from pandas import DataFrame, Series
5+
from pandas import Categorical, DataFrame, Series
66

77
from .pandas_vb_common import tm
88

@@ -16,6 +16,10 @@ def setup(self, dtype):
1616
self.series_arr = tm.rands_array(nchars=10, size=10 ** 5)
1717
self.frame_arr = self.series_arr.reshape((50_000, 2)).copy()
1818

19+
# GH37371. Testing construction of string series/frames from ExtensionArrays
20+
self.series_cat_arr = Categorical(self.series_arr)
21+
self.frame_cat_arr = Categorical(self.frame_arr)
22+
1923
def time_series_construction(self, dtype):
2024
Series(self.series_arr, dtype=dtype)
2125

@@ -28,6 +32,18 @@ def time_frame_construction(self, dtype):
2832
def peakmem_frame_construction(self, dtype):
2933
DataFrame(self.frame_arr, dtype=dtype)
3034

35+
def time_cat_series_construction(self, dtype):
36+
Series(self.series_cat_arr, dtype=dtype)
37+
38+
def peakmem_cat_series_construction(self, dtype):
39+
Series(self.series_cat_arr, dtype=dtype)
40+
41+
def time_cat_frame_construction(self, dtype):
42+
DataFrame(self.frame_cat_arr, dtype=dtype)
43+
44+
def peakmem_cat_frame_construction(self, dtype):
45+
DataFrame(self.frame_cat_arr, dtype=dtype)
46+
3147

3248
class Methods:
3349
def setup(self):

‎azure-pipelines.yml

+25
Original file line numberDiff line numberDiff line change
@@ -26,3 +26,28 @@ jobs:
2626
parameters:
2727
name: Windows
2828
vmImage: vs2017-win2016
29+
30+
- job: py37_32bit
31+
pool:
32+
vmImage: ubuntu-18.04
33+
34+
steps:
35+
- script: |
36+
docker pull quay.io/pypa/manylinux2014_i686
37+
docker run -v $(pwd):/pandas quay.io/pypa/manylinux2014_i686 \
38+
/bin/bash -xc "cd pandas && \
39+
/opt/python/cp37-cp37m/bin/python -m venv ~/virtualenvs/pandas-dev && \
40+
. ~/virtualenvs/pandas-dev/bin/activate && \
41+
python -m pip install --no-deps -U pip wheel setuptools && \
42+
pip install cython numpy python-dateutil pytz pytest pytest-xdist hypothesis pytest-azurepipelines && \
43+
python setup.py build_ext -q -i -j2 && \
44+
python -m pip install --no-build-isolation -e . && \
45+
pytest -m 'not slow and not network and not clipboard' pandas --junitxml=test-data.xml"
46+
displayName: 'Run 32-bit manylinux2014 Docker Build / Tests'
47+
48+
- task: PublishTestResults@2
49+
condition: succeededOrFailed()
50+
inputs:
51+
testResultsFiles: '**/test-*.xml'
52+
failTaskOnFailedTests: true
53+
testRunTitle: 'Publish test results for Python 3.7-32 bit full Linux'

‎ci/azure/windows.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ jobs:
1616
py38_np18:
1717
ENV_FILE: ci/deps/azure-windows-38.yaml
1818
CONDA_PY: "38"
19-
PATTERN: "not slow and not network"
19+
PATTERN: "not slow and not network and not high_memory"
2020

2121
steps:
2222
- powershell: |

‎ci/code_checks.sh

+3-35
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,8 @@ function invgrep {
3838
}
3939

4040
function check_namespace {
41-
local -r CLASS="${1}"
42-
grep -R -l --include "*.py" " ${CLASS}(" pandas/tests | xargs grep -n "pd\.${CLASS}("
41+
local -r CLASS=${1}
42+
grep -R -l --include "*.py" " ${CLASS}(" pandas/tests | xargs grep -n "pd\.${CLASS}[(\.]"
4343
test $? -gt 0
4444
}
4545

@@ -73,38 +73,6 @@ if [[ -z "$CHECK" || "$CHECK" == "lint" ]]; then
7373
cpplint --quiet --extensions=c,h --headers=h --recursive --filter=-readability/casting,-runtime/int,-build/include_subdir pandas/_libs/src/*.h pandas/_libs/src/parser pandas/_libs/ujson pandas/_libs/tslibs/src/datetime pandas/_libs/*.cpp
7474
RET=$(($RET + $?)) ; echo $MSG "DONE"
7575

76-
MSG='Check for use of not concatenated strings' ; echo $MSG
77-
if [[ "$GITHUB_ACTIONS" == "true" ]]; then
78-
$BASE_DIR/scripts/validate_unwanted_patterns.py --validation-type="strings_to_concatenate" --format="##[error]{source_path}:{line_number}:{msg}" .
79-
else
80-
$BASE_DIR/scripts/validate_unwanted_patterns.py --validation-type="strings_to_concatenate" .
81-
fi
82-
RET=$(($RET + $?)) ; echo $MSG "DONE"
83-
84-
MSG='Check for strings with wrong placed spaces' ; echo $MSG
85-
if [[ "$GITHUB_ACTIONS" == "true" ]]; then
86-
$BASE_DIR/scripts/validate_unwanted_patterns.py --validation-type="strings_with_wrong_placed_whitespace" --format="##[error]{source_path}:{line_number}:{msg}" .
87-
else
88-
$BASE_DIR/scripts/validate_unwanted_patterns.py --validation-type="strings_with_wrong_placed_whitespace" .
89-
fi
90-
RET=$(($RET + $?)) ; echo $MSG "DONE"
91-
92-
MSG='Check for import of private attributes across modules' ; echo $MSG
93-
if [[ "$GITHUB_ACTIONS" == "true" ]]; then
94-
$BASE_DIR/scripts/validate_unwanted_patterns.py --validation-type="private_import_across_module" --included-file-extensions="py" --excluded-file-paths=pandas/tests,asv_bench/,pandas/_vendored --format="##[error]{source_path}:{line_number}:{msg}" pandas/
95-
else
96-
$BASE_DIR/scripts/validate_unwanted_patterns.py --validation-type="private_import_across_module" --included-file-extensions="py" --excluded-file-paths=pandas/tests,asv_bench/,pandas/_vendored pandas/
97-
fi
98-
RET=$(($RET + $?)) ; echo $MSG "DONE"
99-
100-
MSG='Check for use of private functions across modules' ; echo $MSG
101-
if [[ "$GITHUB_ACTIONS" == "true" ]]; then
102-
$BASE_DIR/scripts/validate_unwanted_patterns.py --validation-type="private_function_across_module" --included-file-extensions="py" --excluded-file-paths=pandas/tests,asv_bench/,pandas/_vendored,doc/ --format="##[error]{source_path}:{line_number}:{msg}" pandas/
103-
else
104-
$BASE_DIR/scripts/validate_unwanted_patterns.py --validation-type="private_function_across_module" --included-file-extensions="py" --excluded-file-paths=pandas/tests,asv_bench/,pandas/_vendored,doc/ pandas/
105-
fi
106-
RET=$(($RET + $?)) ; echo $MSG "DONE"
107-
10876
fi
10977

11078
### PATTERNS ###
@@ -178,7 +146,7 @@ if [[ -z "$CHECK" || "$CHECK" == "patterns" ]]; then
178146
RET=$(($RET + $?)) ; echo $MSG "DONE"
179147

180148
MSG='Check for inconsistent use of pandas namespace in tests' ; echo $MSG
181-
for class in "Series" "DataFrame" "Index"; do
149+
for class in "Series" "DataFrame" "Index" "MultiIndex" "Timestamp" "Timedelta" "TimedeltaIndex" "DatetimeIndex" "Categorical"; do
182150
check_namespace ${class}
183151
RET=$(($RET + $?))
184152
done

‎ci/run_tests.sh

+6
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,12 @@ fi
2222

2323
PYTEST_CMD="${XVFB}pytest -m \"$PATTERN\" -n $PYTEST_WORKERS --dist=loadfile -s --strict --durations=30 --junitxml=test-data.xml $TEST_ARGS $COVERAGE pandas"
2424

25+
if [[ $(uname) != "Linux" && $(uname) != "Darwin" ]]; then
26+
# GH#37455 windows py38 build appears to be running out of memory
27+
# skip collection of window tests
28+
PYTEST_CMD="$PYTEST_CMD --ignore=pandas/tests/window/"
29+
fi
30+
2531
echo $PYTEST_CMD
2632
sh -c "$PYTEST_CMD"
2733

‎doc/redirects.csv

+10
Original file line numberDiff line numberDiff line change
@@ -542,7 +542,9 @@ generated/pandas.DatetimeIndex.date,../reference/api/pandas.DatetimeIndex.date
542542
generated/pandas.DatetimeIndex.day,../reference/api/pandas.DatetimeIndex.day
543543
generated/pandas.DatetimeIndex.day_name,../reference/api/pandas.DatetimeIndex.day_name
544544
generated/pandas.DatetimeIndex.dayofweek,../reference/api/pandas.DatetimeIndex.dayofweek
545+
generated/pandas.DatetimeIndex.day_of_week,../reference/api/pandas.DatetimeIndex.day_of_week
545546
generated/pandas.DatetimeIndex.dayofyear,../reference/api/pandas.DatetimeIndex.dayofyear
547+
generated/pandas.DatetimeIndex.day_of_year,../reference/api/pandas.DatetimeIndex.day_of_year
546548
generated/pandas.DatetimeIndex.floor,../reference/api/pandas.DatetimeIndex.floor
547549
generated/pandas.DatetimeIndex.freq,../reference/api/pandas.DatetimeIndex.freq
548550
generated/pandas.DatetimeIndex.freqstr,../reference/api/pandas.DatetimeIndex.freqstr
@@ -839,7 +841,9 @@ generated/pandas.option_context,../reference/api/pandas.option_context
839841
generated/pandas.Period.asfreq,../reference/api/pandas.Period.asfreq
840842
generated/pandas.Period.day,../reference/api/pandas.Period.day
841843
generated/pandas.Period.dayofweek,../reference/api/pandas.Period.dayofweek
844+
generated/pandas.Period.day_of_week,../reference/api/pandas.Period.day_of_week
842845
generated/pandas.Period.dayofyear,../reference/api/pandas.Period.dayofyear
846+
generated/pandas.Period.day_of_year,../reference/api/pandas.Period.day_of_year
843847
generated/pandas.Period.days_in_month,../reference/api/pandas.Period.days_in_month
844848
generated/pandas.Period.daysinmonth,../reference/api/pandas.Period.daysinmonth
845849
generated/pandas.Period.end_time,../reference/api/pandas.Period.end_time
@@ -850,7 +854,9 @@ generated/pandas.Period,../reference/api/pandas.Period
850854
generated/pandas.PeriodIndex.asfreq,../reference/api/pandas.PeriodIndex.asfreq
851855
generated/pandas.PeriodIndex.day,../reference/api/pandas.PeriodIndex.day
852856
generated/pandas.PeriodIndex.dayofweek,../reference/api/pandas.PeriodIndex.dayofweek
857+
generated/pandas.PeriodIndex.day_of_week,../reference/api/pandas.PeriodIndex.day_of_week
853858
generated/pandas.PeriodIndex.dayofyear,../reference/api/pandas.PeriodIndex.dayofyear
859+
generated/pandas.PeriodIndex.day_of_year,../reference/api/pandas.PeriodIndex.day_of_year
854860
generated/pandas.PeriodIndex.days_in_month,../reference/api/pandas.PeriodIndex.days_in_month
855861
generated/pandas.PeriodIndex.daysinmonth,../reference/api/pandas.PeriodIndex.daysinmonth
856862
generated/pandas.PeriodIndex.end_time,../reference/api/pandas.PeriodIndex.end_time
@@ -993,7 +999,9 @@ generated/pandas.Series.dt.date,../reference/api/pandas.Series.dt.date
993999
generated/pandas.Series.dt.day,../reference/api/pandas.Series.dt.day
9941000
generated/pandas.Series.dt.day_name,../reference/api/pandas.Series.dt.day_name
9951001
generated/pandas.Series.dt.dayofweek,../reference/api/pandas.Series.dt.dayofweek
1002+
generated/pandas.Series.dt.day_of_week,../reference/api/pandas.Series.dt.day_of_week
9961003
generated/pandas.Series.dt.dayofyear,../reference/api/pandas.Series.dt.dayofyear
1004+
generated/pandas.Series.dt.day_of_year,../reference/api/pandas.Series.dt.day_of_year
9971005
generated/pandas.Series.dt.days,../reference/api/pandas.Series.dt.days
9981006
generated/pandas.Series.dt.days_in_month,../reference/api/pandas.Series.dt.days_in_month
9991007
generated/pandas.Series.dt.daysinmonth,../reference/api/pandas.Series.dt.daysinmonth
@@ -1326,7 +1334,9 @@ generated/pandas.Timestamp.date,../reference/api/pandas.Timestamp.date
13261334
generated/pandas.Timestamp.day,../reference/api/pandas.Timestamp.day
13271335
generated/pandas.Timestamp.day_name,../reference/api/pandas.Timestamp.day_name
13281336
generated/pandas.Timestamp.dayofweek,../reference/api/pandas.Timestamp.dayofweek
1337+
generated/pandas.Timestamp.day_of_week,../reference/api/pandas.Timestamp.day_of_week
13291338
generated/pandas.Timestamp.dayofyear,../reference/api/pandas.Timestamp.dayofyear
1339+
generated/pandas.Timestamp.day_of_year,../reference/api/pandas.Timestamp.day_of_year
13301340
generated/pandas.Timestamp.days_in_month,../reference/api/pandas.Timestamp.days_in_month
13311341
generated/pandas.Timestamp.daysinmonth,../reference/api/pandas.Timestamp.daysinmonth
13321342
generated/pandas.Timestamp.dst,../reference/api/pandas.Timestamp.dst

‎doc/source/development/contributing.rst

+9-3
Original file line numberDiff line numberDiff line change
@@ -598,7 +598,7 @@ Building master branch documentation
598598

599599
When pull requests are merged into the pandas ``master`` branch, the main parts of
600600
the documentation are also built by Travis-CI. These docs are then hosted `here
601-
<https://dev.pandas.io>`__, see also
601+
<https://pandas.pydata.org/docs/dev/>`__, see also
602602
the :ref:`Continuous Integration <contributing.ci>` section.
603603

604604
.. _contributing.code:
@@ -837,8 +837,14 @@ to run its checks by running::
837837

838838
without having to have done ``pre-commit install`` beforehand.
839839

840-
Note that if you have conflicting installations of ``virtualenv``, then you may get an
841-
error - see `here <https://github.com/pypa/virtualenv/issues/1875>`_.
840+
.. note::
841+
842+
If you have conflicting installations of ``virtualenv``, then you may get an
843+
error - see `here <https://github.com/pypa/virtualenv/issues/1875>`_.
844+
845+
Also, due to a `bug in virtualenv <https://github.com/pypa/virtualenv/issues/1986>`_,
846+
you may run into issues if you're using conda. To solve this, you can downgrade
847+
``virtualenv`` to version ``20.0.33``.
842848

843849
Backwards compatibility
844850
~~~~~~~~~~~~~~~~~~~~~~~

0 commit comments

Comments
 (0)