Merge branch 'pandas-dev:main' into read-json-engine-argument

Author: abkosar

.github/ISSUE_TEMPLATE/feature_request.yaml

@@ -68,5 +68,5 @@ body:
     attributes:
       label: Additional Context
       description: >
-        Please provide any relevant Github issues, code examples or references that help describe and support
+        Please provide any relevant GitHub issues, code examples or references that help describe and support
         the feature request.

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,4 +1,4 @@
-- [ ] closes #xxxx (Replace xxxx with the Github issue number)
+- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
 - [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
 - [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
 - [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

.github/workflows/macos-windows.yml

@@ -1,4 +1,4 @@
-name: Windows-MacOS
+name: Windows-macOS
 
 on:
   push:

.github/workflows/python-dev.yml

@@ -3,7 +3,7 @@
 #
 # In general, this file will remain frozen(present, but not running) until:
 #    - The next unreleased Python version has released beta 1
-#      - This version should be available on Github Actions.
+#      - This version should be available on GitHub Actions.
 #    - Our required build/runtime dependencies(numpy, pytz, Cython, python-dateutil)
 #      support that unreleased Python version.
 #    To unfreeze, comment out the ``if: false`` condition, and make sure you update

.github/workflows/ubuntu.yml

@@ -31,9 +31,7 @@ jobs:
       matrix:
         env_file: [actions-38.yaml, actions-39.yaml, actions-310.yaml]
         pattern: ["not single_cpu", "single_cpu"]
-        # Don't test pyarrow v2/3: Causes timeouts in read_csv engine
-        # even if tests are skipped/xfailed
-        pyarrow_version: ["5", "6", "7"]
+        pyarrow_version: ["7", "8", "9"]
         include:
           - name: "Downstream Compat"
             env_file: actions-38-downstream_compat.yaml
@@ -77,7 +75,6 @@ jobs:
           - name: "Numpy Dev"
             env_file: actions-310-numpydev.yaml
             pattern: "not slow and not network and not single_cpu"
-            pandas_testing_mode: "deprecate"
             test_args: "-W error::DeprecationWarning:numpy -W error::FutureWarning:numpy"
         exclude:
           - env_file: actions-39.yaml
@@ -96,7 +93,6 @@ jobs:
       EXTRA_APT: ${{ matrix.extra_apt || '' }}
       LANG: ${{ matrix.lang || '' }}
       LC_ALL: ${{ matrix.lc_all || '' }}
-      PANDAS_TESTING_MODE: ${{ matrix.pandas_testing_mode || '' }}
       PANDAS_DATA_MANAGER: ${{ matrix.pandas_data_manager || 'block' }}
       PANDAS_COPY_ON_WRITE: ${{ matrix.pandas_copy_on_write || '0' }}
       TEST_ARGS: ${{ matrix.test_args || '' }}

.github/workflows/wheels.yml

@@ -44,7 +44,7 @@ jobs:
       # Ensure that a wheel builder finishes even if another fails
       fail-fast: false
       matrix:
-        # Github Actions doesn't support pairing matrix values together, let's improvise
+        # GitHub Actions doesn't support pairing matrix values together, let's improvise
         # https://github.com/github/feedback/discussions/7835#discussioncomment-1769026
         buildplat:
         - [ubuntu-20.04, manylinux_x86_64]

.pre-commit-config.yaml

@@ -251,7 +251,7 @@ repos:
         files: ^(ci/deps/actions-.*-minimum_versions\.yaml|pandas/compat/_optional\.py)$
     -   id: validate-errors-locations
         name: Validate errors locations
-        description: Validate errors are in approriate locations.
+        description: Validate errors are in appropriate locations.
         entry: python scripts/validate_exception_location.py
         language: python
         files: ^pandas/

README.md

@@ -13,7 +13,7 @@
 [![Coverage](https://codecov.io/github/pandas-dev/pandas/coverage.svg?branch=main)](https://codecov.io/gh/pandas-dev/pandas)
 [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/pandas-dev/pandas/badge)](https://api.securityscorecards.dev/projects/github.com/pandas-dev/pandas)
 [![Downloads](https://static.pepy.tech/personalized-badge/pandas?period=month&units=international_system&left_color=black&right_color=orange&left_text=PyPI%20downloads%20per%20month)](https://pepy.tech/project/pandas)
-[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/pydata/pandas)
+[![Slack](https://img.shields.io/badge/join_Slack-information-brightgreen.svg?logo=slack)](https://pandas.pydata.org/docs/dev/development/community.html?highlight=slack#community-slack)
 [![Powered by NumFOCUS](https://img.shields.io/badge/powered%20by-NumFOCUS-orange.svg?style=flat&colorA=E1523D&colorB=007D8A)](https://numfocus.org)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
@@ -152,7 +152,7 @@ For usage questions, the best place to go to is [StackOverflow](https://stackove
 Further, general questions and discussions can also take place on the [pydata mailing list](https://groups.google.com/forum/?fromgroups#!forum/pydata).
 
 ## Discussion and Development
-Most development discussions take place on GitHub in this repo. Further, the [pandas-dev mailing list](https://mail.python.org/mailman/listinfo/pandas-dev) can also be used for specialized discussions or design issues, and a [Gitter channel](https://gitter.im/pydata/pandas) is available for quick development related questions.
+Most development discussions take place on GitHub in this repo. Further, the [pandas-dev mailing list](https://mail.python.org/mailman/listinfo/pandas-dev) can also be used for specialized discussions or design issues, and a [Slack channel](https://pandas.pydata.org/docs/dev/development/community.html?highlight=slack#community-slack) is available for quick development related questions.
 
 ## Contributing to pandas [![Open Source Helpers](https://www.codetriage.com/pandas-dev/pandas/badges/users.svg)](https://www.codetriage.com/pandas-dev/pandas)
 

asv_bench/benchmarks/array.py

@@ -44,6 +44,24 @@ def time_from_integer_array(self):
         pd.array(self.values_integer, dtype="Int64")
 
 
+class StringArray:
+    def setup(self):
+        N = 100_000
+        values = tm.rands_array(3, N)
+        self.values_obj = np.array(values, dtype="object")
+        self.values_str = np.array(values, dtype="U")
+        self.values_list = values.tolist()
+
+    def time_from_np_object_array(self):
+        pd.array(self.values_obj, dtype="string")
+
+    def time_from_np_str_array(self):
+        pd.array(self.values_str, dtype="string")
+
+    def time_from_list(self):
+        pd.array(self.values_list, dtype="string")
+
+
 class ArrowStringArray:
 
     params = [False, True]
@@ -71,3 +89,6 @@ def time_setitem_list(self, multiple_chunks):
 
     def time_setitem_slice(self, multiple_chunks):
         self.array[::10] = "foo"
+
+    def time_tolist(self, multiple_chunks):
+        self.array.tolist()

asv_bench/benchmarks/join_merge.py

@@ -4,6 +4,7 @@
 
 from pandas import (
     DataFrame,
+    Index,
     MultiIndex,
     Series,
     array,
@@ -92,6 +93,39 @@ def time_f_ordered(self, axis, ignore_index):
         concat(self.frame_f, axis=axis, ignore_index=ignore_index)
 
 
+class ConcatIndexDtype:
+
+    params = (
+        ["datetime64[ns]", "int64", "Int64", "string[python]", "string[pyarrow]"],
+        [0, 1],
+        [True, False],
+        [True, False],
+    )
+    param_names = ["dtype", "axis", "sort", "is_monotonic"]
+
+    def setup(self, dtype, axis, sort, is_monotonic):
+        N = 10_000
+        if dtype == "datetime64[ns]":
+            vals = date_range("1970-01-01", periods=N)
+        elif dtype in ("int64", "Int64"):
+            vals = np.arange(N, dtype=np.int64)
+        elif dtype in ("string[python]", "string[pyarrow]"):
+            vals = tm.makeStringIndex(N)
+        else:
+            raise NotImplementedError
+
+        idx = Index(vals, dtype=dtype)
+        if is_monotonic:
+            idx = idx.sort_values()
+        else:
+            idx = idx[::-1]
+
+        self.series = [Series(i, idx[i:]) for i in range(5)]
+
+    def time_concat_series(self, dtype, axis, sort, is_monotonic):
+        concat(self.series, axis=axis, sort=sort)
+
+
 class Join:
 
     params = [True, False]

asv_bench/benchmarks/multiindex_object.py

@@ -239,10 +239,11 @@ class SetOperations:
         ("monotonic", "non_monotonic"),
         ("datetime", "int", "string", "ea_int"),
         ("intersection", "union", "symmetric_difference"),
+        (False, None),
     ]
-    param_names = ["index_structure", "dtype", "method"]
+    param_names = ["index_structure", "dtype", "method", "sort"]
 
-    def setup(self, index_structure, dtype, method):
+    def setup(self, index_structure, dtype, method, sort):
         N = 10**5
         level1 = range(1000)
 
@@ -272,8 +273,8 @@ def setup(self, index_structure, dtype, method):
         self.left = data[dtype]["left"]
         self.right = data[dtype]["right"]
 
-    def time_operation(self, index_structure, dtype, method):
-        getattr(self.left, method)(self.right)
+    def time_operation(self, index_structure, dtype, method, sort):
+        getattr(self.left, method)(self.right, sort=sort)
 
 
 class Difference:

ci/deps/actions-38-minimum_versions.yaml

@@ -16,7 +16,7 @@ dependencies:
   - boto3
 
   # required dependencies
-  - python-dateutil=2.8.1
+  - python-dateutil=2.8.2
   - numpy=1.20.3
   - pytz=2020.1
 
@@ -39,7 +39,7 @@ dependencies:
   - openpyxl=3.0.7
   - pandas-gbq=0.15.0
   - psycopg2=2.8.6
-  - pyarrow=1.0.1
+  - pyarrow=6.0.0
   - pymysql=1.0.2
   - pyreadstat=1.1.2
   - pytables=3.6.1

doc/_templates/pandas_footer.html

@@ -1,3 +1,3 @@
 <p class="copyright">
-    &copy {{copyright}} pandas via <a href="https://numfocus.org">NumFOCUS, Inc.</a> Hosted by <a href="https://www.ovhcloud.com">OVH Cloud</a>.
+    &copy {{copyright}} pandas via <a href="https://numfocus.org">NumFOCUS, Inc.</a> Hosted by <a href="https://www.ovhcloud.com">OVHcloud</a>.
 </p>

doc/redirects.csv

@@ -45,6 +45,7 @@ contributing_docstring,development/contributing_docstring
 developer,development/developer
 extending,development/extending
 internals,development/internals
+development/meeting,community
 
 # api moved function
 reference/api/pandas.io.json.json_normalize,pandas.json_normalize

doc/source/development/community.rst

@@ -0,0 +1,117 @@
+.. _community:
+
+=====================
+Contributor community
+=====================
+
+pandas is a community-driven open source project developed by a large group
+of `contributors <https://github.com/pandas-dev/pandas/graphs/contributors>`_
+and a smaller group of `maintainers <https://pandas.pydata.org/about/team.html>`_.
+The pandas leadership has made a strong commitment to creating an open,
+inclusive, and positive community. Please read the pandas `Code of Conduct
+<https://pandas.pydata.org/community/coc.html>`_ for guidance on how to
+interact with others in a way that makes the community thrive.
+
+We offer several meetings and communication channels to share knowledge and
+connect with others within the pandas community.
+
+Community meeting
+-----------------
+
+The pandas Community Meeting is a regular sync meeting for the project's
+maintainers which is open to the community. Everyone is welcome to attend and
+contribute to conversations.
+
+The meetings take place on the second Wednesday of each month at 18:00 UTC.
+
+The minutes of past meetings are available in `this Google Document <https://docs.google.com/document/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-licoBmBU/edit?usp=sharing>`__.
+
+
+New contributor meeting
+-----------------------
+
+On the third Wednesday of the month, we hold meetings to welcome and support
+new contributors in our community.
+
+| 👋 you all are invited
+| 💬 everyone can present (add yourself to the hackMD agenda)
+| 👀 anyone can sit in and listen
+
+Attendees are new and experienced contributors, as well as a few maintainers.
+We aim to answer questions about getting started, or help with work in
+progress when possible, as well as get to know each other and share our
+learnings and experiences.
+
+The agenda for the next meeting and minutes of past meetings are available in
+`this HackMD <https://hackmd.io/@pandas-dev/HJgQt1Tei>`__.
+
+Calendar
+--------
+
+This calendar shows all the community meetings. Our community meetings are
+ideal for anyone wanting to contribute to pandas, or just curious to know how
+current development is going.
+
+.. raw:: html
+
+   <iframe src="https://calendar.google.com/calendar/embed?src=pgbn14p6poja8a1cf2dv2jhrmg%40group.calendar.google.com" style="border: 0" width="800" height="600" frameborder="0" scrolling="no"></iframe>
+
+You can subscribe to this calendar with the following links:
+
+* `iCal <https://calendar.google.com/calendar/ical/pgbn14p6poja8a1cf2dv2jhrmg%40group.calendar.google.com/public/basic.ics>`__
+* `Google calendar <https://calendar.google.com/calendar/[email protected]>`__
+
+Additionally, we'll sometimes have one-off meetings on specific topics.
+These will be published on the same calendar.
+
+`GitHub issue tracker <https://github.com/pandas-dev/pandas/issues>`_
+----------------------------------------------------------------------
+
+The pandas contributor community conducts conversations mainly via this channel.
+Any community member can open issues to:
+
+- Report bugs, e.g. "I noticed the behavior of a certain function is
+  incorrect"
+- Request features, e.g. "I would like this error message to be more readable"
+- Request documentation improvements, e.g. "I found this section unclear"
+- Ask questions, e.g. "I noticed the behavior of a certain function
+  changed between versions. Is this expected?".
+
+    Ideally your questions should be related to how pandas work rather
+    than how you use pandas. `StackOverflow <https://stackoverflow.com/>`_ is
+    better suited for answering usage questions, and we ask that all usage
+    questions are first asked on StackOverflow. Thank you for respecting are
+    time and wishes. 🙇
+
+Maintainers and frequent contributors might also open issues to discuss the
+ongoing development of the project. For example:
+
+- Report issues with the CI, GitHub Actions, or the performance of pandas
+- Open issues relating to the internals
+- Start roadmap discussion aligning on proposals what to do in future
+  releases or changes to the API.
+- Open issues relating to the project's website, logo, or governance
+
+The developer mailing list
+--------------------------
+
+The pandas mailing list `[email protected] <mailto://pandas-dev@python
+.org>`_ is used for long form
+conversations and to engages people in the wider community who might not
+be active on the issue tracker but we would like to include in discussions.
+
+Community slack
+---------------
+
+We have a chat platform for contributors, maintainers and potential
+contributors. This is not a space for user questions, rather for questions about
+contributing to pandas. The slack is a private space, specifically meant for
+people who are hesitant to bring up their questions or ideas on a large public
+mailing list or GitHub.
+
+If this sounds like the right place for you, you are welcome to join! Email us
+at `[email protected] <mailto://[email protected]>`_ and let us
+know that you read and agree to our `Code of Conduct <https://pandas.pydata.org/community/coc.html>`_
+😉 to get an invite. And please remember the slack is not meant to replace the
+mailing list or issue tracker - all important announcements and conversations
+should still happen there.