Merge remote-tracking branch 'upstream/main' into implementation-pdep-4

Author: MarcoGorelli

asv_bench/benchmarks/multiindex_object.py

@@ -239,10 +239,11 @@ class SetOperations:
         ("monotonic", "non_monotonic"),
         ("datetime", "int", "string", "ea_int"),
         ("intersection", "union", "symmetric_difference"),
+        (False, None),
     ]
-    param_names = ["index_structure", "dtype", "method"]
+    param_names = ["index_structure", "dtype", "method", "sort"]
 
-    def setup(self, index_structure, dtype, method):
+    def setup(self, index_structure, dtype, method, sort):
         N = 10**5
         level1 = range(1000)
 
@@ -272,8 +273,8 @@ def setup(self, index_structure, dtype, method):
         self.left = data[dtype]["left"]
         self.right = data[dtype]["right"]
 
-    def time_operation(self, index_structure, dtype, method):
-        getattr(self.left, method)(self.right)
+    def time_operation(self, index_structure, dtype, method, sort):
+        getattr(self.left, method)(self.right, sort=sort)
 
 
 class Difference:

doc/redirects.csv

@@ -45,6 +45,7 @@ contributing_docstring,development/contributing_docstring
 developer,development/developer
 extending,development/extending
 internals,development/internals
+development/meeting,community
 
 # api moved function
 reference/api/pandas.io.json.json_normalize,pandas.json_normalize

doc/source/development/community.rst

@@ -0,0 +1,117 @@
+.. _community:
+
+=====================
+Contributor community
+=====================
+
+pandas is a community-driven open source project developed by a large group
+of `contributors <https://github.com/pandas-dev/pandas/graphs/contributors>`_
+and a smaller group of `maintainers <https://pandas.pydata.org/about/team.html>`_.
+The pandas leadership has made a strong commitment to creating an open,
+inclusive, and positive community. Please read the pandas `Code of Conduct
+<https://pandas.pydata.org/community/coc.html>`_ for guidance on how to
+interact with others in a way that makes the community thrive.
+
+We offer several meetings and communication channels to share knowledge and
+connect with others within the pandas community.
+
+Community meeting
+-----------------
+
+The pandas Community Meeting is a regular sync meeting for the project's
+maintainers which is open to the community. Everyone is welcome to attend and
+contribute to conversations.
+
+The meetings take place on the second Wednesday of each month at 18:00 UTC.
+
+The minutes of past meetings are available in `this Google Document <https://docs.google.com/document/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-licoBmBU/edit?usp=sharing>`__.
+
+
+New contributor meeting
+-----------------------
+
+On the third Wednesday of the month, we hold meetings to welcome and support
+new contributors in our community.
+
+| 👋 you all are invited
+| 💬 everyone can present (add yourself to the hackMD agenda)
+| 👀 anyone can sit in and listen
+
+Attendees are new and experienced contributors, as well as a few maintainers.
+We aim to answer questions about getting started, or help with work in
+progress when possible, as well as get to know each other and share our
+learnings and experiences.
+
+The agenda for the next meeting and minutes of past meetings are available in
+`this HackMD <https://hackmd.io/@pandas-dev/HJgQt1Tei>`__.
+
+Calendar
+--------
+
+This calendar shows all the community meetings. Our community meetings are
+ideal for anyone wanting to contribute to pandas, or just curious to know how
+current development is going.
+
+.. raw:: html
+
+   <iframe src="https://calendar.google.com/calendar/embed?src=pgbn14p6poja8a1cf2dv2jhrmg%40group.calendar.google.com" style="border: 0" width="800" height="600" frameborder="0" scrolling="no"></iframe>
+
+You can subscribe to this calendar with the following links:
+
+* `iCal <https://calendar.google.com/calendar/ical/pgbn14p6poja8a1cf2dv2jhrmg%40group.calendar.google.com/public/basic.ics>`__
+* `Google calendar <https://calendar.google.com/calendar/[email protected]>`__
+
+Additionally, we'll sometimes have one-off meetings on specific topics.
+These will be published on the same calendar.
+
+`GitHub issue tracker <https://github.com/pandas-dev/pandas/issues>`_
+----------------------------------------------------------------------
+
+The pandas contributor community conducts conversations mainly via this channel.
+Any community member can open issues to:
+
+- Report bugs, e.g. "I noticed the behavior of a certain function is
+  incorrect"
+- Request features, e.g. "I would like this error message to be more readable"
+- Request documentation improvements, e.g. "I found this section unclear"
+- Ask questions, e.g. "I noticed the behavior of a certain function
+  changed between versions. Is this expected?".
+
+    Ideally your questions should be related to how pandas work rather
+    than how you use pandas. `StackOverflow <https://stackoverflow.com/>`_ is
+    better suited for answering usage questions, and we ask that all usage
+    questions are first asked on StackOverflow. Thank you for respecting are
+    time and wishes. 🙇
+
+Maintainers and frequent contributors might also open issues to discuss the
+ongoing development of the project. For example:
+
+- Report issues with the CI, GitHub Actions, or the performance of pandas
+- Open issues relating to the internals
+- Start roadmap discussion aligning on proposals what to do in future
+  releases or changes to the API.
+- Open issues relating to the project's website, logo, or governance
+
+The developer mailing list
+--------------------------
+
+The pandas mailing list `[email protected] <mailto://pandas-dev@python
+.org>`_ is used for long form
+conversations and to engages people in the wider community who might not
+be active on the issue tracker but we would like to include in discussions.
+
+Community slack
+---------------
+
+We have a chat platform for contributors, maintainers and potential
+contributors. This is not a space for user questions, rather for questions about
+contributing to pandas. The slack is a private space, specifically meant for
+people who are hesitant to bring up their questions or ideas on a large public
+mailing list or GitHub.
+
+If this sounds like the right place for you, you are welcome to join! Email us
+at `[email protected] <mailto://[email protected]>`_ and let us
+know that you read and agree to our `Code of Conduct <https://pandas.pydata.org/community/coc.html>`_
+😉 to get an invite. And please remember the slack is not meant to replace the
+mailing list or issue tracker - all important announcements and conversations
+should still happen there.

doc/source/development/contributing_codebase.rst

@@ -139,7 +139,7 @@ Otherwise, you need to do it manually:
         warnings.warn(
             'Use new_func instead.',
             FutureWarning,
-            stacklevel=find_stack_level(inspect.currentframe()),
+            stacklevel=find_stack_level(),
         )
         new_func()
 
@@ -790,14 +790,11 @@ Or with one of the following constructs::
     pytest pandas/tests/[test-module].py::[TestClass]
     pytest pandas/tests/[test-module].py::[TestClass]::[test_method]
 
-Using `pytest-xdist <https://pypi.org/project/pytest-xdist>`_, one can
-speed up local testing on multicore machines. To use this feature, you will
-need to install ``pytest-xdist`` via::
-
-    pip install pytest-xdist
-
-The ``-n`` flag then can be specified when running ``pytest`` to parallelize a test run
-across the number of specified cores or ``auto`` to utilize all the available cores on your machine.
+Using `pytest-xdist <https://pypi.org/project/pytest-xdist>`_, which is
+included in our 'pandas-dev' environment, one can speed up local testing on
+multicore machines. The ``-n`` number flag then can be specified when running
+pytest to parallelize a test run across the number of specified cores or auto to
+utilize all the available cores on your machine.
 
 .. code-block:: bash
 
@@ -807,8 +804,57 @@ across the number of specified cores or ``auto`` to utilize all the available co
    # Utilizes all available cores
    pytest -n auto pandas
 
-This can significantly reduce the time it takes to locally run tests before
-submitting a pull request.
+If you'd like to speed things along further a more advanced use of this
+command would look like this
+
+.. code-block:: bash
+
+    pytest pandas -n 4 -m "not slow and not network and not db and not single_cpu" -r sxX
+
+In addition to the multithreaded performance increase this improves test
+speed by skipping some tests using the ``-m`` mark flag:
+
+- slow: any test taking long (think seconds rather than milliseconds)
+- network: tests requiring network connectivity
+- db: tests requiring a database (mysql or postgres)
+- single_cpu: tests that should run on a single cpu only
+
+You might want to enable the following option if it's relevant for you:
+
+- arm_slow: any test taking long on arm64 architecture
+
+These markers are defined `in this toml file <https://github.com/pandas-dev/pandas/blob/main/pyproject.toml>`_
+, under ``[tool.pytest.ini_options]`` in a list called ``markers``, in case
+you want to check if new ones have been created which are of interest to you.
+
+The ``-r`` report flag will display a short summary info (see `pytest
+documentation <https://docs.pytest.org/en/4.6.x/usage.html#detailed-summary-report>`_)
+. Here we are displaying the number of:
+
+- s: skipped tests
+- x: xfailed tests
+- X: xpassed tests
+
+The summary is optional and can be removed if you don't need the added
+information. Using the parallelization option can significantly reduce the
+time it takes to locally run tests before submitting a pull request.
+
+If you require assistance with the results,
+which has happened in the past, please set a seed before running the command
+and opening a bug report, that way we can reproduce it. Here's an example
+for setting a seed on windows
+
+.. code-block:: bash
+
+    set PYTHONHASHSEED=314159265
+    pytest pandas -n 4 -m "not slow and not network and not db and not single_cpu" -r sxX
+
+On Unix use
+
+.. code-block:: bash
+
+    export PYTHONHASHSEED=314159265
+    pytest pandas -n 4 -m "not slow and not network and not db and not single_cpu" -r sxX
 
 For more, see the `pytest <https://docs.pytest.org/en/latest/>`_ documentation.
 

doc/source/development/index.rst

@@ -23,4 +23,4 @@ Development
     developer
     policies
     roadmap
-    meeting
+    community

doc/source/development/meeting.rst

@@ -21,10 +21,6 @@ libraries, we care about the following things:
 This page is also here to offer a bit of a translation guide for users of these
 R packages.
 
-For transfer of ``DataFrame`` objects from pandas to R, one option is to
-use HDF5 files, see :ref:`io.external_compatibility` for an
-example.
-
 
 Quick reference
 ---------------

doc/source/getting_started/comparison/comparison_with_r.rst

@@ -5245,99 +5245,6 @@ You could inadvertently turn an actual ``nan`` value into a missing value.
    store.append("dfss2", dfss, nan_rep="_nan_")
    store.select("dfss2")
 
-.. _io.external_compatibility:
-
-External compatibility
-''''''''''''''''''''''
-
-``HDFStore`` writes ``table`` format objects in specific formats suitable for
-producing loss-less round trips to pandas objects. For external
-compatibility, ``HDFStore`` can read native ``PyTables`` format
-tables.
-
-It is possible to write an ``HDFStore`` object that can easily be imported into ``R`` using the
-``rhdf5`` library (`Package website`_). Create a table format store like this:
-
-.. _package website: https://www.bioconductor.org/packages/release/bioc/html/rhdf5.html
-
-.. ipython:: python
-
-   df_for_r = pd.DataFrame(
-       {
-           "first": np.random.rand(100),
-           "second": np.random.rand(100),
-           "class": np.random.randint(0, 2, (100,)),
-       },
-       index=range(100),
-   )
-   df_for_r.head()
-
-   store_export = pd.HDFStore("export.h5")
-   store_export.append("df_for_r", df_for_r, data_columns=df_dc.columns)
-   store_export
-
-.. ipython:: python
-   :suppress:
-
-   store_export.close()
-   os.remove("export.h5")
-
-In R this file can be read into a ``data.frame`` object using the ``rhdf5``
-library. The following example function reads the corresponding column names
-and data values from the values and assembles them into a ``data.frame``:
-
-.. code-block:: R
-
-   # Load values and column names for all datasets from corresponding nodes and
-   # insert them into one data.frame object.
-
-   library(rhdf5)
-
-   loadhdf5data <- function(h5File) {
-
-   listing <- h5ls(h5File)
-   # Find all data nodes, values are stored in *_values and corresponding column
-   # titles in *_items
-   data_nodes <- grep("_values", listing$name)
-   name_nodes <- grep("_items", listing$name)
-   data_paths = paste(listing$group[data_nodes], listing$name[data_nodes], sep = "/")
-   name_paths = paste(listing$group[name_nodes], listing$name[name_nodes], sep = "/")
-   columns = list()
-   for (idx in seq(data_paths)) {
-     # NOTE: matrices returned by h5read have to be transposed to obtain
-     # required Fortran order!
-     data <- data.frame(t(h5read(h5File, data_paths[idx])))
-     names <- t(h5read(h5File, name_paths[idx]))
-     entry <- data.frame(data)
-     colnames(entry) <- names
-     columns <- append(columns, entry)
-   }
-
-   data <- data.frame(columns)
-
-   return(data)
-   }
-
-Now you can import the ``DataFrame`` into R:
-
-.. code-block:: R
-
-   > data = loadhdf5data("transfer.hdf5")
-   > head(data)
-            first    second class
-   1 0.4170220047 0.3266449     0
-   2 0.7203244934 0.5270581     0
-   3 0.0001143748 0.8859421     1
-   4 0.3023325726 0.3572698     1
-   5 0.1467558908 0.9085352     1
-   6 0.0923385948 0.6233601     1
-
-.. note::
-   The R function lists the entire HDF5 file's contents and assembles the
-   ``data.frame`` object from all matching nodes, so use this only as a
-   starting point if you have stored multiple ``DataFrame`` objects to a
-   single HDF5 file.
-
 
 Performance
 '''''''''''