Merge remote-tracking branch 'upstream/main' into implementation-pdep-4

Author: MarcoGorelli

.github/ISSUE_TEMPLATE/feature_request.yaml

@@ -68,5 +68,5 @@ body:
     attributes:
       label: Additional Context
       description: >
-        Please provide any relevant Github issues, code examples or references that help describe and support
+        Please provide any relevant GitHub issues, code examples or references that help describe and support
         the feature request.

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,4 +1,4 @@
-- [ ] closes #xxxx (Replace xxxx with the Github issue number)
+- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
 - [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
 - [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
 - [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

.github/workflows/macos-windows.yml

@@ -1,4 +1,4 @@
-name: Windows-MacOS
+name: Windows-macOS
 
 on:
   push:

.github/workflows/python-dev.yml

@@ -3,7 +3,7 @@
 #
 # In general, this file will remain frozen(present, but not running) until:
 #    - The next unreleased Python version has released beta 1
-#      - This version should be available on Github Actions.
+#      - This version should be available on GitHub Actions.
 #    - Our required build/runtime dependencies(numpy, pytz, Cython, python-dateutil)
 #      support that unreleased Python version.
 #    To unfreeze, comment out the ``if: false`` condition, and make sure you update

.github/workflows/wheels.yml

@@ -44,7 +44,7 @@ jobs:
       # Ensure that a wheel builder finishes even if another fails
       fail-fast: false
       matrix:
-        # Github Actions doesn't support pairing matrix values together, let's improvise
+        # GitHub Actions doesn't support pairing matrix values together, let's improvise
         # https://github.com/github/feedback/discussions/7835#discussioncomment-1769026
         buildplat:
         - [ubuntu-20.04, manylinux_x86_64]

.pre-commit-config.yaml

@@ -251,7 +251,7 @@ repos:
         files: ^(ci/deps/actions-.*-minimum_versions\.yaml|pandas/compat/_optional\.py)$
     -   id: validate-errors-locations
         name: Validate errors locations
-        description: Validate errors are in approriate locations.
+        description: Validate errors are in appropriate locations.
         entry: python scripts/validate_exception_location.py
         language: python
         files: ^pandas/

doc/source/development/contributing_codebase.rst

@@ -651,7 +651,7 @@ Example
 ^^^^^^^
 
 Here is an example of a self-contained set of tests in a file ``pandas/tests/test_cool_feature.py``
-that illustrate multiple features that we like to use. Please remember to add the Github Issue Number
+that illustrate multiple features that we like to use. Please remember to add the GitHub Issue Number
 as a comment to a new test.
 
 .. code-block:: python

doc/source/development/maintaining.rst

@@ -210,15 +210,15 @@ pandas supports point releases (e.g. ``1.4.3``) that aim to:
 
   * e.g. If a feature worked in ``1.2`` and stopped working since ``1.3``, a fix can be applied in ``1.4.3``.
 
-Since pandas minor releases are based on Github branches (e.g. point release of ``1.4`` are based off the ``1.4.x`` branch),
+Since pandas minor releases are based on GitHub branches (e.g. point release of ``1.4`` are based off the ``1.4.x`` branch),
 "backporting" means merging a pull request fix to the ``main`` branch and correct minor branch associated with the next point release.
 
-By default, if a pull request is assigned to the next point release milestone within the Github interface,
+By default, if a pull request is assigned to the next point release milestone within the GitHub interface,
 the backporting process should happen automatically by the ``@meeseeksdev`` bot once the pull request is merged.
 A new pull request will be made backporting the pull request to the correct version branch.
 Sometimes due to merge conflicts, a manual pull request will need to be made addressing the code conflict.
 
-If the bot does not automatically start the backporting process, you can also write a Github comment in the merged pull request
+If the bot does not automatically start the backporting process, you can also write a GitHub comment in the merged pull request
 to trigger the backport::
 
     @meeseeksdev backport version-branch
@@ -271,14 +271,14 @@ being helpful on the issue tracker.
 The required steps for adding a maintainer are:
 
 1. Contact the contributor and ask their interest to join.
-2. Add the contributor to the appropriate `Github Team <https://github.com/orgs/pandas-dev/teams>`_ if accepted the invitation.
+2. Add the contributor to the appropriate `GitHub Team <https://github.com/orgs/pandas-dev/teams>`_ if accepted the invitation.
 
   * ``pandas-core`` is for core team members
   * ``pandas-triage`` is for pandas triage members
 
 3. Add the contributor to the pandas Google group.
-4. Create a pull request to add the contributor's Github handle to ``pandas-dev/pandas/web/pandas/config.yml``.
-5. Create a pull request to add the contributor's name/Github handle to the `governance document <https://github.com/pandas-dev/pandas-governance/blob/master/people.md>`_.
+4. Create a pull request to add the contributor's GitHub handle to ``pandas-dev/pandas/web/pandas/config.yml``.
+5. Create a pull request to add the contributor's name/GitHub handle to the `governance document <https://github.com/pandas-dev/pandas-governance/blob/master/people.md>`_.
 
 The current list of core-team members is at
 https://github.com/pandas-dev/pandas-governance/blob/master/people.md
@@ -328,7 +328,7 @@ The machine can be configured with the `Ansible <http://docs.ansible.com/ansible
 Publishing
 ``````````
 
-The results are published to another Github repository, https://github.com/tomaugspurger/asv-collection.
+The results are published to another GitHub repository, https://github.com/tomaugspurger/asv-collection.
 Finally, we have a cron job on our docs server to pull from https://github.com/tomaugspurger/asv-collection, to serve them from ``/speed``.
 Ask Tom or Joris for access to the webserver.
 

doc/source/ecosystem.rst

@@ -19,7 +19,7 @@ development to remain focused around it's original requirements.
 This is an inexhaustive list of projects that build on pandas in order to provide
 tools in the PyData space. For a list of projects that depend on pandas,
 see the
-`Github network dependents for pandas <https://github.com/pandas-dev/pandas/network/dependents>`_
+`GitHub network dependents for pandas <https://github.com/pandas-dev/pandas/network/dependents>`_
 or `search pypi for pandas <https://pypi.org/search/?q=pandas>`_.
 
 We'd like to make it easier for users to find these projects, if you know of other
@@ -599,4 +599,4 @@ Install pandas-stubs to enable basic type coverage of pandas API.
 
 Learn more by reading through :issue:`14468`, :issue:`26766`, :issue:`28142`.
 
-See installation and usage instructions on the `github page <https://github.com/pandas-dev/pandas-stubs>`__.
+See installation and usage instructions on the `GitHub page <https://github.com/pandas-dev/pandas-stubs>`__.

doc/source/getting_started/overview.rst

@@ -133,7 +133,7 @@ untouched. In general we like to **favor immutability** where sensible.
 Getting support
 ---------------
 
-The first stop for pandas issues and ideas is the `Github Issue Tracker
+The first stop for pandas issues and ideas is the `GitHub Issue Tracker
 <https://github.com/pandas-dev/pandas/issues>`__. If you have a general question,
 pandas community experts can answer through `Stack Overflow
 <https://stackoverflow.com/questions/tagged/pandas>`__.

doc/source/whatsnew/v1.5.0.rst

@@ -654,7 +654,7 @@ Deprecations
     In the next major version release, 2.0, several larger API changes are being considered without a formal deprecation such as
     making the standard library `zoneinfo <https://docs.python.org/3/library/zoneinfo.html>`_ the default timezone implementation instead of ``pytz``,
     having the :class:`Index` support all data types instead of having multiple subclasses (:class:`CategoricalIndex`, :class:`Int64Index`, etc.), and more.
-    The changes under consideration are logged in `this Github issue <https://github.com/pandas-dev/pandas/issues/44823>`_, and any
+    The changes under consideration are logged in `this GitHub issue <https://github.com/pandas-dev/pandas/issues/44823>`_, and any
     feedback or concerns are welcome.
 
 .. _whatsnew_150.deprecations.int_slicing_series:

doc/source/whatsnew/v1.6.0.rst

@@ -151,6 +151,7 @@ Other API changes
 ^^^^^^^^^^^^^^^^^
 - Passing ``nanoseconds`` greater than 999 or less than 0 in :class:`Timestamp` now raises a ``ValueError`` (:issue:`48538`, :issue:`48255`)
 - :func:`read_csv`: specifying an incorrect number of columns with ``index_col`` of now raises ``ParserError`` instead of ``IndexError`` when using the c parser.
+- Default value of ``dtype`` in :func:`get_dummies` is changed to ``bool`` from ``uint8`` (:issue:`45848`)
 - :meth:`DataFrame.astype`, :meth:`Series.astype`, and :meth:`DatetimeIndex.astype` casting datetime64 data to any of "datetime64[s]", "datetime64[ms]", "datetime64[us]" will return an object with the given resolution instead of coercing back to "datetime64[ns]" (:issue:`48928`)
 - :meth:`DataFrame.astype`, :meth:`Series.astype`, and :meth:`DatetimeIndex.astype` casting timedelta64 data to any of "timedelta64[s]", "timedelta64[ms]", "timedelta64[us]" will return an object with the given resolution instead of coercing to "float64" dtype (:issue:`48963`)
 -
@@ -222,7 +223,7 @@ Timezones
 
 Numeric
 ^^^^^^^
--
+- Bug in :meth:`DataFrame.add` cannot apply ufunc when inputs contain mixed DataFrame type and Series type (:issue:`39853`)
 -
 
 Conversion

doc/sphinxext/announce.py

@@ -3,7 +3,7 @@
 Script to generate contributor and pull request lists
 
 This script generates contributor and pull request lists for release
-announcements using Github v3 protocol. Use requires an authentication token in
+announcements using GitHub v3 protocol. Use requires an authentication token in
 order to have sufficient bandwidth, you can get one following the directions at
 `<https://help.github.com/articles/creating-an-access-token-for-command-line-use/>_
 Don't add any scope, as the default is read access to public information. The
@@ -112,7 +112,7 @@ def get_pull_requests(repo, revision_range):
     issues = re.findall("^.*\\(\\#(\\d+)\\)$", commits, re.M)
     prnums.extend(int(s) for s in issues)
 
-    # get PR data from github repo
+    # get PR data from GitHub repo
     prnums.sort()
     prs = [repo.get_pull(n) for n in prnums]
     return prs

pandas/_libs/tslibs/parsing.pyx

@@ -976,7 +976,6 @@ def guess_datetime_format(dt_str: str, bint dayfirst=False) -> str | None:
         (('hour',), '%H', 2),
         (('minute',), '%M', 2),
         (('second',), '%S', 2),
-        (('microsecond',), '%f', 6),
         (('second', 'microsecond'), '%S.%f', 0),
         (('tzinfo',), '%z', 0),
         (('tzinfo',), '%Z', 0),
@@ -1048,16 +1047,7 @@ def guess_datetime_format(dt_str: str, bint dayfirst=False) -> str | None:
 
         parsed_formatted = parsed_datetime.strftime(attr_format)
         for i, token_format in enumerate(format_guess):
-            if '.' not in tokens[i]:
-                token_filled = tokens[i].zfill(padding)
-            else:
-                seconds, nanoseconds = tokens[i].split('.')
-                seconds = f'{int(seconds):02d}'
-                # right-pad so we get nanoseconds, then only take
-                # first 6 digits (microseconds) as stdlib datetime
-                # doesn't support nanoseconds
-                nanoseconds = nanoseconds.ljust(9, '0')[:6]
-                token_filled = f'{seconds}.{nanoseconds}'
+            token_filled = _fill_token(tokens[i], padding)
             if token_format is None and token_filled == parsed_formatted:
                 format_guess[i] = attr_format
                 tokens[i] = token_filled
@@ -1100,6 +1090,19 @@ def guess_datetime_format(dt_str: str, bint dayfirst=False) -> str | None:
     else:
         return None
 
+cdef str _fill_token(token: str, padding: int):
+    cdef str token_filled
+    if '.' not in token:
+        token_filled = token.zfill(padding)
+    else:
+        seconds, nanoseconds = token.split('.')
+        seconds = f'{int(seconds):02d}'
+        # right-pad so we get nanoseconds, then only take
+        # first 6 digits (microseconds) as stdlib datetime
+        # doesn't support nanoseconds
+        nanoseconds = nanoseconds.ljust(9, '0')[:6]
+        token_filled = f'{seconds}.{nanoseconds}'
+    return token_filled
 
 cdef void _maybe_warn_about_dayfirst(format: str, bint dayfirst):
     cdef:

pandas/_version.py

@@ -1,5 +1,5 @@
 # This file helps to compute a version number in source trees obtained from
-# git-archive tarball (such as those provided by githubs download-from-tag
+# git-archive tarball (such as those provided by GitHub's download-from-tag
 # feature). Distribution tarballs (built by setup.py sdist) and build
 # directories (produced by setup.py build) will contain a much shorter file
 # that just contains the computed version number.

pandas/core/algorithms.py

@@ -762,10 +762,6 @@ def factorize(
     if not isinstance(values, ABCMultiIndex):
         values = extract_array(values, extract_numpy=True)
 
-    # GH35667, if na_sentinel=None, we will not dropna NaNs from the uniques
-    # of values, assign na_sentinel=-1 to replace code value for NaN.
-    dropna = na_sentinel is not None
-
     if (
         isinstance(values, (ABCDatetimeArray, ABCTimedeltaArray))
         and values.freq is not None
@@ -793,17 +789,8 @@ def factorize(
 
     else:
         values = np.asarray(values)  # convert DTA/TDA/MultiIndex
-        # TODO: pass na_sentinel=na_sentinel to factorize_array. When sort is True and
-        #       na_sentinel is None we append NA on the end because safe_sort does not
-        #       handle null values in uniques.
-        if na_sentinel is None and sort:
-            na_sentinel_arg = -1
-        elif na_sentinel is None:
-            na_sentinel_arg = None
-        else:
-            na_sentinel_arg = na_sentinel
 
-        if not dropna and not sort and is_object_dtype(values):
+        if na_sentinel is None and is_object_dtype(values):
             # factorize can now handle differentiating various types of null values.
             # These can only occur when the array has object dtype.
             # However, for backwards compatibility we only use the null for the
@@ -816,32 +803,15 @@ def factorize(
 
         codes, uniques = factorize_array(
             values,
-            na_sentinel=na_sentinel_arg,
+            na_sentinel=na_sentinel,
             size_hint=size_hint,
         )
 
     if sort and len(uniques) > 0:
-        if na_sentinel is None:
-            # TODO: Can remove when na_sentinel=na_sentinel as in TODO above
-            na_sentinel = -1
         uniques, codes = safe_sort(
             uniques, codes, na_sentinel=na_sentinel, assume_unique=True, verify=False
         )
 
-    if not dropna and sort:
-        # TODO: Can remove entire block when na_sentinel=na_sentinel as in TODO above
-        if na_sentinel is None:
-            na_sentinel_arg = -1
-        else:
-            na_sentinel_arg = na_sentinel
-        code_is_na = codes == na_sentinel_arg
-        if code_is_na.any():
-            # na_value is set based on the dtype of uniques, and compat set to False is
-            # because we do not want na_value to be 0 for integers
-            na_value = na_value_for_dtype(uniques.dtype, compat=False)
-            uniques = np.append(uniques, [na_value])
-            codes = np.where(code_is_na, len(uniques) - 1, codes)
-
     uniques = _reconstruct_data(uniques, original.dtype, original)
 
     return _re_wrap_factorize(original, uniques, codes)
@@ -1796,7 +1766,7 @@ def diff(arr, n: int, axis: AxisInt = 0):
 def safe_sort(
     values,
     codes=None,
-    na_sentinel: int = -1,
+    na_sentinel: int | None = -1,
     assume_unique: bool = False,
     verify: bool = True,
 ) -> np.ndarray | MultiIndex | tuple[np.ndarray | MultiIndex, np.ndarray]:
@@ -1813,8 +1783,8 @@ def safe_sort(
     codes : list_like, optional
         Indices to ``values``. All out of bound indices are treated as
         "not found" and will be masked with ``na_sentinel``.
-    na_sentinel : int, default -1
-        Value in ``codes`` to mark "not found".
+    na_sentinel : int or None, default -1
+        Value in ``codes`` to mark "not found", or None to encode null values as normal.
         Ignored when ``codes`` is None.
     assume_unique : bool, default False
         When True, ``values`` are assumed to be unique, which can speed up
@@ -1920,24 +1890,25 @@ def safe_sort(
         # may deal with them here without performance loss using `mode='wrap'`
         new_codes = reverse_indexer.take(codes, mode="wrap")
 
-        mask = codes == na_sentinel
-        if verify:
-            mask = mask | (codes < -len(values)) | (codes >= len(values))
+        if na_sentinel is not None:
+            mask = codes == na_sentinel
+            if verify:
+                mask = mask | (codes < -len(values)) | (codes >= len(values))
 
-    if mask is not None:
+    if na_sentinel is not None and mask is not None:
         np.putmask(new_codes, mask, na_sentinel)
 
     return ordered, ensure_platform_int(new_codes)
 
 
 def _sort_mixed(values) -> np.ndarray:
-    """order ints before strings in 1d arrays, safe in py3"""
+    """order ints before strings before nulls in 1d arrays"""
     str_pos = np.array([isinstance(x, str) for x in values], dtype=bool)
-    none_pos = np.array([x is None for x in values], dtype=bool)
-    nums = np.sort(values[~str_pos & ~none_pos])
+    null_pos = np.array([isna(x) for x in values], dtype=bool)
+    nums = np.sort(values[~str_pos & ~null_pos])
     strs = np.sort(values[str_pos])
     return np.concatenate(
-        [nums, np.asarray(strs, dtype=object), np.array(values[none_pos])]
+        [nums, np.asarray(strs, dtype=object), np.array(values[null_pos])]
     )
 
 

pandas/core/arraylike.py

@@ -250,6 +250,10 @@ def array_ufunc(self, ufunc: np.ufunc, method: str, *inputs: Any, **kwargs: Any)
     --------
     numpy.org/doc/stable/reference/arrays.classes.html#numpy.class.__array_ufunc__
     """
+    from pandas.core.frame import (
+        DataFrame,
+        Series,
+    )
     from pandas.core.generic import NDFrame
     from pandas.core.internals import BlockManager
 
@@ -295,8 +299,8 @@ def array_ufunc(self, ufunc: np.ufunc, method: str, *inputs: Any, **kwargs: Any)
         # At the moment, there aren't any ufuncs with more than two inputs
         # so this ends up just being x1.index | x2.index, but we write
         # it to handle *args.
-
-        if len(set(types)) > 1:
+        set_types = set(types)
+        if len(set_types) > 1 and {DataFrame, Series}.issubset(set_types):
             # We currently don't handle ufunc(DataFrame, Series)
             # well. Previously this raised an internal ValueError. We might
             # support it someday, so raise a NotImplementedError.