API/TST: expand tests for string any/all reduction + fix pyarrow-based implementation

[jorisvandenbossche]

While working on adding any/all support for the object-dtype version of the StringDtype in #58451, I bumped into some issues with the current testing and implementation.

It seemed that our current testing is a bit limited, so I expanded the existing test to cover more cases and to be parametrized over all string dtype variants (including plain object dtype).

But that uncovered some issues with the current implementation of the pyarrow-backed version:

• any could return pd.NA in case of skipna=False, while for this version of the string dtype (not using Kleene logic), the result should always be True or False. This is an easy fix (we currently were only filling missing values in case of all, also do this for any)
• How to treat missing values? As truthy or falsey? The current pyarrow-based implementation explicitly filled the missing values with False, but the documentation says "If skipna is False, then NA are treated as True, because these are not equal to zero."
  I find that a bit strange (my first expectation would be to treat missing values as False), but it is explicitly documented, and follows the behaviour of current object dtype (eg pd.Series(["a". np.nan], dtype=object).all(skipna=False) gives True, not False.

So for now this PR updated the pyarrow implementation to follow the documented behaviour.

• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[jorisvandenbossche]

Before, for converting the string array to a boolean array, True values were considered as "not "", and not null", and this diff changed that to "not "", or null"

[jorisvandenbossche]

The places where I currently commented this # NaN is considered truthy are the test cases where the result would change from True to False if NaN would be considered as falsey instead.

[mroeschke]

In 3.0, adjacently .idxmax/min will raise a ValueError if skipna=False and a NA value is encountered. There could be an API behavior argument to align with that behavior in the future.

Besides that point, I would also agree with your initial expectation that NA values are falsey.

[jorisvandenbossche]

In 3.0, adjacently .idxmax/min will raise a ValueError if skipna=False and a NA value is encountered. There could be an API behavior argument to align with that behavior in the future.

For any/all, I think in the future we don't have to raise about missing values, but we can use Kleene logic and return pd.NA in some cases (although that might then just give an error downstream in your code ..).

That's the behaviour that is currently implemented for the nullable BooleanDtype, and if we plan to stick to that behaviour, I would say it's not worth to change the current NaN-based dtypes to start raise an error for NaNs (and we can preserve the strange "NaN is considered as True because it is not equal to 0" for now).

[mroeschke]

Thanks @jorisvandenbossche