ENH: Add use_nullable_dtypes in csv internals

[phofl]

• xref ENH: add option to get nullable dtypes to pd.read_csv #36712 (Replace xxxx with the Github issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

This adds the required casting logic for extension array dtypes. Want to get this done as a first step before adding a public option to read_csv. The unit tests can be used as a guideline for the required behaviour.

This always casts to ea dtypes, even when no missing values are present. From a users perspective this makes the most sense imo. You don't want to do a follow up operation and end up with float in a pipeline, when you set nullable dtypes (reindexing, enlargement in indexing ops, ...). That said, if there are concerns about this, we could also do something like

use_nullable_dtypes=False | always | when necessary

cc @jorisvandenbossche

[jbrockmendel]

is there a viable way to do this outside of the cython code?

[jorisvandenbossche]

A possible option might be to change _maybe_upcast to optionally return a values array + mask array, and then do the actual construction in python?

Although since _maybe_upcast is only being called in this file (and thus from cython), that won't help. Unless if we would propagate such a (values, mask) tuple (instead of ArrayLike) through the different calls and return that from the TextReader.read method.

[jbrockmendel]

yah not worth contorting ourselves to avoid non-cython imports. just if there's a convenient alternative

[phofl]

Could be done yes, but would make logic more complex, which is something we should avoid here I think

[jbrockmendel]

yah, never mind then. thanks for taking a look

[mroeschke]

LGTM

[mroeschke]

Thanks @phofl