BUG: also use Float Nullable dtypes in read_parquet when use_nullable_dtypes=True

[Jaafarben2]

fixes #45694
Add float types in the mapping dictionary

[rhshadrach]

Thanks for the PR! Please add tests, whatsnew line, and link this PR to the original issue.

[Jaafarben2]

Before Fix : float64 is returned instead of Float64 #45700

>>> import pandas as pd
>>> df = pd.DataFrame({'a':[1, 2, 3], 'b':[1.0, 2.0, 3]})
>>> df['c'] = pd.Series([1, 2, 3], dtype='uint8')
>>> df.to_parquet('a')
>>> pd.read_parquet('a').dtypes
a      int64
b    float64
c      uint8
dtype: object
>>> pd.read_parquet('a', use_nullable_dtypes=True).dtypes
a      Int64
b    float64
c      UInt8
dtype: object

After Fix : Float64 is returned instead of float64 and bug is fixed

>>> import pandas as pd 

>>> df = pd.DataFrame({'a':[1, 2, 3], 'b':[1.0, 2.0, 3]})

>>> df['c'] = pd.Series([1, 2, 3], dtype='uint8')

>>> pd.read_parquet('a').dtypes
a      int64
b    float64
c      uint8
dtype: object

>>> df.to_parquet('a')

>>> pd.read_parquet('a', use_nullable_dtypes=True).dtypes
a      Int64
b    Float64
c      UInt8
dtype: object

[rhshadrach]

@Jaafarben2 - the tests need to be added to pandas.tests, see the dev docs for more details.

[jreback]

yup, w/o tests we can't even tell if this works

[jorisvandenbossche]

Thanks for the PR!

To be explicit for the required test, there is an existing test for this functionality at

pandas/pandas/tests/io/test_parquet.py

Lines 603 to 645 in 5c757e9

	def test_use_nullable_dtypes(self, engine, request):
	import pyarrow.parquet as pq
	if engine == "fastparquet":
	# We are manually disabling fastparquet's
	# nullable dtype support pending discussion
	mark = pytest.mark.xfail(
	reason="Fastparquet nullable dtype support is disabled"
	)
	request.node.add_marker(mark)
	table = pyarrow.table(
	{
	"a": pyarrow.array([1, 2, 3, None], "int64"),
	"b": pyarrow.array([1, 2, 3, None], "uint8"),
	"c": pyarrow.array(["a", "b", "c", None]),
	"d": pyarrow.array([True, False, True, None]),
	# Test that nullable dtypes used even in absence of nulls
	"e": pyarrow.array([1, 2, 3, 4], "int64"),
	}
	)
	with tm.ensure_clean() as path:
	# write manually with pyarrow to write integers
	pq.write_table(table, path)
	result1 = read_parquet(path, engine=engine)
	result2 = read_parquet(path, engine=engine, use_nullable_dtypes=True)
	assert result1["a"].dtype == np.dtype("float64")
	expected = pd.DataFrame(
	{
	"a": pd.array([1, 2, 3, None], dtype="Int64"),
	"b": pd.array([1, 2, 3, None], dtype="UInt8"),
	"c": pd.array(["a", "b", "c", None], dtype="string"),
	"d": pd.array([True, False, True, None], dtype="boolean"),
	"e": pd.array([1, 2, 3, 4], dtype="Int64"),
	}
	)
	if engine == "fastparquet":
	# Fastparquet doesn't support string columns yet
	# Only int and boolean
	result2 = result2.drop("c", axis=1)
	expected = expected.drop("c", axis=1)
	tm.assert_frame_equal(result2, expected)

This tests the roundtrip of a dataframe with some columns that have a nullable dtype variant in pandas. I think it should be sufficient to add two extra columns to the test dataframe (float32 and float64) in that existing test.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.