COMPAT: box int/floats in __iter__

[jreback]

xref #13236 as .map was recently clarified to box ints/floats (and all other dtypes) to python/pandas types (rather than numpy scalars). __iter__ should do the same (it already is done for datetimelikes), need to add for int/floats.

Furthermore Series.tolist() also returns python types (#10904)

This will make it more consistent with pandas iteration strategy.

We also already do this for object types

In [12]: type(list(Series([Timestamp('20130101'),Timestamp('20130101',tz='US/Eastern'),1,1.0,'foo']))[2])
Out[12]: int

.tolist()

In [3]: type(Series([1,2,3]).tolist()[2])
Out[3]: int

let's fix for pure types

In [14]: type(list(Series([1,2,3]))[2])
Out[14]: numpy.int64

When solving this issue, also have to look at:

• Inconsistent conversion in to_dict to numpy/python scalar (Inconsistent type casting between DataFrame and Series #14216)

[jreback]

cc @glaucouri
cc @gliptak
@sinhrks

[jreback]

In [4]: arr = np.array([1,2,3])

In [7]: type(list(arr)[0])
Out[7]: numpy.int64

In [9]: type(arr.tolist()[0])
Out[9]: int

interesting that numpy converts with .tolist() but NOT with list(..); seems very odd to me.

[jreback]

cc @njsmith do you know why numpy is like this? (my last comment)
@shoyer

[njsmith]

There was a big debate in the early days about whether numpy indexing should return python scalars or what. The compromise was that it doesn't, but there is also this bolted on notion of "convert to a Python style object" put in to appease the other side. This python conversion API lost the battle for users and mindshare, and these days no one uses intentionally or even realizes that it's in there, but it's still in there. Specifically the python object API is .item(...) for index-and-convert-to-python, and .tolist() as a shorthand for, well, .item(...) (I.e. passing in a literal ellipsis analogous to arr[...], so, converting the whole array recursively to python objects.) That's what you're invoking with .tolist(). The names of these functions are very misleading.

OTOH list(arr) just uses list's usual convert-from-iterable logic, so you get the expected thing.

[glaucouri]

To be honest, i'm puzzled as Jeff, the question is: Why in numpy algo you must convert a ndarray to a python list? Is this unequivocally to go in a python types (so is better to convert to native data types) ?

I think is more clear to mantain always native types, without do any automatic conversion.
"""Explicit is better than implicit"""

Eventually if someone need to exit from this logic, he know exaclty what is doing, and why.
Probably is more clear a method for this purpose , something like:
to_py_list()
iter_py()

I appreciate this approach used in pandas for Timestamp.

my 50 cents.
Gla

[sinhrks]

+1. When we iterate numpy dtype, can use .values

[jstray]

This has just bitten me in a surprising way, along the lines of #16048: converting a dataframe to a dict results in an object that cannot be converted to json.

import pandas as pd
import json

t = pd.DataFrame({'A':[1], 'B':[2]})
d = t.to_dict(orient="records") 

// at this point type(d[0]['A']) = numpy.int64

json.dumps(d)  // fails with "TypeError: 2 is not JSON serializable"

There seems to be no easy to way to generate a dict of purely Python types. Maybe one could hack something together using np.asscalar()? But it seems like this should not be necessary. Perhaps an option to use only native Python types in to_dict()?

And before someone suggests t.to_json(), that won't work if you want the dataframe to be just one part of the json output. I've resorted to concatenating strings to produce my final json result. Like a savage.

[jreback]

@jstray actually there is an easy way

PR to fix this issue! it's pretty straightforward :)

[jstray]

@jreback what is the easy way?

[jstray]

Thanks @jreback for your fix! This was way deeper into the Pandas codebase than I was qualified to go.

[makmanalp]

Thanks @jreback ! This saves my butt!

[alexlenail]

HUUUGE +1