Subclassing is lost when using DataFrame.apply

[jaumebonet]

Code Sample, a copy-pastable example if possible

import pandas as pd

# Define a subclass of Series
class ExtendedSeries( pd.Series ):
    @property
    def _constructor(self):
        return ExtendedSeries

# Define a subclass of DataFrame that slices into the ExtendedSeries class
class ExtendedFrame( pd.DataFrame ):
    @property
    def _constructor(self):
        return ExtendedFrame
    _constructor_sliced = ExtendedSeries

# Declare instance of ExtendedFrame
df = ExtendedFrame({"c1": [1, 2, 3, 4], "c2": [2, 3, 4, 5], "c3": [3, 4, 5, 6]})

print(type(df))
# <class '__main__.ExtendedFrame'>

print(type(df.iloc[0]))
# <class '__main__.ExtendedSeries'>

for x in df.apply(lambda x: type(x), axis=1):
    print(x)
#<class 'pandas.core.series.Series'>
#<class 'pandas.core.series.Series'>
#<class 'pandas.core.series.Series'>
#<class 'pandas.core.series.Series'>

Problem description

The class type of the applied objects should still be <class '__main__.ExtendedSeries'> instead of going back to <class 'pandas.core.series.Series'>.

The issue here is that in pandas.core.apply, whenever the returning objects are generated, they are declared as DataFrame and Series instead of self.obj._constructor and self.obj._constructor_slice.

This behaviour makes it difficult to properly work with subclasses from pandas.

Expected Output

Following the initial code example, the expected output should be:

import pandas as pd

# Define a subclass of Series
class ExtendedSeries( pd.Series ):
    @property
    def _constructor(self):
        return ExtendedSeries

# Define a subclass of DataFrame that slices into the ExtendedSeries class
class ExtendedFrame( pd.DataFrame ):
    @property
    def _constructor(self):
        return ExtendedFrame
    _constructor_sliced = ExtendedSeries

# Declare instance of ExtendedFrame
df = ExtendedFrame({"c1": [1, 2, 3, 4], "c2": [2, 3, 4, 5], "c3": [3, 4, 5, 6]})

print(type(df))
# <class '__main__.ExtendedFrame'>

print(type(df.iloc[0]))
# <class '__main__.ExtendedSeries'>

for x in df.apply(lambda x: type(x), axis=1):
    print(x)
#<class '__main__.ExtendedSeries'>
#<class '__main__.ExtendedSeries'>
#<class '__main__.ExtendedSeries'>
#<class '__main__.ExtendedSeries'>

(note the change in the last printed lines)

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.12.final.0
python-bits: 64
OS: Darwin
OS-release: 17.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.21.1
pytest: None
pip: 9.0.1
setuptools: 38.4.0
Cython: None
numpy: 1.14.0
scipy: 0.18.1
pyarrow: None
xarray: None
IPython: 5.4.1
sphinx: 1.6.6
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.1.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.5.1
html5lib: 0.999999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None