Custom Array Types Dropped on pd.Merge

[achapkowski]

Code Sample, a copy-pastable example if possible

I have written a custom dtype and array for my own API.

When I do a pd.merge, the type reverts back to object not geometry .

In cyberpandas, I do not see the same behavior. What needs to be set to accomplish this?

from cyberpandas import IPArray

import pandas as pd

arr = IPArray(['192.168.1.1',
               '2001:0db8:85a3:0000:0000:8a2e:0370:7334'])
ser = pd.Series(arr)
data = {'idarray': ser,
'OBJECTID' : [1,2] }

dfother = pd.DataFrame({'OBJECTID' : [1,2], 'CLASS': ['a', 'b']})

df = pd.DataFrame(data)
pd.merge(df, dfother).dtypes

[jreback]

duplicate for #23020 there is a PR in the works

[achapkowski]

@jreback thanks Jeff for letting me know.

[achapkowski]

@TomAugspurger How did you account for this on cyberpandas? Just curious

[TomAugspurger]

Not sure. #23020 looks a bit different an the original post. IIUC, #23020 is about merging on an ExtensionArray column.

Merging two dfs on a non-EA column should be fine. I suspect it'll involve a take on the EAs at some point, so perhaps make sure that's working correctly.

[jorisvandenbossche]

It's indeed different than #23020 (which is about merging on an EA columns). @achapkowski We already have tests in the BaseExtensionTests classes that checks that the EA type is preserved when doing such a merge. Do you run those tests for your custom ExtensionArray and do they pass?
If they do, but you still have a problem, you will need to provide a reproducible example.

[achapkowski]

@jorisvandenbossche They did when I created the extension array, let me re-run them and I'll get back to you today.

[achapkowski]

@jorisvandenbossche Which BaseExtensionTests are you referring to? In v.23 there are no tests called BaseExtensionTests in from pandas.tests.extension import base

[jorisvandenbossche]

I mean all the test that inherit from BaseExtensionTests that live in the pandas/tests/extension/base directory. You might need to work with pandas master to make sure to have all the latest tests.