[Question] Apply the same transformation to several features

[devforfu]

It is possible to apply several transformations to a single column. But is it possible to do it vice versa, i.e. like:

df = pd.DataFrame({
    'cat_first':  list('abaabababb'),
    'cat_second': list('yyyynyynny'),
    'num_first':  list(range(10)),
    'num_second': list(np.random.normal(size=10))
})

mapper = DataFrameMapper([
    (['cat_first', 'cat_second'], LabelBinarizer()),
    (['num_first', 'num_second'], StandardScaler())
], df_out=True)

output_df = mapper.fit_transform(df)
# dataframe contains 4 columns with transformed features

I know that list syntax like ['cat_first', 'cat_second'] is reserved to pass several columns into single transformer, but is it possible to somehow configure mapper to treat this syntax in a way that I've described above? Or do I need just repeat columns as separate items, like:

mapper = DataFrameMapper([
    ('cat_first', LabelBinarizer()),
    ('cat_second', LabelBinarizer()),
    # etc ...
])

[devforfu]

Or even for many to many mappings, like:

mapper = DataFrameMapper([
    (['col1', 'col2', 'col3'], [CategoricalImputer(), LabelBinarizer()])
])

So, each column is imputed and one-hot encoded.

Probably one could pass transformer class instead of instance, so each column could get its own instance.

[dukebody]

Hi @devforfu! Thanks for the proposals.

The "apply this class of transformer to multiple columns" feature can be useful. Right now I believe the easiest way to do that is:

columns = ['colA', 'colB', 'colC']
feature_defs = [(col_name, Transformer()) for col_name in columns]
mapper = DataFrameMapper(feature_defs)

I'm not sure I like the idea of making the same syntax input to the dataframe mapper mean different things in different context, but we can perhaps use an alternative approach: use a feature definition generator.

Something like:

feature_defs = gen_features(klasses=[CategoricalImputer, LabelBinarizer], columns=['colA', 'colB', 'colC'])
mapper = DataFrameMapper(feature_defs)

that would be equivalent to:

mapper = DataFrameMapper([
    ('colA', [CategoricalImputer(), LabelBinarizer()]),
    ('colB', [CategoricalImputer(), LabelBinarizer()]),
    ('colC', [CategoricalImputer(), LabelBinarizer()]),
])

This way we get the syntax sugar we want without making the DataFrameMapper code more complex. I believe this is similar to what Django does with the url function when passed to the URL routing mechanism.

Do you think you would be able to implement such a feature definition generator with an appropriate interface in a PR?

[devforfu]

Hi @dukebody, ok, understood. Yes, the generator thing sounds good. I'll do some PoC and PR then.

[scotthuang1989]

@dukebody don't you think this is more nature to a user?

mapper = DataFrameMapper([
    (['col1', 'col2', 'col3'], [CategoricalImputer(), LabelBinarizer()])
])

[dukebody]

@scotthuang1989 it might look more natural but the syntax would be ambiguous: does it mean passing the three columns at the same time to CategoricalImpute and then to LabelBinarizer, or the three columns separately and then concatenate them horizontally?

The first option is how it already works and I believe changing this could be too confusing, sorry.