Logical follow up to https://github.com/pandas-dev/pandas/pull/24423/files#r244284618 cc @gfyoung