Allow definition of pd.CategoricalDtype with a specific categories.dtype

[lr4d]

Code Sample, a copy-pastable example if possible

from pandas.api.types import union_categoricals
import pandas as pd
import numpy as np

dt1 = pd.Categorical([], categories=[pd.Timestamp("2020-01-01")])
dt2 = pd.Categorical([], categories=[pd.Timestamp("2020-01-02")])
dt3 = pd.Categorical([])

pd.CategoricalDtype([]).categories.dtype == np.dtype("O")   # should probably be `None` or undefined

union_categoricals([dt1, dt2])
Out[113]: [], Categories (2, datetime64[ns]): [2020-01-01, 2020-01-02]
union_categoricals([dt1, dt3])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-19-2496f54dfb94> in <module>
----> 1 union_categoricals([dt1, dt3])

~/miniconda3/envs/dask/lib/python3.6/site-packages/pandas/core/dtypes/concat.py in union_categoricals(to_union, sort_categories, ignore_order)
    306         for other in to_union[1:]
    307     ):
--> 308         raise TypeError("dtype of categories must be the same")
    309 
    310     ordered = False

TypeError: dtype of categories must be the same

Problem description

At the moment, it is not possible to define the data type of categories of a CategoricalDtype. This makes things difficult when trying to merge categorical data types e.g. before performing a concatenation of dataframes, especially when one dataframe has no defined categories.

Related: dask/dask#5756

Expected Output

# Add flag `dtype` to `pd.CategoricalDtype`
dt3 = pd.Categorical([], dtype=pd.CategoricalDtype(dtype=np.dtype("<M8[ns]"))

union_categoricals([dt1, dt3])
Out[113]: [], Categories (1, datetime64[ns]): [2020-01-01]

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]

INSTALLED VERSIONS

commit : None
python : 3.6.7.final.0
python-bits : 64
OS : Darwin
OS-release : 18.5.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.1
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.2.0.post20200209
Cython : None
pytest : 5.3.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.12.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.3.5
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[TomAugspurger]

It's not pretty, but passing an empty Index object does the trick

In [8]: pd.CategoricalDtype(pd.DatetimeIndex([])).categories.dtype
Out[8]: dtype('<M8[ns]')

The CategoricalDtype constructor is already complicated, I wouldn't want to make it more so by adding a dtype argument. I'd be happy with document / testing providing an empty index as the recommended way to get a categorical dtype with a specific dtype but no categories.