BUG: Dataframe.to_dict(orient='records', index=True) doesn't include the index

[charlie-corus]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> df_a = pd.DataFrame({'col1': [1, 2],
                         'col2': [0.5, 0.75],
                         'row': ['row1', 'row2']})
>>> df_a.set_index('row', inplace=True)
>>> df_a
      col1  col2
row
row1     1  0.50
row2     2  0.75

>>> df_a.to_dict(orient="records", index=True)
[{'col1': 1, 'col2': 0.5}, {'col1': 2, 'col2': 0.75}]

Issue Description

Calling DataFrame.to_dict(orient='records', index=True) doesn't return the index, and doesn't warn that it isn't going to. This is true whether or not the index is named.

Expected Behavior

If the index is named, then its values should be returned with the records.

>>> df_a = pd.DataFrame({'col1': [1, 2],
                         'col2': [0.5, 0.75],
                         'row': ['row1', 'row2']})
>>> df_a.set_index('row', inplace=True)
>>> df_a
      col1  col2
row
row1     1  0.50
row2     2  0.75

>>> df_a.to_dict(orient="records", index=True)
[{'row': 'row1', 'col1': 1, 'col2': 0.5}, {'row': 'row2', 'col1': 2, 'col2': 0.75}]

If the index is not named, and the index=True flag is set, there should either be a warning, or the values should be included with a placeholder name like index:

>>> df_b = pd.DataFrame({'col1': [1, 2],
                         'col2': [0.5, 0.75]},
                         'index'= ['row1', 'row2'])
>>> df_b
      col1  col2
row1     1  0.50
row2     2  0.75
>>> df_b.to_dict(orient="records", index=True)
[{'index': 'row1', 'col1': 1, 'col2': 0.5}, {'index': 'row2', 'col1': 2, 'col2': 0.75}]

Installed Versions

INSTALLED VERSIONS

commit : a671b5a
python : 3.11.0.final.0
python-bits : 64
OS : Darwin
OS-release : 23.2.0
Version : Darwin Kernel Version 23.2.0: Wed Nov 15 21:59:33 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8112
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 2.1.4
numpy : 1.24.3
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 68.2.2
pip : 23.3.1
Cython : None
pytest : 7.4.3
hypothesis : None
sphinx : 7.2.6
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.2
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.18.1
pandas_datareader : None
bs4 : 4.12.2
bottleneck : 1.3.7
dataframe-api-compat: None
fastparquet : None
fsspec : 2023.10.0
gcsfs : None
matplotlib : 3.8.2
numba : 0.58.1
numexpr : 2.8.7
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 11.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.11.4
sqlalchemy : 2.0.23
tables : 3.8.0
tabulate : 0.9.0
xarray : 2023.12.0
xlrd : None
zstandard : 0.22.0
tzdata : 2023.3
qtpy : 2.4.1
pyqt5 : None

[rhshadrach]

Thanks for the report. The documentation indicates this is the expected behavior:

‘records’ : list like [{column -> value}, … , {column -> value}]

and this is clear from the PR that added records as an option (#4936).

Both warning or raising when index=True seems to me to be a not so great user experience as this is the default value. We could also change the argument to something like index=[["infer"], True, False], but I'm not sure this is worthwhile.

The documentation for index could certainly be made more clear. I'm going to recast this as a docs improvement, but any further discussion / ideas on changes are welcome.