BUG: Dataframe.to_xml() fails with control characters

[CarlosHartmann]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

csv_content = b'''FOO,BAR,BYTES\nfoo,bar,"Next byte is a control char \x13"\n'''

with open('file.csv', 'wb') as outfile:
    outfile.write(csv_content)

df = pd.read_csv('file.csv', sep=',')

df.to_xml('file.xml')

Issue Description

The process fails without writing anything into an XML file and without giving any information about the positiong or which char/byte caused the failure.

This is the error message:

File "/opt/miniconda3/envs/pylaia/lib/python3.7/site-packages/pandas/core/frame.py", line 2986, in to_xml
    return xml_formatter.write_output()
  File "/opt/miniconda3/envs/pylaia/lib/python3.7/site-packages/pandas/io/formats/xml.py", line 265, in write_output
    xml_doc = self.build_tree()
  File "/opt/miniconda3/envs/pylaia/lib/python3.7/site-packages/pandas/io/formats/xml.py", line 485, in build_tree
    self.build_elems()
  File "/opt/miniconda3/envs/pylaia/lib/python3.7/site-packages/pandas/io/formats/xml.py", line 575, in build_elems
    SubElement(self.elem_row, elem_name).text = val
  File "src/lxml/etree.pyx", line 1024, in lxml.etree._Element.text.__set__
  File "src/lxml/apihelpers.pxi", line 747, in lxml.etree._setNodeText
  File "src/lxml/apihelpers.pxi", line 735, in lxml.etree._createTextNode
  File "src/lxml/apihelpers.pxi", line 1540, in lxml.etree._utf8
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters

Expected Behavior

Please forgive if my assumption is wrong or Pandas is not the right place to report this, I'm fairly new to Pandas.

I would expect either of those behaviors:

1. Delete the control characters without feedback to the user. Or maybe have delete_non-utf8 as an option for to_xml.
2. Replace them with an HTML entity, like  for the character above that causes the failure.
3. Fail but indicate which character caused the issue and perhaps also its position.

Again I'm fairly new. I thought 1. was going to happen because I am fairly sure that in the data I was handling there was ANSI content in an otherwise UTF-8-encoded file. Somehow, Pandas handled it without issue. That's why I thought Pandas would be robust to any such issues.

If my assumption is wrong and the current behavior is preferred over 1, I would like to suggest Nr. 3 as it would have made work slightly easier with a large amount of data in my dataframe.

Installed Versions

INSTALLED VERSIONS

commit : 66e3805
python : 3.7.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-84-generic
Version : #94-Ubuntu SMP Thu Aug 26 20:27:37 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.5
numpy : 1.20.2
pytz : 2022.1
dateutil : 2.8.1
pip : 21.1.2
setuptools : 52.0.0.post20210125
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : 2.8.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.7.0
sqlalchemy : None
tables : 3.7.0
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[ParfaitG]

This error derives from the underlying lxml package (default parser in to_xml) that pandas raises on the top-level call as seen in traceback. But had you used the etree parser you would received a different error indicating the problematic position:

df.to_xml(parser="etree")

...
ExpatError: not well-formed (invalid token): line 1, column 95

And line 1, column 95 is positioned exactly at the problematic byte in attempted XML markup:

<data><row><index>0</index><FOO>foo</FOO><BAR>bar</BAR><BYTES>Next byte is a control char char</BYTES>
line 1 column 95 -----------------------------------------------------------------------------^

Given the XML specifications of allowable characters are a subset of the universe of all character sets, such invalid strings/bytes can range from one to many. Therefore, it is upon the user to know their data and properly clean content for well-formed XML.

Lxml may have decided it is not worthwhile to raise every specific, potentially many character set issue that does not conform to XML specifications. But consider raising the concern on their mailing list.

[mroeschke]

Thanks for the report, but as described above, this appears to be an upstream issue with the parsing library & underlying data and not appropriate to address in pandas itself. Closing