BUG: Errors using to_sql(), read_sql_table() when data contains percent signs

[philipnye]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

# %%
import os
import urllib

import pandas as pd
from sqlalchemy import create_engine

# %%
# CONNECT TO D/B
driver = '{ODBC Driver 17 for SQL Server}'
server = os.environ['odbc_server']
database = os.environ['odbc_database']
authentication = 'ActiveDirectoryInteractive'
username = os.environ['odbc_username']

conn = urllib.parse.quote_plus(
    'DRIVER=' + driver +
    ';SERVER=' + server +
    ';DATABASE=' + database +
    ';UID=' + username +
    ';AUTHENTICATION=' + authentication + ';'
)
engine = create_engine('mssql+pyodbc:///?odbc_connect={}'.format(conn))

# %%
df = pd.DataFrame(
    data=[[1, 2, 3], [4, 5, 6]],
    columns=['a', 'b', 'c']
)

# %%
df.to_sql(
    name='Classified roads where maintenance should be considered, %',
    schema='[Reference.Infrastructure and transport.Roads.Maintenance.Source links]',
    con=engine,
    if_exists='replace',
    index=False
)

# %%
df2 = pd.read_sql_table(
    table_name='Classified roads where maintenance should be considered, %',
    schema='[Reference.Infrastructure and transport.Roads.Maintenance.Source links]',
    con=engine,
    index_col=False
)

Issue Description

In some circumstances such as this, schema in read_sql_table() is being interpreted as a database name + schema, rather than just a schema.

This results in a use <database> statement - and as ODBC Driver 17 for SQL Server/Azure SQL Database doesn't accept use <database> statements this leads to an error:

ProgrammingError: (pyodbc.ProgrammingError) ('42000', '[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]USE statement is not supported to switch between databases. Use a new connection to connect to a different database. (40508) (SQLExecDirectW)') [SQL: use [Reference.Infrastructure and transport.Roads.Maintenance]]

In this example, Reference.Infrastructure and transport.Roads.Maintenance.Source links exists as a schema within the database. (In case of relevance, Reference.Infrastructure and transport.Roads.Maintenance is also a schema in the database.)

I'm struggling to work out the exact trigger for the error - if it's connected to the length of the schema name/the number of full stops (periods) in it. I get the error consistently using this schema name, but am struggling to reproduce it using other schemas that follow a similar structure. E.g. the following doesn't give me an error:

df2 = pd.read_sql_table(
    table_name='test',
    schema='[a.b.c.d.e e]',
    con=engine,
    index_col=False
)

Where a.b.c.d.e e is a schema within the database (as is a.b.c.d.e).

Expected Behavior

schema is just interpreted as a schema name, rather than database name + schema. Resulting in read_sql_table() retrieving table Classified roads where maintenance should be considered, % in schema Reference.Infrastructure and transport.Roads.Maintenance.Source links in my example.

Installed Versions

INSTALLED VERSIONS

commit : 37ea63d
python : 3.9.10.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United Kingdom.1252

pandas : 2.0.1
numpy : 1.22.1
pytz : 2021.3
dateutil : 2.8.2
setuptools : 58.1.0
pip : 22.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.0.2
lxml.etree : 4.8.0
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 8.0.1
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : 6.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : 1.4.44
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

Also using pyodbc==4.0.35
I'm connecting to a database hosted in Azure SQL Database.

[philipnye]

I've been doing further investigation and:

1. This seems to apply to to_sql() too (I should have spotted - as in my minimal example above), and quite possibly will apply to other functions
2. The cause seems to be the percentage sign in the table name - combined with the idiosyncratic use of schema names with full stops in

There are at least a couple of other issues that mention problems with percent signs (#11896, #34211), though they mention percent signs in column names and in SQL commands rather than table names, so I think are distinct issues.

It makes me think the issue is quite likely a SQLAlchemy one - though I don't know whether pandas has control over how it uses SQLAlchemy for these operations as in #34211. I'll try updating SQLAlchemy and will post back with the outcome of that.

[philipnye]

1. I've updated to SQLAlchemy v2.0.15 and I'm still experiencing the issue
2. Investigating further, I think it's the data being saved to the database in to_sql()/read from the database in read_sql_table() that's causing the problem, rather than the naming of the schema/table itself.

My data looks like this:

id	data_id	metadata_id	source_schema	source_table	source_id
cf8f807c-9adb-4a07-8748-003a6ca90d7a	88a6d1d7-71ed-43d8-a54f-18ff32ec4dd6	f485668c-8afb-4ad7-8a76-56a45f675085	Source.Infrastructure and transport.Roads.Maintenance.Dataset	Classified roads where maintenance should be considered, 'A' roads and motorways, %, 2021/22	614d3695-b6c1-45ab-8a40-aab3bbe900c5
2b7ea97-0537-4d48-beae-003e1ba8ac7d	2127b15-21a1-4b9b-9316-7324907df504	f485668c-8afb-4ad7-8a76-56a45f675085	Source.Infrastructure and transport.Roads.Maintenance.Dataset	Classified roads where maintenance should be considered, 'A' roads and motorways, %, 2021/22	f438858a-83dc-4780-9b64-c9597a51f446

If I save the same data minus the source_table column - which contains percent signs - then things seem to work fine.

So this seems to bear some resemblance to #11896 - but that only mentions percent signs in column names, and relates to postgresql.

[topper-123]

Can you check if this also happens with other databases, especially (ideally) POSTGres and/or SQLite?

[philipnye]

Thanks, I don't have any experience with either Postgres or SQLite.

I should add to my reports above - I spent quite a while longer going round in circles with the issue, and if anything became less rather than more clear on what the cause is. I wonder if it's in fact to do with holding names of valid schemas in a table column (possibly in combination with those schema names being quite idiosyncratic, and containing full stops and in some cases spaces), rather than anything to do with percent signs. But I can't say that with great confidence.

Aware that's very unhelpful for trying to debug the issue - as I say, I feel like I've been going round and round with this one.