Skip to content

fix: Handle double quotes inside quoted identifiers correctly #411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Feb 7, 2022
Merged

fix: Handle double quotes inside quoted identifiers correctly #411

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
113722e
fix: Handle double quotes inside quoted identifiers correctly
Feb 4, 2022
96e695f
refactor: Make quoted identifier parsing a seperate function
Feb 7, 2022
e344aac
test: Check that quoted identifier tokenization works
Feb 7, 2022
f85c399
test: Check that quoted identifiers work in mysql
Feb 7, 2022
2c9cd20
chore: cargo clippy
Feb 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions src/ast/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ use alloc::{
string::{String, ToString},
vec::Vec,
};
use core::fmt;
use core::fmt::{self, Write};

#[cfg(feature = "serde")]
use serde::{Deserialize, Serialize};
Expand Down Expand Up @@ -127,7 +127,18 @@ impl From<&str> for Ident {
impl fmt::Display for Ident {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match self.quote_style {
Some(q) if q == '"' || q == '\'' || q == '`' => write!(f, "{}{}{}", q, self.value, q),
Some(q) if q == '"' || q == '\'' || q == '`' => {
f.write_char(q)?;
let mut first = true;
for s in self.value.split_inclusive(q) {
if !first {
f.write_char(q)?;
}
first = false;
f.write_str(s)?;
}
f.write_char(q)
}
Some(q) if q == '[' => write!(f, "[{}]", self.value),
None => f.write_str(&self.value),
_ => panic!("unexpected quote style"),
Expand Down
22 changes: 20 additions & 2 deletions src/tokenizer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,8 +418,26 @@ impl<'a> Tokenizer<'a> {
quote_start if self.dialect.is_delimited_identifier_start(quote_start) => {
chars.next(); // consume the opening quote
let quote_end = Word::matching_end_quote(quote_start);
let s = peeking_take_while(chars, |ch| ch != quote_end);
if chars.next() == Some(quote_end) {
let mut last_char = None;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this code would be easier to understand if it were pulled out into its own function (peeking_take_while_ident?) which could be unit tested, to ensure it handles cases such as

foo "" bar "" 
foo bar""

let s = {
let mut s = String::new();
while let Some(ch) = chars.next() {
if ch == quote_end {
if chars.peek() == Some(&quote_end) {
chars.next();
s.push(ch);
} else {
last_char = Some(quote_end);
break;
}
} else {
s.push(ch);
}
}
s
};

if last_char == Some(quote_end) {
Ok(Some(Token::make_word(&s, Some(quote_start))))
} else {
self.tokenizer_error(format!(
Expand Down
5 changes: 5 additions & 0 deletions tests/sqlparser_postgres.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -834,6 +834,11 @@ fn parse_comments() {
}
}

#[test]
fn parse_quoted_identifier() {
pg_and_generic().verified_stmt(r#"SELECT "quoted "" ident""#);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps we can add a test for dialects (e.g. mysql) that this isn't parsed as an identifier?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a test for ` quoted strings in mysql

}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely how postgres handles things

alamb=# SELECT "quoted "" ident" from t1;
ERROR:  column "quoted " ident" does not exist
LINE 1: SELECT "quoted "" ident" from t1;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, msql treats this differently

Database changed
mysql> SELECT "quoted "" ident" from t1;
+----------------+
| quoted " ident |
+----------------+
| quoted " ident |
| quoted " ident |
| quoted " ident |
| quoted " ident |
| quoted " ident |
+----------------+
5 rows in set (0.00 sec)

mysql> SELECT "quoted  ident" from t1;
+---------------+
| quoted  ident |
+---------------+
| quoted  ident |
| quoted  ident |
| quoted  ident |
| quoted  ident |
| quoted  ident |
+---------------+
5 rows in set (0.00 sec)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

" is used for strings in mysql unless ANSI_QUOTES is enabled https://dev.mysql.com/doc/refman/8.0/en/identifiers.html . Neither double quoted strings or ANSI_QUOTES seems implemented in this crate and I don't think it is up to this PR to fix that.


fn pg() -> TestedDialects {
TestedDialects {
dialects: vec![Box::new(PostgreSqlDialect {})],
Expand Down