Skip to content

enable supports_filter_during_aggregation for Generic dialect #1815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 19, 2025

Conversation

goldmedal
Copy link
Contributor

@goldmedal
Copy link
Contributor Author

@alamb, What do you think about this? I think this change is related to the default behavior of DataFusion.

@alamb alamb changed the title enable supports_filter_during_aggregation for Generic enable supports_filter_during_aggregation for Generic dialect Apr 17, 2025
@alamb
Copy link
Contributor

alamb commented Apr 17, 2025

Makes sense to me @goldmedal -- I think the intent is that the GenericDialect should support as many SQL constructs as possible

The only thing I think this PR needs is a test and it should be good to go

@goldmedal
Copy link
Contributor Author

The only thing I think this PR needs is a test and it should be good to go

oops , I forgot to mention the tests. I found it will be covered by the original case:

fn test_selective_aggregation() {
let sql = concat!(
"SELECT ",
"ARRAY_AGG(name) FILTER (WHERE name IS NOT NULL), ",
"ARRAY_AGG(name) FILTER (WHERE name LIKE 'a%') AS agg2 ",
"FROM region"
);
assert_eq!(
all_dialects_where(|d| d.supports_filter_during_aggregation())
.verified_only_select(sql)
.projection,
vec![

I guess it's enough?

@alamb
Copy link
Contributor

alamb commented Apr 17, 2025

oops , I forgot to mention the tests. I found it will be covered by the original case:

I was thinking a test that would prevent against regressions in a future refactor -- namely a test that will fail if this code change is reverted,

I think as it stands, if the code change is reverted all the tests will still pass 🤔

@goldmedal
Copy link
Contributor Author

oops , I forgot to mention the tests. I found it will be covered by the original case:

I was thinking a test that would prevent against regressions in a future refactor -- namely a test that will fail if this code change is reverted,

I think as it stands, if the code change is reverted all the tests will still pass 🤔

I see. Make sense. I'll add another test to protect it.

Comment on lines +11645 to +11657
let testing_dialects = all_dialects_where(|d| d.supports_filter_during_aggregation());
let expected_dialects: Vec<Box<dyn Dialect>> = vec![
Box::new(PostgreSqlDialect {}),
Box::new(DatabricksDialect {}),
Box::new(HiveDialect {}),
Box::new(SQLiteDialect {}),
Box::new(DuckDbDialect {}),
Box::new(GenericDialect {}),
];
assert_eq!(testing_dialects.dialects.len(), expected_dialects.len());
expected_dialects
.into_iter()
.for_each(|d| assert!(d.supports_filter_during_aggregation()));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I list all the dialects expected to support filtering during aggregation.

@goldmedal goldmedal requested a review from alamb April 18, 2025 13:34
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @goldmedal

FYI @iffyio

@alamb alamb merged commit 3ec80e1 into apache:main Apr 19, 2025
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants