Skip to content

Support NULL literals in where clause #11266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 9, 2024

Conversation

xinlifoobar
Copy link
Contributor

@xinlifoobar xinlifoobar commented Jul 4, 2024

Which issue does this PR close?

Closes #11248

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) labels Jul 4, 2024
@xinlifoobar xinlifoobar changed the title Try fix where clause incorrectly reject NULL literal [POC] Try fix where clause incorrectly reject NULL literal Jul 4, 2024
@xinlifoobar xinlifoobar changed the title [POC] Try fix where clause incorrectly reject NULL literal Try fix where clause incorrectly reject NULL literal Jul 5, 2024
},
Err(_) => {
let Ok(null_array) = as_null_array(&array) else {
return internal_err!("Cannot create filter_array from non-boolean predicates, unable to continute");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the previous it will throw a downcast_error, which mess up the debugging logs.

@@ -24,3 +24,17 @@ query TT?
select 'foo', '', NULL
----
foo (empty) NULL

# Where clause accept NULL literal
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From postgres,

postgres=# select 1 where null and 1=1;
 ?column? 
----------
(0 rows)

postgres=# select 1 where null and 1=1;
 ?column? 
----------
(0 rows)

postgres=# select 1 where null or 1=1;
 ?column? 
----------
        1
(1 row)

@alamb alamb changed the title Try fix where clause incorrectly reject NULL literal Support NULL literals in where clause Jul 8, 2024
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @xinlifoobar -- this makes sense to me

Another potential way to fix this issue would be to cast / coerce NULL literals in the WHERE clause to boolean

Basically we could add the equivalent of NULL::boolean

For example, given the input

SELECT ... FROM foo WHERE NULL and x = 5;

We could transform it to

SELECT ... FROM foo WHERE NULL::boolean and x = 5;

That might result in simpler code as FilterExec would not have to be aware of DataType::null

Perhaps it would fit in https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/analyzer/type_coercion.rs

However, I think this PR is better than main and we could try to see if the alternate approach loked better as a future work (or never)

@alamb alamb merged commit fa01917 into apache:main Jul 9, 2024
25 of 26 checks passed
@alamb
Copy link
Contributor

alamb commented Jul 9, 2024

Thanks again @xinlifoobar

@xinlifoobar
Copy link
Contributor Author

Thanks @xinlifoobar -- this makes sense to me

Another potential way to fix this issue would be to cast / coerce NULL literals in the WHERE clause to boolean

Basically we could add the equivalent of NULL::boolean

For example, given the input

SELECT ... FROM foo WHERE NULL and x = 5;

We could transform it to

SELECT ... FROM foo WHERE NULL::boolean and x = 5;

That might result in simpler code as FilterExec would not have to be aware of DataType::null

Perhaps it would fit in https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/analyzer/type_coercion.rs

However, I think this PR is better than main and we could try to see if the alternate approach loked better as a future work (or never)

This is a lot... Let me looking into this.

findepi pushed a commit to findepi/datafusion that referenced this pull request Jul 16, 2024
* Try fix where clause incorrectly reject NULL literal

* check null in filter
xinlifoobar added a commit to xinlifoobar/datafusion that referenced this pull request Jul 16, 2024
xinlifoobar added a commit to xinlifoobar/datafusion that referenced this pull request Jul 18, 2024
* Try fix where clause incorrectly reject NULL literal

* check null in filter
alamb pushed a commit that referenced this pull request Jul 18, 2024
…11491)

* Revert "Support `NULL` literals in where clause  (#11266)"

This reverts commit fa01917.

* Followup Support NULL literals in where clause

* misc err change

* adopt comparison_coercion

* Fix comments

* Fix comments
Lordworms pushed a commit to Lordworms/arrow-datafusion that referenced this pull request Jul 23, 2024
…pache#11491)

* Revert "Support `NULL` literals in where clause  (apache#11266)"

This reverts commit fa01917.

* Followup Support NULL literals in where clause

* misc err change

* adopt comparison_coercion

* Fix comments

* Fix comments
wiedld pushed a commit to influxdata/arrow-datafusion that referenced this pull request Jul 31, 2024
…pache#11491)

* Revert "Support `NULL` literals in where clause  (apache#11266)"

This reverts commit fa01917.

* Followup Support NULL literals in where clause

* misc err change

* adopt comparison_coercion

* Fix comments

* Fix comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

where clause incorrectly reject NULL literal (by SQLancer-NoREC)
2 participants