Skip to content

Support creating tables via SQL with FixedSizeList column (e.g. a int[3]) #12810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 9, 2024

Conversation

jandremarais
Copy link
Contributor

@jandremarais jandremarais commented Oct 8, 2024

Which issue does this PR close?

Closes #10303.

Rationale for this change

See #10303.
To make it possible to create a table with a fixed size list column without resorting to casting.

What changes are included in this PR?

Enable SQL query planner to convert the SQL data type Array(ArrayElemTypeDef::SquareBracket) to a FixedSizeList.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

@github-actions github-actions bot added the sql SQL Planner label Oct 8, 2024
@jandremarais
Copy link
Contributor Author

@jayzhan211 @alamb can I please get some guidance on how/where to test that this data type mapping from SQL to DF works as expected?

@jayzhan211
Copy link
Contributor

jayzhan211 commented Oct 8, 2024

The result of arrow_typeof should be fixedsizelist and any non-fixed size list query should failed

query
create table t (a int[3]) as values ([1, 2, 3]), ([4, 5, 6]);
----
Success

query
select arrow_typeof(a) from t;
----
FixedSizeList

query
create table t (a int[3]) as values ([1, 2, 3]), ([4, 5]);
----
Error since the second array size is not 3

@jandremarais
Copy link
Contributor Author

Thanks for your help! Do I add these tests under sqllogictest? If so, is there an existing test_files/*.slt I should add them to? Maybe arrow_typeof.slt?

@alamb
Copy link
Contributor

alamb commented Oct 8, 2024

Thanks for your help! Do I add these tests under sqllogictest? If so, is there an existing test_files/*.slt I should add them to? Maybe arrow_typeof.slt?

Perhaps you can add them to https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/test_files/ddl.slt

Here are the descriptions:
https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest#readme

@github-actions github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Oct 8, 2024
@jandremarais jandremarais changed the title [WIP] Create table with FixedSizeList column Create table with FixedSizeList column Oct 8, 2024
@jandremarais jandremarais marked this pull request as ready for review October 8, 2024 18:22
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice @jandremarais 👏

Thank you!

@alamb alamb changed the title Create table with FixedSizeList column Support creating tables with FixedSizeList column (e.g. a int[3]) Oct 9, 2024
@alamb alamb changed the title Support creating tables with FixedSizeList column (e.g. a int[3]) Support creating tables via SQL with FixedSizeList column (e.g. a int[3]) Oct 9, 2024
@jandremarais
Copy link
Contributor Author

Very nice @jandremarais 👏

Thank you!

Thank you for your friendly support. Looking forward to contribute more

@alamb alamb merged commit c92d303 into apache:main Oct 9, 2024
27 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sql SQL Planner sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create fixed size list table with syntax <type name>[<length>]
3 participants