-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Closed
Labels
enhancementNew feature or requestNew feature or requestgood first issueGood for newcomersGood for newcomers
Description
Is your feature request related to a problem or challenge?
Given the importance of Apache Spark in the analytics ecosystem, many users of DataFusion are interested in Spark compatible functions. We are in the process of adding a Spark compatible function library in the datafusion-spark
crate in DataFusion: https://github.com/apache/datafusion/tree/main/datafusion/spark
This ticket is part of a series of ticket to implement those functions in DataFusion.
Describe the solution you'd like
Implement the function listed above in the datafusion-spark
crate
Describe alternatives you've considered
- Find the relevant test file https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/test_files/spark/datetime/next_day.slt
- Add the function implementation (copy/paste from sail https://github.com/lakehq/sail/blob/main/crates/sail-plan/src/extension/function/datetime/spark_next_day.rs)
- Add additional tests / documentation as needed. Instructions for testing are here and you can see some examples in the PRs linked below
Notes:
- You can run the tests via
cargo test --test sqllogictests -- spark
Examples to follow:
- Here is an example PR that implements the
factorial
function and tests [datafusion-spark] Implementfactorical
function #16125
Reference Implementations
- ScalarFunction Implementations in Sail from: https://github.com/lakehq/sail/tree/main/crates/sail-plan/src/function
- Spark SQL documentation (example, here are the
factorial
docs) - Implementations in Comet (note these do not have the ScalarFunction definitions): https://github.com/apache/datafusion-comet/blob/main/native/spark-expr
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestgood first issueGood for newcomersGood for newcomers