Skip to content

Move basic SQL query examples to user guide #11217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions datafusion-examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ cd datafusion
# Download test data
git submodule update --init

# Run the `csv_sql` example:
# Run the `dataframe` example:
# ... use the equivalent for other examples
cargo run --example csv_sql
cargo run --example dataframe
```

## Single Process
Expand All @@ -47,10 +47,8 @@ cargo run --example csv_sql
- [`advanced_udf.rs`](examples/advanced_udf.rs): Define and invoke a more complicated User Defined Scalar Function (UDF)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The removed examples were just inlined into the guide

- [`advanced_udwf.rs`](examples/advanced_udwf.rs): Define and invoke a more complicated User Defined Window Function (UDWF)
- [`advanced_parquet_index.rs`](examples/advanced_parquet_index.rs): Creates a detailed secondary index that covers the contents of several parquet files
- [`avro_sql.rs`](examples/avro_sql.rs): Build and run a query plan from a SQL statement against a local AVRO file
- [`catalog.rs`](examples/catalog.rs): Register the table into a custom catalog
- [`composed_extension_codec`](examples/composed_extension_codec.rs): Example of using multiple extension codecs for serialization / deserialization
- [`csv_sql.rs`](examples/csv_sql.rs): Build and run a query plan from a SQL statement against a local CSV file
- [`csv_sql_streaming.rs`](examples/csv_sql_streaming.rs): Build and run a streaming query plan from a SQL statement against a local CSV file
- [`custom_datasource.rs`](examples/custom_datasource.rs): Run queries against a custom datasource (TableProvider)
- [`dataframe-to-s3.rs`](examples/external_dependency/dataframe-to-s3.rs): Run a query using a DataFrame against a parquet file from s3 and writing back to s3
Expand All @@ -66,12 +64,11 @@ cargo run --example csv_sql
- [`memtable.rs`](examples/memtable.rs): Create an query data in memory using SQL and `RecordBatch`es
- [`optimizer_rule.rs`](examples/optimizer_rule.rs): Use a custom OptimizerRule to replace certain predicates
- [`parquet_index.rs`](examples/parquet_index.rs): Create an secondary index over several parquet files and use it to speed up queries
- [`parquet_sql.rs`](examples/parquet_sql.rs): Build and run a query plan from a SQL statement against a local Parquet file
- [`parquet_sql_multiple_files.rs`](examples/parquet_sql_multiple_files.rs): Build and run a query plan from a SQL statement against multiple local Parquet files
- [`parquet_exec_visitor.rs`](examples/parquet_exec_visitor.rs): Extract statistics by visiting an ExecutionPlan after execution
- [`parse_sql_expr.rs`](examples/parse_sql_expr.rs): Parse SQL text into Datafusion `Expr`.
- [`plan_to_sql.rs`](examples/plan_to_sql.rs): Generate SQL from Datafusion `Expr` and `LogicalPlan`
- [`pruning.rs`](examples/parquet_sql.rs): Use pruning to rule out files based on statistics
- [`pruning.rs`](examples/pruning.rs): Use pruning to rule out files based on statistics
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drive by cleanup

- [`query-aws-s3.rs`](examples/external_dependency/query-aws-s3.rs): Configure `object_store` and run a query against files stored in AWS S3
- [`query-http-csv.rs`](examples/query-http-csv.rs): Configure `object_store` and run a query against files vi HTTP
- [`regexp.rs`](examples/regexp.rs): Examples of using regular expression functions
Expand Down
51 changes: 0 additions & 51 deletions datafusion-examples/examples/avro_sql.rs

This file was deleted.

70 changes: 0 additions & 70 deletions datafusion-examples/examples/csv_sql.rs

This file was deleted.

51 changes: 0 additions & 51 deletions datafusion-examples/examples/parquet_sql.rs

This file was deleted.

6 changes: 6 additions & 0 deletions datafusion/core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -592,3 +592,9 @@ doc_comment::doctest!(
"../../../docs/source/user-guide/example-usage.md",
user_guid_example_tests
);

#[cfg(doctest)]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means these tests are run as part of the doc tests

$ cargo test --doc --features avro,json -- library_user_guide

...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.16s
   Doc-tests datafusion

running 5 tests
test datafusion/core/src/lib.rs - library_user_guide_example_usage (line 764) ... ok
test datafusion/core/src/lib.rs - library_user_guide_example_usage (line 718) ... ok
test datafusion/core/src/lib.rs - library_user_guide_example_usage (line 673) ... ok
test datafusion/core/src/lib.rs - library_user_guide_example_usage (line 775) ... ok
test datafusion/core/src/lib.rs - library_user_guide_example_usage (line 641) ... ok
...

doc_comment::doctest!(
"../../../docs/source/library-user-guide/using-the-sql-api.md",
library_user_guide_example_usage
);
Loading