DataFusion `49.0.0` release post #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

alamb merged 50 commits into apache:main from Omega359:df_49

Jul 28, 2025

Contributor

Omega359 commented Jul 18, 2025 •

edited by alamb

Loading

Posting as a draft as per @alamb's request @ apache/datafusion#16235 (comment)

Omega359 and others added 30 commits

February 20, 2025 12:35


          DF 45 blog post

0d5866f


          Update content/blog/2025-02-20-datafusion-45.0.0.md

7417e4c

Co-authored-by: Yongting You <[email protected]>


          Update content/blog/2025-02-20-datafusion-45.0.0.md

Co-authored-by: Yongting You <[email protected]>


          Set author to PMC.

88d7b6b


          Set author to PMC, incorporated feedback.

5a13332


          Update content/blog/2025-02-20-datafusion-45.0.0.md

b8de014

Co-authored-by: Andrew Lamb <[email protected]>


          expanded GSOC as it may not be obvious what it is and linked it up.

8a46200


          Grammar fix.

5460e1a


          Typo fix

a2e3503


          Typo fix

e8e6734


          Adding spark functions to looking ahead section

7bb8713


          minor change

b330523


          Fixed Jonah Gao's handle.

3b9b11d


          Update content/blog/2025-02-20-datafusion-45.0.0.md

29af566

Co-authored-by: Phillip LeBlanc <[email protected]>


          WIP for DF 49 blog post.

d49a65c


          WIP for DF 49 blog post.


          Update topK dynamic filtering perf section, cleanup the upgrade and c…

6102ade

…hangelog section


          Merge remote-tracking branch 'upstream/main' into origin_main

cbbad27


          DF 47.0.0 blog post

3ece66e


          Remove incomplete and accidentally added DF 49 blog post

ef46a35


          Fix header.

9e0f4e1


          Grammar fix

4f049f4


          Minor formatting

9780da0


          Adding disabling of re-validation of spill files to performance impro…

cdf50f8

…vements


          Merge remote-tracking branch 'apache/main' into Omega359/main

1478e3d


          Formatting and wordsmithing

3859c80


          tweaks

a149ac8


          Update content/blog/2025-07-10-datafusion-47.0.0.md

d433e7d

Co-authored-by: Kevin Liu <[email protected]>


          Fixed link.

af1c645


          Add datafusion-tracing crate mention and logo, make text more concrete

88ad7c6

Contributor

alamb commented Jul 19, 2025

I also added a performance chart from https://alamb.github.io/datafusion-benchmarking/ (I am rechecking these numbers, etc) but it is looking good so far. A preview:


          Reorder sections, add new diagram

alamb mentioned this pull request

Improve async_udf example and docs apache/datafusion#16846

Merged

alamb added 7 commits

July 21, 2025 16:41


          Add note on async udfs

b3e72e5


          update async section

4d08441


          Add note abotu WITHINK GROUP

b0bb055


          note about parquet encryption

c6f7642


          Add spill to disk and regex_instr

44e6e3b


          add regexp_instr

ebb60dc


          Adjust date

4b2a29b

Contributor

alamb commented Jul 21, 2025

😅 -- ok I think I filled out the major content parts of this post. It needs:

More honing / review
Review / rerun the performance chart numbers

Contributor Author

Omega359 commented Jul 21, 2025

Thanks for your help with this @alamb, sorry I haven't done more. I've been deep trying to figure out a performance issue I've been having and haven't had the spare time to put into this

Omega359 marked this pull request as ready for review

July 21, 2025 21:13

Contributor

alamb commented Jul 21, 2025

Thanks for your help with this @alamb, sorry I haven't done more. I've been deep trying to figure out a performance issue I've been having and haven't had the spare time to put into this

I Totally understand -- we are all just trying to scrap together enough time to make progress. It is a struggle for sure! No worries.

All your contributions are appreciated


          Small updates and typo fixes.

15919bc

alamb mentioned this pull request

Support Extension Types / User Defined Types in DataFusion apache/datafusion#12644

Open

alamb added 2 commits

July 23, 2025 16:22


          Wordsmith / OCD obeses

aaa202f


          Gemini AI wordsmith / spelling / style

be89378

Contributor

alamb commented Jul 23, 2025

I took another pass -- I think this post is looking quite good

The major thing that is missing now is actual performance numbers. I will work on that next, likely tomorrow or Friday

Contributor Author

Omega359 commented Jul 23, 2025

Oh my, the ai loves commas, just make sure to leave out any m-dashes :)


          update performance

0ce255f

alamb reviewed

View reviewed changes

Contributor

alamb left a comment

We are getting ready for the DataFusion 49 release and we have several shoutouts in the release notes.

I tagged the relevant people on this PR -- please help / suggest additional comments / content / contributors as you are able

content/blog/2025-07-28-datafusion-49.0.0.md


		Here are some noteworthy optimizations added since DataFusion 48:

		Equivalence system upgrade: The lower levels of the equivalence system, which is used to implement the

Contributor

alamb Jul 25, 2025

content/blog/2025-07-28-datafusion-49.0.0.md

+              Information Passing for joins] (Issue
+              [#15037](https://github.com/apache/datafusion/issues/15037) PR
+              [#15770](https://github.com/apache/datafusion/pull/15770) by
+              [adriangb](https://github.com/adriangb)).

Contributor

alamb Jul 25, 2025

content/blog/2025-07-28-datafusion-49.0.0.md


		The last few months, between `46.0.0` and `49.0.0`, have seen our community grow:

		1. New PMC members and committers: [berkay], [xudong963] and [timsaucer] joined the PMC.

Contributor

alamb Jul 25, 2025

fyi @berkaysynnada @xudong963 @timsaucer @blaginin @milenkovicm @adriangb @kosiew

content/blog/2025-07-28-datafusion-49.0.0.md

+              (Issue [#6518](https://github.com/apache/datafusion/issues/6518),
+              [PR #14837](https://github.com/apache/datafusion/pull/14837) from
+              [goldmedal](https://github.com/goldmedal) 🏆)

Contributor

alamb Jul 25, 2025

content/blog/2025-07-28-datafusion-49.0.0.md

+              [resulting PR]: https://github.com/apache/datafusion/pull/16398
+              [blog post]: https://datafusion.apache.org/blog/2025/06/30/cancellation
+              [contribution upstream to tokio]: https://github.com/tokio-rs/tokio/pull/7405

Contributor

alamb Jul 25, 2025

content/blog/2025-07-28-datafusion-49.0.0.md

+. Pushdown of filters and expressions (see "Dynamic Filters and TopK pushdown" section above)
+              [a long-requested feature]: https://github.com/apache/datafusion/issues/12644
+              [timsaucer]:  https://github.com/timsaucer

Contributor

alamb Jul 25, 2025

content/blog/2025-07-28-datafusion-49.0.0.md

+              );
+              ```
+              ([Issue #15216](https://github.com/apache/datafusion/issues/15216),

Contributor

alamb Jul 25, 2025

FYI @corwinjoy and @adamreeve

content/blog/2025-07-28-datafusion-49.0.0.md

+              (Issue [#11732](https://github.com/apache/datafusion/issues/11732),
+              PR [#13511](https://github.com/apache/datafusion/pull/13511),
+              by [Garamda](https://github.com/Garamda))

Contributor

alamb Jul 25, 2025

content/blog/2025-07-28-datafusion-49.0.0.md

+              [`REGEXP_INSTR` function]: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#regexp-instr
+              ([Issue #13009](https://github.com/apache/datafusion/issues/13009),
+              [PR #15928](https://github.com/apache/datafusion/pull/15928)
+              by [nirnayroy](https://github.com/nirnayroy))

Contributor

alamb Jul 25, 2025

content/blog/2025-07-28-datafusion-49.0.0.md

+              (Issue [#16130](https://github.com/apache/datafusion/issues/16130),
+              PR [#16268](https://github.com/apache/datafusion/pull/16268)
+              by [ding-young](https://github.com/ding-young))

Contributor

alamb Jul 25, 2025

FYI @ding-young

alamb approved these changes

View reviewed changes

alamb mentioned this pull request

Release DataFusion 49.0.0 (July 2025) apache/datafusion#16235

Closed

34 tasks

Contributor

alamb commented Jul 27, 2025

I plan to publish this tomorrow (Monday July 28 2025)

alamb mentioned this pull request

Blog post for the DataFusion 50.0.0 release apache/datafusion#16931

Closed

xudong963 approved these changes

View reviewed changes

Member

xudong963 left a comment

Thank you

Contributor

alamb commented Jul 28, 2025

🚀

alamb merged commit 9dba51d into apache:main

1 check passed

alamb mentioned this pull request

Release DataFusion 50.0.0 (Aug/Sep 2025) apache/datafusion#16799

Closed

45 tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet