Browser-accessible official DataFusion playground / DataFusion fiddle

[findepi]

Is your feature request related to a problem or challenge?

It would be useful to be able to quickly test queries in current DataFusion (latest release or nightly snapshot), without having to check out main branch and running the build.
It would lower the barrier of entry to verifying repro steps in issues, etc.

Describe the solution you'd like

a simple, official website hosted by the DataFusion project allowing to run queries using WASM build of DataFusion

Describe alternatives you've considered

cargo run

Not doing this and requiring contributors to instead

git checkout main && git pull &&
cd datafusion-cli &&
cargo run

This is time consuming, and discourages issue reporters from checking their repro steps.
Or project maintainers from verifying things in current main.

docker image

Having datafusion docker image and using that.
This would be pretty convenient for maintainers, but not less accessible for the contributors or users. Browser-based would be more accessible

Additional context

• [EPIC] A collection of tickets for improved WASM support in DataFusion #13815
• Web Playground with WASM #9834

[alamb]

💯 Thank you @findepi

Here is some inspiration:
https://duckdb.org/docs/api/wasm/overview.html

It would be amazing to have something similar in the datafuion doc site. Something like this:

https://datafusion.apache.org/

[alamb]

As @findepi points out there are at least two implementations of something similar already on #9834

These from @waynexia :

• Demo: https://datafusion-contrib.github.io/datafusion-wasm-playground/
• Source: https://github.com/datafusion-contrib/datafusion-wasm-playground
• https://github.com/datafusion-contrib/datafusion-wasm-bindings

These from @XiangpengHao

• Demo: https://parquet-viewer.haoxp.xyz/
• Code here: https://github.com/XiangpengHao/parquet-viewer/tree/main

[waynexia]

This should greatly reduce the bar of trying out DataFusion 👍

Actually I have tried the "black terminal" style before, as the first version:

It turns out that this style requires a lots of logic in the frontend side to simulate a terminal, which is why I change it to the datafusion-wasm-playground looks like now. But this is not the most important thing, just to mention it.

[XiangpengHao]

It would be useful to be able to quickly test queries in current DataFusion

That's exactly why I started parquet-viewer -- to view paruqet files and experiment with queries without using the cli tool.

If people like it, I can donate it to a more official-looking repo and make it a starting point of datafusion online (or something similar)

[findepi]

parquet viewer is super cool, especially when a user starts with some data in the form of parquet.
I was thinking about something more SQL centric. Most discussions in issues and PRs revolve around SELECT with perhaps CREATE TABLE, etc. It would be nice to be able to run those on current main or last released version, without having to create Parquet file first.

@XiangpengHao Do you think this can be evolved into simple query editor with SELECT and CREATE TABLE capabilities, where Parquet upload is an option to create a table?

[XiangpengHao]

Maybe we are looking for something like this? https://sql-workbench.com

currently parquet viewer is completely in Rust, and we create js binding when we need to interact with browser, so it can be quite challenging to incorporate to many fancy frontend lib, like monaco editor etc...

If we want some cool editor experience, it's better to create a binding of DataFusion and develop from the frontend side, like what @waynexia has done for datafusion-wasm-bindings.
But my frontend knowledge can barely understand what llm generates...

[waynexia]

If people like it, I can donate it to a more official-looking repo and make it a starting point of datafusion online (or something similar)

For the donate part, how about making it the new datafusion-playground. Parquet viewer has implemented a great function set with a more beautiful appearance. Actually I don't want to maintain the old frontend project after seeing the viewer 🙈

[alamb]

I just game across one from @gabotechs : https://datafusion-fiddle.vercel.app/

Repo: https://github.com/gabotechs/datafusion-fiddle

[gabotechs]

Made it a while ago and still use it for quickly trying out stuff, there are lots of low hanging fruits for improving it (local storage based query history, syntax/error highlighting, a proper open source table component, etc...). If there's something I can do to help please let me know.

[findepi]

This looks very cool, @gabotechs! And even better than the CLI itself (#13286) :)
I took the privilege to file the first enhancement idea - gabotechs/datafusion-fiddle#1.