Blog post about using external indexes with Parquet

[alamb]

Is your feature request related to a problem or challenge?

I would like to write a blog that explains how to use DataFusion and external indexes when reading parquet files

We have some existing example which I think shows how to make them work, lacks sufficient context I think for a wider audience

https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_parquet_index.rs
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/parquet_index.rs

Describe the solution you'd like

I recently did a talk ( video and presentation) where I explained a lot of this, which I think can be turned into a blog pretty reasonably

Describe alternatives you've considered

No response

Additional context

fyi @zhuqi-lucas and @JigaoLuo may also be interested

[zhuqi-lucas]

Thank you @alamb, great idea, i can help review and contribute.

[JigaoLuo]

Thanks! I just skimmed through it quickly. I think it’d be interesting to run an experiment with plots comparing performance with and without the specific indexing to evaluate its impact. I’ll try to carve out some time for it—definitely excited to see what we could find!

[zhuqi-lucas]

Thanks! I just skimmed through it quickly. I think it’d be interesting to run an experiment with plots comparing performance with and without the specific indexing to evaluate its impact. I’ll try to carve out some time for it—definitely excited to see what we could find!

Good idea , may be we can add simple clickbench example, to rewrite with some external indexes, and the performance becomes better?

[JigaoLuo]

Thanks! I just skimmed through it quickly. I think it’d be interesting to run an experiment with plots comparing performance with and without the specific indexing to evaluate its impact. I’ll try to carve out some time for it—definitely excited to see what we could find!

Good idea , may be we can add simple clickbench example, to rewrite with some external indexes, and the performance becomes better?

Yes, exactly. I’ll also go through the video and the draft blog—hopefully that’ll help me come up with something more concrete.

[alamb]

Thanks! I just skimmed through it quickly. I think it’d be interesting to run an experiment with plots comparing performance with and without the specific indexing to evaluate its impact. I’ll try to carve out some time for it—definitely excited to see what we could find!

Here is a crazy specific idea:

I think something that would be broadly interesting (aka get a log of clicks / likely to make Hacker news) would be a some example of fast single row lookups with Parquet -- this is a common usecase that is cited by new file formats

The narrative could go something like

• Parquet is often used for analytical, scan heavy queries and the built in indexing structures and writer properties are optimized for this case
• However, you can combine external indexes and the appropriate writer properties (e.g 100 row pages so the overhead of decoding a page is low) to build a system that works well for single row lookups
• Look we prototyped an example using DataFusion that can do single row lookups in under 1ms with the data is locally in cache

[JigaoLuo]

If I’m not mistaken, that aligns pretty well with my understanding of Lance v2.
Also, I’m not sure if it’s a good idea, but could we try reproducing this experiment setup with a Parquet with a special index — or adapt it: https://blog.lancedb.com/benchmarking-random-access-in-lance/

[alamb]

The blog is ready for review

PR:

• [BLOG] Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet datafusion-site#99

Rendered Preview: https://datafusion.staged.apache.org/blog/2025/08/15/external-parquet-indexes/