Optimize repartitioning logic in ShuffleWriterExec using interleave_record_batch

[andygrove]

What is the problem the feature request solves?

In ShfufleWriterExec, we copy data from each input RecordBatch into array builders for each partition and once a partition reaches the specified row count, we create an output batch.

Arrow provides an interleave_record_batch kernel that we could potentially use instead of the builder approach.

https://docs.rs/arrow/latest/arrow/compute/fn.interleave_record_batch.html

Describe the potential solution

Let's POC this and see if it helps with performance.

Additional context

No response

[andygrove]

Related: apache/arrow-rs#6692