-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Description
Is your feature request related to a problem or challenge?
#12920 added FFI Physical Plans. These cannot be encoded or decoded with the DefaultPhysicalExtensionCodec. We cannot serialize these plans that include foreign elements. I would like to add a way to provide for serialization/deserialization across the FFI boundary.
Describe the solution you'd like
Add an additional struct along the lines of FFI_PhysicalExtensionCodec
that implements PhysicalExtensionCodec
.
Describe alternatives you've considered
None that I can think of, other than to find work arounds to not serialize these plans.
Additional context
This is needed to support adding a FFI_QueryPlanner
so that we can do things like use a different library to generate physical plans. The use case I currently have in mind is datafusion-ray
where we want to make a query planner that can automatically distribute work across multiple nodes in a cluster.