[DISCUSSION] Conditional Utf8View support for downstream projects

[comphead]

Is your feature request related to a problem or challenge?

Datafusion Comet encountered a migration issue when upgrading to DataFusion 49, caused by #16290
which introduces Utf8View and replaces Utf8 for md5 function.

Although the runtime behavior is mostly controlled via configuration parameters added in PR #16809 we found that several function signatures have already been hardcoded to Utf8View.

Specifically for Comet project the Utf8View support is still in progress and hardcoding Utf8View makes migrations harder, in this particular case we do a unnecessary cast back to Utf8

It would be great to get more flexible support for Utf8View in DataFusion, ideally in a conditional or opt-in manner, and particularly in builtin functions where it is not feasible to access the SessionContext and make a return type by runtime configuration.

Maybe we can think of features to build the DataFusion with or without Utf8View?

Appreciate your thoughts

Comet migration PR apache/datafusion-comet#2040
Related to #16235

@alamb @andygrove @mbutrovich @zhuqi-lucas

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

I agree we should have a better plan about this

Another thing that I would like to consider is allowing users to pick which of the several string representations to generate code for

Right now, many of our functions have implementations for StringArray, LargeStringArray, StringViewArray. I suspect some will end up with special implementations for DictionaryArray and REEArray as well

This means that we have 3x the code (and compile time, etc). This leads to large wasm sizes, among other problems

• [DISCUSSION] Reduce the DataFusion WASM size #16554

The same thing applies to group by values, sorting, etc that have specialized versions for String and StringView

I wonder if we could/should add config flags for each of the string types 🤔

features=utf8, features=utf8view, etc

However, that sounds like a large testing exercise so maybe it isn't worth the complexity

I would love to hear other ideas

[alamb]

I think @findepi has brought this up when looking at a better way to generate functions

[zhuqi-lucas]

I agree we need a better solution:

1. For high level, currently, we default to mapping sql level string/varchar/char/text to utf8view when we create table, we can use config to control this, but most function has hard coded to return or transform to utf8view.
2. For low level, we only have config to force reading to utf8view for parquet when we infer schema. I am wandering if we can investigate something like force reading to utf8 if we want to transform?
3. The unified solution for all level, i agree @alamb , we may need to have a feature flag, but it's very huge work.

and particularly in builtin functions where it is not feasible to access the SessionContext and make a return type by runtime configuration.

Any other solutions?
Is it possible for us to investigate the SessionContext to support runtime configuration? May be additional field or metadata store, i don't have some ideas about this until now.

[Omega359]

Any other solutions? Is it possible for us to investigate the SessionContext to support runtime configuration? May be additional field or metadata store, i don't have some ideas about this until now.

I think that exists now in different forms. There are VarProviders for runtime variables (@variable and @@variable I think) however those are not accessible to UDFs.

ConfigOptions has Extensions that allow for custom config, however currently that isn't accessible in a UDF either (PR #16970 among others).

For something like that I would think the best option would be a standardized config (datafusion.execution.target_string_type perhaps?) that defaults to Utf8View but can be changed to Utf8 or LargeUtf8. Updating all the UDF's and other code to emit data in the appropriate type would generally be fairly easy but time consuming.

I'm not sold on using feature flags for this tbh as it seems a bit overkill.

[alamb]

In my opinion, given there are many fast and supported ways to convert between string types (e.g. the arrow cast kernels) there is very little value to supporting three different string representations for each function (Utf8, Utf8View and LargeUTF8)

I suggest:

1. we treat the function signatures as "internal" to DataFusion and can change (e.g. only support StringView) which will ease maintenance and allow us to focus our efforts on smaller numbers of implementation
2. Have some options that control the output type (like output_string_type = Utf8) that can control what types are emitted at the output of a plan.
3. Accept any arbitrary type at the input (potentially coercing to Utf8View on input)