Replace parquet metadata thrift version with in memory version.

[liurenjie1024]

In parquet crate, there are two kinds of data structures for metadata: in memory version vs auto generated version from parquet's thrift definition. For example, there are two versions of FileMetadata: in memory vs thrift definition.

We should use the in memory one as it provides more features, while thrift version was only used for ser/de in parquet.

There are several places in our crate which is using thrift version:

1. parquet writer

[jonathanc-n]

@liurenjie1024 I think the current problem with this is that ArrowFileReader (reader) returns ParquetMetadata and AsyncFileWriter (writer) returns the thrift definition. The solution I was thinking of is creating a conversion from thrift -> ParquetMetadata, but this seems like an unnecessary step. I think keeping both functions so that the parquet writer can convert to datafile given any of the two metadatas without an unnecessary conversion step in between seems to be fine.

[liurenjie1024]

@liurenjie1024 I think the current problem with this is that ArrowFileReader (reader) returns ParquetMetadata and AsyncFileWriter (writer) returns the thrift definition. The solution I was thinking of is creating a conversion from thrift -> ParquetMetadata, but this seems like an unnecessary step. I think keeping both functions so that the parquet writer can convert to datafile given any of the two metadatas without an unnecessary conversion step in between seems to be fine.

I think we should always return the in memory representation, rather the thrift one. Is there any case where returning the thrift one is more useful then the in memory one?

[jonathanc-n]

Probably not, so should we cahnge the AsyncFileWriter to return the in memory representation?

[liurenjie1024]

Probably not, so should we cahnge the AsyncFileWriter to return the in memory representation?

Yes, but it seems there is no built no approach to do that? We may need to ask for help in arrow community?

[jonathanc-n]

Yes, I'll look to submit an issue

[liurenjie1024]

Hi, @jonathanc-n I found this method in parquet crate. I think there are two ways to do this:

1. We could use thrift api to serialize thrift version to bytes, and read withi this method.
2. We could simulate the implementation in this method.

[jonathanc-n]

Thanks for that! I'll look into it later today.

[liurenjie1024]

I think this issue is already complete.