`AsyncFileWriter` doesn't need to be `Send`

[jpopesculian]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The AsyncFileWriter doesn't need to be Send and is therefore too restrictive. For context, I want to implement AsyncFileWriter in a WASM context with js_sys types. However, all js_sys types are !Send which makes it impossible to implement.

Describe the solution you'd like

Remove the Send requirement for AsyncFileWriter

Describe alternatives you've considered

I can implement the AsyncFileWriter using only rust types which involves writing to a buffer and waiting for completion, but I don't particularly want to do that in this context because I'd like to upload while writing to reduce memory usage in the browser.

[Xuanwo]

One idea I have is to introduce a MaybeSend, similar to what we did in opendal: https://github.com/apache/opendal/blob/37efe24235388788b892f46e4101d59ecd37918c/core/src/raw/futures_util.rs#L43-L58

This approach will work until we have a true multi-threaded WASM runtime.

It's also worth mentioning that many Parquet users rely on Tokio to spawn tasks for concurrent writing. Simply removing the Send bound could cause breaking changes for our downstream users.

[alamb]

I am hoping a new API we are working on will make this easier: #7983

[alamb]

An update here:

I have made significant progress on a "SansIO" type of API for the parquet readers, described here:

• Decouple IO and CPU operations in the Parquet Reader (push decoder) #7983

The idea is that with those APIs, you could use your own IO routines (aka not the AsyncRead traits).

We recently released a metadata parsing version here:

• https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaDataPushDecoder.html

This lets you decode ParquetMetadata with any IO (e.g this example):

use tokio::io::{AsyncRead, AsyncReadExt, AsyncSeek, AsyncSeekExt};
// This function decodes Parquet Metadata from anything that implements
// [`AsyncRead`] and [`AsyncSeek`] such as a tokio::fs::File
async fn decode_metadata(
  file_len: u64,
  mut async_source: impl AsyncRead + AsyncSeek + Unpin
) -> Result<ParquetMetaData, ParquetError> {
  // We need a ParquetMetaDataPushDecoder to decode the metadata.
  let mut decoder = ParquetMetaDataPushDecoder::try_new(file_len).unwrap();
  loop {
    match decoder.try_decode() {
       Ok(DecodeResult::Data(metadata)) => { return Ok(metadata); } // decode successful
       Ok(DecodeResult::NeedsData(ranges)) => {
          // The decoder needs more data
          //
          // In this example we use the AsyncRead and AsyncSeek traits to read the
          // required ranges from the async source.
          let mut data = Vec::with_capacity(ranges.len());
          for range in &ranges {
            let mut buffer = vec![0; (range.end - range.start) as usize];
            async_source.seek(std::io::SeekFrom::Start(range.start)).await?;
            async_source.read_exact(&mut buffer).await?;
            data.push(Bytes::from(buffer));
          }
          // Push the data into the decoder and try to decode again on the next iteration.
          decoder.push_ranges(ranges, data).unwrap();
       }
       Ok(DecodeResult::Finished) => { unreachable!("returned metadata in previous match arm") }
       Err(e) => return Err(e),
    }
  }
}

I also have code to do the same for the actual parquet decoder here:

• Implement Push Parquet Decoder #7997

Would this work better with wasm?

[alamb]

In terms of writing, I think you could use the lower level ArrowColumnWriter APIs to write to any arbitrary async writer: https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowColumnWriter.html

However, we don't have a great example of doing that yet -- if this sounds promising I can try and write one up