Variant: Read/Parse JSON value as Variant

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

• Part of [EPIC] [Parquet] Implement Variant type support in Parquet #6736
• Depends on Variant: Rust API to Create Variant Values #7424

A major usecase for Variant values in Parquet and Arrow is efficiently
processing JSON encoded data. Thus an important capability is being able to
efficiently read JSON encoded bytes into the Variant binary encoding described in
VariantEncoding.md. This ticket covers an API to parse one JSON value to
one Variant value. Other tickets will cover converting converting Variants to
JSON as well as converting to/from Arrow Utf8* arrays and Variant arrays as
well as writing this to/from parquet.

Describe the solution you'd like

I would like an API to convert JSON encoded bytes to Variant encoded bytes

Describe alternatives you've considered

I suggest is an API like this:

// Provide location to write metadata, and value output
// (should be anything that implements `std::io::Write` or some trait)
let mut metadata_buffer = vec![];
let mut value_buffer = vec![];
// Input json encoded bytes
let json_data: &[u8] = ...;
// Call the new API
json_to_variant(&mut metadata_buffer, &mut value_buffer, json_data)?;
// metadata_buffer and value_buffer contain the variant information

Prior art:

• @ scovich's PR here: [PATHFINDING] Parse json as variant #7403

Additional context

Considerations:

Reusing metadata across values?

I think it will be common that same metadata is used across many different
variant values (e.g. because the schema of the json documents is the same). Thus
we should probably permit reusing validated metadata somehow (rather than
requiring recreating it for each decoded json value)

One option would be to add a json function to the VariantBuilder envisioned in
the ticket linked above for reading Variant values.

// Location to write metadata
let mut metadata_buffer = vec![]
// Create a builder
let builder = VariantBuilder::new(&mut metadata_buffer);
// Location to write the output variant value
let mut value_buffer = vec![];
builder.json(&mut value_buffer, json_data)?;
// value_buffer contains the result of converting json_data to `Variant`)

Support for "streaming" / a push API

As sketched above, this API would require the entire JSON value in a single
buffer. A potentially more efficient API might be a "push" api, similar to how
the arrow JSON reader works, which would support smaller buffer sizes and lower
peak memory usage as well as interleaving variant parsing with IO fetch.

Perhaps something like

let mut metadata_buffer = vec![]
let builder = VariantBuilder::new(&mut metadata_buffer);
let mut value_buffer = vec![];
let mut parser = builder.json_parser(&mut value_buffer)?;
// json data comes in from some source
while let Some(json_data) = source.next() {
  parser.push(json_data); // incrementally parses json,
}
parser.finish(); // complete in-progress variant
// value_buffer contains the result of converting json_data to Variant)