Description
Apache Iceberg version
main (development)
Please describe the bug 🐞
Pulling this issue out of #289
In #289, it was discovered that the library cannot write to the Local Filesystem, such as the /tmp/warehouse/ directory.
This is due to the subtle discrepancy between the S3FileSystem and the LocalFilesystem.
S3FileSystem allows writes to arbitrary "path" since it is a blob store.
LocalFilesystem requires the path to be created first before writes can happen.
For example:
In S3, writing to s3://blah/a/b/c/file.txt is allowed even when the parent directory s3://blah/a/b/c/ does not yet exist at the time of writing.
In the local filesystem, writing to /blah/a/b/c/file.txt will fail if the parent directory /blah/a/b/c/ does not yet exist.
This issue is also present when writing data files and metadata files that require subdirectories to exist first, such as
{warehouse}/data/blah.parquet and {warehouse}/metadata/blah.metadata