Async File IO APIs mimicking Win32 OVERLAPPED

[alexbudmsft]

Edit by @carlossanlop: The updated API Proposal is here.

---

It would be great if the .NET framework had async file APIs that closely modelled Win32 OVERLAPPED without the need for building one ourselves with Interop (e.g. https://dschenkelman.github.io/2013/10/29/asynchronous-io-in-c-io-completion-ports/)

Today, the APIs do not take an offset parameter and instead operate on the internal file position. While there are some workarounds (creating multiple Filestream objects and changing their file pointer before issuing the IO) this is unwieldy and inefficient. Ideally there'd be a single FILE_OBJECT (in kernel mode) opened with FILE_FLAG_OVERLAPPED against which you can queue many async IOs where .NET would allocate an OVERLAPPED internally and set the OffsetLow/High fields.

You'd have to integrate this somehow with completions but keep it as flexible as possible. In Win32 you can check for completions via polling (looking at OVERLAPPED.Internal for STATUS_IO_PENDING), waiting on an event (OVERLAPPED.hEvent), waiting on the FILE_OBJECT's internal event (by waiting on the file handle itself), associating the handle with an IOCP and calling GetQueuedCompletionStatus(), and finally putting the thread in an alertable wait state via SleepEx et al and using ReadFileEx.

We don't necessarily have to support all of these, but at the very least polling and event based should be supported somehow. The .NET "Task" model already wraps an event-like scheme with Task.WaitAny() for example, so we could try to plug into that. For polling, we'd want a method on the OVERLAPPED wrapper (whether we use the existing .NET Overlapped class or hide it is up to you) that checks the Internal field (i.e. HasOverlappedIoCompleted macro equivalent).

This will allow a lot more high performance server style applications that care about the full functionality of the Win32 IO model to be written in C# without any hand-rolled Interop.

[adamsitnik]

Hi @alexbudmsft

We are currently working on a FileStream rewrite for .NET 6.0. I've done an initial analysis of the problems that we want to solve and it seems that using OVERLAPPED for not just async (we are already doing that) but also sync IO (we are not doing that yet) might help us solve #16354, #16341, #27047, and #25905.

If I am right, it could mean that we could fulfill your request by adding the following overloads to FileStream:

public partial class FileStream : IO.Stream
{
    public override int Read(byte[] buffer, int offset, int count)
    public override int Read(Span<byte> buffer)
+   public virtual int Read(Span<byte> buffer, long fileOffset)
    public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
    public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken)
+   public virtual ValueTask<int> ReadAsync(Memory<byte> buffer, long fileOffset, CancellationToken cancellationToken)
    public override void Write(byte[] buffer, int offset, int count) { }
    public override void Write(ReadOnlySpan<byte> buffer) { }
+   public virtual void Write(ReadOnlySpan<byte> buffer, long fileOffset) { }    
    public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
    public override ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken)
+   public virtual ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, long fileOffset, CancellationToken cancellationToken)
}

Would that be enough for you as a potential end-user of the feature?

Edit: I think that we can get Position, ReadByte, Seek and WriteByte working fine without throwing.

What would be the expected behavior for you as an end-user? Would throwing for these methods be OK? Or even expected?

This will allow a lot more high performance server style applications that care about the full functionality of the Win32 IO model to be written in C# without any hand-rolled Interop.

Could you say something more about possible usages? I wonder what kind of problems this feature could allow solving. The more I know the better prioritization I can try to get for this request.

[nockawa]

One of the use case I have is about raw data reading/writing where a Stream is not particularly useful: I don't need an object that keeps track of the current Position into the file and updates it. I have no use of it and it is a problem if I want to issue concurrent reads and writes.
So yes, being able to issue read/write specifying the position would be ideal and I'd expect this to be as fast as possible, thread-safe, async, and using overlapped io of course.
We now have a very efficient way to address memory with Span/Memory<T> so I'd say the next step if having a very low level API that perform raw read/write to enable many kind of database related operation where you don't need to particularly process the data, but just read/write lots of them with the highest throughput, async and multithreaded.

[mriehm]

@adamsitnik
Your proposal above shows adding these new APIs just to the FileStream type, but I'm wondering if you are also considering including these in the base Stream type?

Otherwise, if I own an API that takes a Stream parameter, I cannot use these higher performance APIs even if the runtime type of my parameter happens to be FileStream, unless I try casting to FileStream and have a dedicated code path for that case. Though, in the case that the FileStream has been wrapped in some delegating Stream, then even casting to FileStream would not work.
Also, I understand that #48813 is going to disable parallel calls to FileStream.ReadAsync(), with the justification that consumers could switch to this new, higher-performance API to re-enable that behavior. However, it will not be possible for all Stream consumers to switch to this new API if this API is not on the base Stream type.

I'm not sure how much other Stream implementations would benefit from overriding this API. Perhaps something like MemoryStream could improve performance a tiny bit due to not needing to update the position. It could also enable some degree of thread-safety in the case of only readers.
For everything else, this seems a reasonable base Stream implementation:

public virtual int Read(Span<byte> buffer, long fileOffset)
{
    if (!CanSeek)
    {
        throw new NotSupportedException();
    }

    int oldPosition = Position;
    Position = fileOffset;
    int bytesRead = Read(buffer);
    Position = oldPosition;
    return bytesRead;
}

[stephentoub]

I don't think these should be on either Stream or FileStream.

The purpose of Stream is in reading a sequence of data in a linear manner. Concurrent usage of a file where every request is for a specific position is not a stream. It has dubious interactions with any buffering in the instance. It doesn't work with everything the FileStream could wrap (e.g. a pipe). Suggesting these APIs could be used in parallel but other overloads of the same method can't is confusing and likely to lead to production bugs. And then doing so in a way that overrides a base method that isn't also thread-safe is a recipe for more production bugs.

If we want to add APIs for this scenario, I'm against them being instance methods on FileStream. They could be methods related to a SafeFileHandle, on File, or on some new low-level File I/O type.

[nockawa]

@stephentoub I totally agree. Today FileStream is acting as the entry point for disk access and it's stream based, which I can understand. But for low level usage we would ideally need a new set of types, things would be less confusing for sure, and easier to use.

[GSPP]

Maybe there should be a type that contains a file handle and that exposes low-level operations on files. As @stephentoub has pointed out, the Stream interface is not capable of exposing everything that an OS-level file is comprised of. There's at least offset-based IO, but there also is deletion by handle for example and I'm sure there are many things more.

It could be an informative exercise to scan the entire catalog of Windows and Linux file API functions that take a file handle. Any such API potentially is a candidate for being exposed on a low-level file handle type.

[adamsitnik]

If we want to add APIs for this scenario, I'm against them being instance methods on FileStream.

👍

They could be methods related to a SafeFileHandle

To obtain a SafeFileHandle users would need to call native OS API, which would lead to a bad user experience and likely introduce bugs.

The other way is to create a FileStream and then call the SafeFileHandle getter. This is expensive (you need to create a FileStream which is a non-trivial reference type). Users would also need to know that isAsync needs to be set to true in the FileStream ctor.

Moreover, accessing the handle has side-effects, like Flushing the buffered content:

runtime/src/libraries/System.Private.CoreLib/src/System/IO/LegacyFileStreamStrategy.cs

Lines 292 to 300 in 63a5d5b

	internal override SafeFileHandle SafeFileHandle
	{
	get
	{
	Flush();
	_exposedHandle = true;
	return _fileHandle;
	}
	}

Or tracking the file offset:

runtime/src/libraries/System.Private.CoreLib/src/System/IO/LegacyFileStreamStrategy.cs

Lines 311 to 334 in 63a5d5b

	private void VerifyOSHandlePosition()
	{
	bool verifyPosition = _exposedHandle; // in release, only verify if we've given out the handle such that someone else could be manipulating it
	#if DEBUG
	verifyPosition = true; // in debug, always make sure our position matches what the OS says it should be
	#endif
	if (verifyPosition && CanSeek)
	{
	long oldPos = _filePosition; // SeekCore will override the current _position, so save it now
	long curPos = SeekCore(_fileHandle, 0, SeekOrigin.Current);
	if (oldPos != curPos)
	{
	// For reads, this is non-fatal but we still could have returned corrupted
	// data in some cases, so discard the internal buffer. For writes,
	// this is a problem; discard the buffer and error out.
	_readPos = _readLength = 0;
	if (_writePos > 0)
	{
	_writePos = 0;
	throw new IOException(SR.IO_FileStreamHandlePosition);
	}
	}
	}
	}

When we switch to tracking the offset only in memory (#49145), we might need to sync it with OS offset for cases where SafeFileHandle was exposed.

on File

This would require using handles or opening the file every time we want to perform an operation?

or on some new low-level File I/O type.

IMO this is the best solution. We could introduce a very simple API and moreover separate the sync and async implementations:

public sealed class OverlappedFile : IDisposable
{
    public string Path { get; }
    public FileAccess FileAccess { get; }
    public long Length { get; set; }

    public OverlappedFile(string path, FileMode mode = FileMode.Open, FileAccess access = FileAccess.Read, FileShare share = FileShare.Read) { }

    public int Read(Span<byte> buffer, long fileOffset) { };
    public void Write(ReadOnlySpan<byte> buffer, long fileOffset) { }
}

public sealed class AsynchronousOverlappedFile : IDisposable
{
    public string Path { get; }
    public FileAccess FileAccess { get; }
    public long Length { get; set; }

    public AsynchronousOverlappedFile(string path, FileMode mode = FileMode.Open, FileAccess access = FileAccess.Read, FileShare share = FileShare.Read) { }

    public ValueTask<int> ReadAsync(Memory<byte> buffer, long fileOffset, CancellationToken cancellationToken) { }
    public ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, long fileOffset, CancellationToken cancellationToken) { }
}

Since for Windows and Unix the file must be seekable to perform overlapped IO, there is no need for the CanSeek property. I would also prefer not to add handle-based ctor (keep it simple, don't use handles if you don't have to).

The biggest problem for me here is... naming. My current ideas:

• OverlappedFile and AsynchronousOverlappedFile
• RandomAccessFile and AsynchronousRandomAccessFile

[stephentoub]

To obtain a SafeFileHandle users would need to call native OS API, which would lead to a bad user experience and likely introduce bugs.

This whole issue is about introducing new APIs. I'm not sure why new APIs that return a SafeFileHandle would be off the table.

This would require using handles or opening the file every time we want to perform an operation?

Yes, the former.

[adamsitnik]

I'm not sure why new APIs that return a SafeFileHandle would be off the table.

One of the problems with SafeFileHandle is that (at least as of today) it does not contain any information about the type of the handle (file, pipe etc) and whether seeking is allowed or not.

On Windows, this is fetched with an additional syscall:

runtime/src/libraries/System.Private.CoreLib/src/System/IO/LegacyFileStreamStrategy.Windows.cs

Lines 142 to 146 in 29dbd32

	int handleType = Interop.Kernel32.GetFileType(handle);
	Debug.Assert(handleType == Interop.Kernel32.FileTypes.FILE_TYPE_DISK || handleType == Interop.Kernel32.FileTypes.FILE_TYPE_PIPE || handleType == Interop.Kernel32.FileTypes.FILE_TYPE_CHAR, "FileStream was passed an unknown file type!");
	_canSeek = handleType == Interop.Kernel32.FileTypes.FILE_TYPE_DISK;
	_isPipe = handleType == Interop.Kernel32.FileTypes.FILE_TYPE_PIPE;

Which we of course want to avoid in high-perf APIs (not completely, but just do it once for a given file like we do it in FileStream ctor as of today).

If we skip the file type check and just set the offsets for the OVERLAPPED structure, they will be ignored and Windows will just perform the read of the next n bytes, rather than n bytes for given offset:

https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-readfile#parameters

For an hFile that does not support byte offsets, Offset and OffsetHigh are ignored.

So this would give us a silent error.

On Linux, there are some edge cases as well:

https://man7.org/linux/man-pages/man2/pwrite.2.html

POSIX requires that opening a file with the O_APPEND flag should
have no effect on the location at which pwrite() writes data.
However, on Linux, if a file is opened with O_APPEND, pwrite()
appends data to the end of the file, regardless of the value of
offset.

To avoid that, we would have to open the file on our own and ensure that O_APPEND is not set.

Moreover, having a dedicated type that can be more future-proof because allows for a very easy extension of the type and introduction of a new state. Let's say that we want to use new APIs like io_uring which might for example require to have a reference to something like a ring that should be reused for all subsequent operations on the given file.

[mriehm]

I can only speak for my own use cases, but an OverlappedFile type that cannot be constructed with a file handle would not likely be useful to me.

Performance Concerns
While I would of course appreciate higher performance Read() and Write() operations, the performance of opening file handles has also been a significant source of performance problems for my application (exacerbated by conditions such as accessing files over the network or on-access antivirus scanning), which has led me to evolve my code to reuse file handles as much as possible. In cases where .NET does not support an operation by handle (not yet anyway 😉), I'll use FileStream.Handle with calls into native code to perform operations such as deletions, renames, or basic setting of file info. If using an OverlappedFile type meant that I could no longer perform these other operations by handle (or that I need to open both FileStream and OverlappedFile), I would likely consider the performance degradation to outweigh the potential performance improvement. Though perhaps this OverlappedFile type could still be useful for a different application, such as a database, that prioritizes differently between Read()/Write() performance and Open() performance.

Other Concerns
-Lack of atomicity. For example, if I want to open a file, read the contents, and then delete the file, if I have to delete the file by path instead of by handle, how can I guarantee that I am deleting the same file that I just read? Maybe the original file has already been deleted/renamed and replaced? Perhaps most applications would not be concerned with this degree of robustness in the face of concurrent file operations, but my application is. Also, these types of issues can sometimes surface as security vulnerabilities (e.g. reading a file inserted by a malicious actor instead of the file that you intended to read).

-File locking issues. If I have to open multiple file handles to do different operations (write, delete, set file info, etc.), and I want these file handles open concurrently, I cannot open these file handles with very restrictive FileShare options, since all of these handles must be compatible with each other. This means that I cannot as effectively lock other applications out of the files that I am currently modifying.

-Specifying File Open Options Though I understand that your API above is a very preliminary proposal, it doesn't currently provide for specifying FileSystemRights or FileOptions or FileSecurity like can be done for FileStream today. Personally, I actually go to native code today for even more control of creating the file handle. I understand that some options will necessarily be incompatible with any implementation of OverlappedFile, but I would consider using a file handle a sufficiently advanced scenario that it is acceptable to put the responsibility on the consumer to use compatible options, as long as OverlappedFile fails in an obvious way if the file handle wasn't opened with the correct options. For example, I would consider not respecting the fileOffset parameter at all to be an obvious failure that would surely be noticed quickly.

I do recognize your concerns about a lack of encapsulation making future growth more difficult. However, generally, couldn't you just have consumers that pass file handles fall back to an older code path if necessary? I think, it's better to have a usable API with limited growth potential, as opposed to an API that can't be used at all (for my use cases).

Thanks.