proposal: os: make Readdir return lazy FileInfo implementations

[rsc]

An os.File provides two ways to read a directory: Readdirnames returns a list of the names of the directory entries, and Readdir returns the names along with stat information.

On Plan 9 and Windows, Readdir can be implemented with only a directory read - the directory read operation provides the full stat information.

But many Go users use Unix systems.

On most Unix systems, the directory read does not provide full stat information. So the implementation of Readdir reads the names from the directory and then calls Lstat for each file. This is fairly expensive.

Much of the time, such as in the implementation of filepath.Glob and other file system walking, the only information the caller of Readdir really needs is the name and whether the name denotes a directory. On most Unix systems, that single bit of information—is this name a directory?—is available from the plain directory read, without an additional stat. If the caller is only using that bit, the extra Lstat calls are unnecessary and slow. (Goimports, for example, has its own directory walker to avoid this cost.)

Various people have proposed adding a third directory reading option of one form or another, to get names and IsDir bits. This would certainly address the slow directory walk issue on Unix systems, but it seems like overfitting to Unix.

Note that os.FileInfo is an interface. What if we make Readdir return a slice of lazily-filled os.FileInfo? That is, on Unix, Readdir would stop calling Lstat. Each returned FileInfo would already know the answer for its Name and IsDir methods. The first call to any of the other methods would incur an Lstat at that moment to find out the rest of the information. A directory walk that uses Readdir and then only calls Name and IsDir would have all its Lstat calls optimized away with no code changes in the caller.

The downside of this is that the laziness would be visible when you do the Readdir and wait a while before looking at the results. For example if you did Readdir, then touched one of the files in the list, then called the ModTime method on the os.FileInfo that Readdir retruned, you'd see the updated modification time. And then if you touched the file again and called ModTime again, you wouldn't see the further-updated modification time. That could be confusing. But I expect that the vast majority of uses of Readdir use the results immediately or at least before making changes to files listed in the results. I suspect the vast majority of users would not notice this change.

I propose we make this change—make Readdir return lazy os.FileInfo—soon, intending it to land in Go 1.16, but ready to roll back the change if the remainder of the Go 1.16 dev cycle or beta/rc testing turns up important problems with it.

/cc @robpike @bradfitz @ianthehat @kr

[bradfitz]

Last time this was proposed there was debate about what the behavior for ModTime and Mode and Size should be if the lazy Lstat fails later, as they don't return errors. Panic is bad. Logging is weird. Zero values I guess?

[bcmills]

I think this is likely to introduce subtle changes in behavior. Perhaps more importantly, I don't think this is the sort of change that we can reliably verify during a development cycle.

In my experience, very few users who are not either Go contributors or Googlers test Beta or RC releases of the Go toolchain, and changes in the os package are less likely to turn up during Google testing because a significant fraction of Google programs do most of their I/O without using the os package directly.

[tv42]

os.FileInfo can't be lazy as-is, because it can't return an error. Returning a 0 size on transient errors is not acceptable.

[ianlancetaylor]

See also #40352, which is about different approaches to efficiently uncover similar information.

[tv42]

(Ignoring the POSIX API for a moment) NFS, and likely many other network filesystems, can do completely separate operations depending on whether the stat info is going to be needed (NFSv3 readdir vs readdirplus, NFSv4 "bulk LOOKUP", FUSE_READDIRPLUS).

There's also been a lot of talk about a Linux syscall that would fetch getdents+lstat info, for example https://lwn.net/Articles/606995/ -- they all seem to revolve around the idea of the client knowing beforehand whether it will be doing the lstat calls or not, and communicating that to the kernel.

These combined make me think the right path forward would be a Readdir method that takes arguments that inform it which os.FileInfo fields will be wanted; the rest could be zero values.

(That extended Readdir could also take a flag for whether to sort the results or not, removing one common cause of forks for performance reasons.)

[networkimprov]

EDIT: This has a detailed proposal in #41265

I believe we need a new dirent abstraction.

After reviewing suggestions from the FS API discussion...

• ReadDir() argument gives fields to populate
• DirEntry gets native dirent fields and lazy-loads others; .Has() indicates is-loaded

Let's consider a hybrid:

1. ReadDir(path string, n int, opt uint64) ([]DirItem, error) - opt is fields to load and sorting (0 is OS defaults)
   (may return more fields than requested)
2. (d *DirEntry) Load(fields uint64) error - (re-)loads the fields
   (returns error if inode doesn't match)
3. (d *DirEntry) Has(fields uint64) bool - indicates whether the fields are loaded
4. (d *DirEntry) Id() FileId - gives unix inode or winapi fileId; could take an argument re device info
5. (d *DirEntry) Xyz() T - panics for any field not loaded (a programmer mistake)

That solves the .ModTime() etc issue with lazy-loading, and avoids an error check after every field access.

EDIT: The interface which DirEntry implements and ReadDir() returns:

type DirItem interface {
   Load(fields uint64) error
   Has(fields uint64) bool
   Name() string
   IsDir() bool
}

Rationale:
a) If you need certain fields for every item, request them in ReadDir().
b) If you need certain fields for some items, request them in DirEntry.Load().
c) If you need certain fields only when they're the OS default, check for them with DirEntry.Has().
d) If you need the latest data for an item, request it with DirEntry.Load().

[tv42]

3. (d *DirEntry) Id() FileId - gives unix inode or winapi fileId

Wasn't the Windows FileId 128-bit? (Seen somewhere on go issues around the greater topic in the last few days.)

Either way, the unix inode number isn't very useful without the device major:minor. For example, you can't filepath.Walk and expect inode alone to identify hardlinked files, because you may have crossed a mountpoint.

Also, inode number and such belong in a .Sys() style, platform-specific, extension point.

[networkimprov]

I gave the .Id() type as FileId, which can be whatever (e.g. opaque array), and store major:minor. The fact that a FileId can't be compared across platforms isn't a reason to hide it. It's needed to replicate a tree containing multiple hard links for a file -- which I do.

Today on Windows, FileInfo.Sys() can't even provide the fileId! Adding it was debated and discarded.

Is there any practical value in .Sys() besides providing .Ino on unix?

Winapi fileId is 64-bit on NTFS, 128-bit on ReFS (an alternative for Windows Server):
https://docs.microsoft.com/en-us/windows/win32/api/fileapi/ns-fileapi-by_handle_file_information
https://docs.microsoft.com/en-us/windows-server/storage/refs/refs-overview

[benhoyt]

I really like the intent, but I agree with other commenters that the API just doesn't quite fit as is, because of potential errors returned by the lazy methods. We actually debated a very similar issue when designing the os.scandir / os.DirEntry API in Python. At first we wanted to make the DirEntry methods properties, like entry.stat without the function call parentheses. That works in Python, but it looks like a plain attribute access, and people aren't expecting to have to catch exceptions (errors) when accessing an attribute, so we made it a function call. Per the docs:

Because the os.DirEntry methods can make operating system calls, they may also raise OSError. If you need very fine-grained control over errors, you can catch OSError when calling one of the os.DirEntry methods and handle as appropriate.

I believe this decision was based on theoretical concerns, not from actual testing, but still, the logic seems sound. Especially in Go, where all error handling is super-explicit (we always want "find grained control over errors"). Panic-ing is not going to work, and silently returning a zero value is arguably worse.