os: IsDir doesn't work with Windows OneDrive

[ttrunck]

I think there is an issue with FileInfo.IsDir in Windows 10 using the new Files On-Demand feature of OneDrive. (This occurs only after the Fall Creator Update that introduce this feature).

If I create a folder in OneDrive (with the Files-On-Demand feature on) IsDir return false. Note that just after the creation it return true but once OneDrive finish its sync it then return false. Also if I deactivate the Files-On-Demand feature I no longer repro (for newly created folder, the old one still have the issue). Using the "Always keep on the device option" or the "Free up space" doesn't change the repro.

Also I may miss something obvious, I never used Go before but I found this issue while debugging and issue with Hugo (a site generator in Go).

What version of Go are you using (go version)?

go version go1.9.2 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\theo\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

My OneDrive version is: Version 2017 (Build 17.3.7105.1024)

What did you do?

package main

import (
	"fmt"
	"os"
)

func main() {
	src := "C:\\Users\\theo\\folder"
	srcO := "C:\\Users\\theo\\OneDrive\\folder"
	fi, err := os.Stat(src)
	fmt.Println(fi.IsDir())
	fmt.Println(err)

	fiO, errO := os.Stat(srcO)
	fmt.Println(fiO.IsDir())
	fmt.Println(errO)
}

What did you expect to see?

true
<nil>
true
<nil>

What did you see instead?

true
<nil>
**false**
<nil>

[mvdan]

What does fi0.Mode().String() print?

[bradfitz]

/cc @alexbrainman

[ttrunck]

Thanks for the quick answers.
fmt.Println(fiO.Mode().String()) returns Lrw-rw-rw-

It seems that the issue is here:

go/src/os/types_windows.go

Line 41 in 936b977

if fs.sys.FileAttributes&syscall.FILE_ATTRIBUTE_REPARSE_POINT != 0 {

I'm not sure why OneDrive folders became symlinks so maybe that's not a bug. But this whole behavior breaks assumptions in other programs so that's kind of an issue.

[ttrunck]

Or this condition shouldn't be a return but just a m |= ModeSymlink.

But in that case that contradict this old fix (1989921#diff-befffd222e4e39277f81cd4cb7116a0b).

The fact that OneDrive transform stuff in symlinks will probably cause lots of weird issues.

[ssylvan]

IMHO the ideal situation is that the standard library treats symlinks as transparently as possible, because otherwise many apps doing IO will have to write special code in order to work correctly with symlinks (but since this is subtle, it would probably mean that in practice apps written in Go wouldn't work correctly with symlinks).

While the change introducing this issue was in response to some valid bugs, it would seem preferable that the specific cases where symlinks should not be handled transparently should explicitly check for it (e.g. when doing recursive traversal).

[NexWeb]

With the same configuration, go version go1.9.2 windows/amd64, os.Stat()
returns Directory = true, however Lstat() correctly identifies the symbolic link directory.
File name: folderSymlink
Size in bytes: 0
Permissions: Lrw-rw-rw-
Is Directory: false

IMO, this is appropriate behavior because certain considerations need to be addressed and it provides the opportunity to specify, among other things a FILE_FLAG_OPEN_REPARSE_POINT flag.

Symbolic links, as reparse points, have certain programming considerations specific to them.

• Symbolic links can point to a non-existent target.
• When creating a symbolic link, the operating system does not check to see if the target exists.
• If an application tries to open a non-existent target, ERROR_FILE_NOT_FOUND is returned.
• Symbolic links are reparse points.
• There is a maximum of 31 reparse points (and therefore symbolic links) allowed in a particular path.

The OpenFileById function will either open the file or the reparse point, depending on the use of the FILE_FLAG_OPEN_REPARSE_POINT flag.

Backup applications that use file streams should specify BACKUP_REPARSE_DATA in the WIN32_STREAM_ID structure when backing up files with reparse points.
Applications that use the CreateFile function should specify the FILE_FLAG_OPEN_REPARSE_POINT flag when opening the file, if it is a reparse point.

Also, consider the following information regarding FILE_FLAG_OPEN_REPARSE_POINT:
If FILE_FLAG_OPEN_REPARSE_POINT is specified:
If an existing file is opened and it is a symbolic link, the handle returned is a handle to the symbolic link.
If TRUNCATE_EXISTING or FILE_FLAG_DELETE_ON_CLOSE are specified, the file affected is a symbolic link.
If FILE_FLAG_OPEN_REPARSE_POINT is not specified:
If an existing file is opened and it is a symbolic link, the handle returned is a handle to the target.
If CREATE_ALWAYS, TRUNCATE_EXISTING, or FILE_FLAG_DELETE_ON_CLOSE are specified, the file affected is the target.

var fileInfo os.FileInfo
func main() {
	src := "C:\\Users\\nexweb\\go\\src"
	srcO := "C:\\Users\\nexweb\\OneDrive\\folderSymlink"
	fi, err := os.Stat(src)
	switch mode := fi.Mode(); {
	case mode.IsRegular():
		fmt.Println("regular file")
	case mode.IsDir():
		fmt.Println("directory")
	case mode&os.ModeSymlink != 0:
		fmt.Println("symbolic link")
	case mode&os.ModeNamedPipe != 0:
		fmt.Println("named pipe")
	}
	fmt.Println(err)
	fmt.Println(fi.Mode().String())

	fiO, errO := os.Stat(srcO)
	switch mode := fiO.Mode(); {
	case mode.IsRegular():
		fmt.Println("regular file")
	case mode.IsDir():
		fmt.Println("directory")
	case mode&os.ModeSymlink != 0:
		fmt.Println("symbolic link")
	case mode&os.ModeNamedPipe != 0:
		fmt.Println("named pipe")
	}
	fmt.Println(errO)
	fmt.Println(fiO.Mode().String())
	fileInfo, err = os.Lstat(srcO)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("File name:", fileInfo.Name())
	fmt.Println("Size in bytes:", fileInfo.Size())
	fmt.Println("Permissions:", fileInfo.Mode())
	fmt.Println("Last modified:", fileInfo.ModTime())
	fmt.Println("Is Directory: ", fileInfo.IsDir())
}

[matthijskooijman]

Users of the Arduino IDE are also reporting problems with OneDrive files not being read by the arduino-builder tool, which is written in go. We're tracking the issue at arduino/arduino-builder#254 The problem there is ReadLink failing, which is (probably) different from this issue, but I think the underlying cause might be the same.

I'm not sure why OneDrive folders became symlinks so maybe that's not a bug.

I believe the OneDrive files/folders are not symlinks, but they are "reparse points" (and the most common types of reparse points seem to be symlinks and junctions, both kinds of links that ReadLink now supports explicitely). This post suggests they are "Cloud reparse points", though I'm not sure what that means exactly.

Perhaps this provides some starting points for fixing this (and perhaps also ReadLink)?

[ssylvan]

Interesting.. If they are not true symlinks, then it would seem that would strengthen the case that the IO library should treat them as files transparently. Especially if there are guarantees that "cloud reparse points" will always behave just as files do (in the sense that they're not going to cause loops while listing directory contents, etc.).