path/filepath: Walk on very long paths fails on Windows

[sandreas]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.9

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\mediacenter\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

What did you do?

Executing the sample code on a current windows 10 machine fails with an error - seems that running filepath.Walk on very long paths fails on windows in general.

Sample code, which works on play: https://play.golang.org/p/wMS5gti4SD
But it DOES NOT WORK on windows systems!

What did you expect to see?

correctly walk over..../package.json

What did you see instead?

ERROR ON PATH node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json: GetFileAttributesEx node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json: Das System kann den angegebenen Pfad nicht finden.

Can anyone confirm this behaviour?

[mvdan]

This seems similar to #20829. Perhaps there's another part of the standard library that needs this fix for long paths.

[odeke-em]

/cc @alexbrainman @hirochachacha

[alexbrainman]

This fails in os.Lstat(path), and path is relative path here. The os.Lstat is using os.fixLongPath, which has been created just for such purpose. But os.fixLongPath does not handle relative paths, and for a good reason - relative paths cannot be longer than 255 characters (see https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath for details). I do not know how to fix this. Perhaps @quentinmit has some suggestions - he wrote os.fixLongPath.

Alex

[sandreas]

I'm not sure, if this is possible, but i think os.Lstat returns the same value for absolute and relative paths. Is that correct?

If so, why not convert all paths to absolute paths (e.g. with filepath.Abs) in os.Lstat on windows before using the syscall?

[alexbrainman]

why not convert all paths to absolute paths (e.g. with filepath.Abs) in os.Lstat on windows before using the syscall?

Maybe we'll do that. I am not sure yet.

Alex

[sandreas]

As a short feedback: I implemented a simple filesystem abstraction based on afero (https://github.com/spf13/afero) and the use filepath.Abs did the job for me.

I used following logic:
If the path is not absolute and it is longer than 200 chars and it is a windows system, apply filepath.Abs before any other operation.

Why 200 chars, not 255?
Because a relative path containing 227 chars led to an error, while less than 200 did not. I don't know the reason but 255 did not work. To exclude unicode errors, i tried to use special chars in filename (äöü€), but the 227 bound was still valid. The slash-count is 14, 255-14 = 241, so diff is also not the amount of slashes of the relative path.

The problem is, that in later testing i noticed, that there errors occurred with some paths lower than 227 chars, so i corrected it down to 200. Would be nice to know the exact bound, but i didn't find out.

The path with less than 255 chars, which caused the first error was:
relative:
node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json

absolute:
C:\inetpub\wwwroot\irgendwas_4.0\node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json

[quentinmit]

why not convert all paths to absolute paths (e.g. with filepath.Abs) in os.Lstat on windows before using the syscall?

Unfortunately, it's very nontrivial to convert relative paths to absolute paths on Windows. I already went down that road and gave up on trying to make it work. syscall.FullPath, which is what filepath.Abs calls, is also only supported on short paths. What makes this hard is that Windows has a separate working directory on each drive, but getcwd and friends only return one of them. e.g. "d:foo.txt" is a relative path that's relative to whatever the current directory is on the D drive.

If you need to Walk on long paths, you should just call Walk with an absolute path. I believe everything inside Walk should work correctly in that case.

[sandreas]

I already fixed it and for me it is working prepending a \\?\. You said that filepath.Abs only will work for short paths. In my case it does work perfectly with long paths. I'm not saying it is working in every case, but for now i did not find a path (including some node modules directories), that did not work. Code sample (taken from my project https://github.com/sandreas/graft/blob/master/filesystem/osfs_windows.go):

func makeAbsolute(name string) string {
	absolutePath, err := filepath.Abs(name)
	if err == nil {
		if strings.HasPrefix(absolutePath, `\\?\UNC\`) || strings.HasPrefix(absolutePath, `\\?\`) {
			return absolutePath
		}

		if strings.HasPrefix(absolutePath, `\\`) {
			return strings.Replace(absolutePath, `\\`, `\\?\UNC\`, 1)
		}

		return `\\?\` + absolutePath
	}
	return name
}

[alexbrainman]

I already went down that road and gave up on trying to make it work. syscall.FullPath, which is what filepath.Abs calls, is also only supported on short paths.

@quentinmit I just tried to test syscall.FullPath:

c:\Users\Alex>type main.go
package main

import (
        "fmt"
        "log"
        "os"
        "path/filepath"
        "syscall"
)

func makeLongPath() string {
        const file = "abcdefghij1234567890"
        var s string
        for i := 0; i < 13; i++ {
                s = filepath.Join(s, file)
        }
        return s
}

func testPath(path string) {
        fullpath, err := syscall.FullPath(path)
        if err != nil {
                log.Fatal(err)
        }
        fmt.Printf("\nFullPath(%q) is %v\n", path, fullpath)
}

func main() {
        err := os.Chdir(`U:\`)
        if err != nil {
                log.Fatal(err)
        }
        longpath := makeLongPath()
        testPath(`C:` + longpath)
        testPath(`C:\` + longpath)
        testPath(`U:` + longpath)
        testPath(`U:\` + longpath)
        testPath(longpath)
}

c:\Users\Alex>go run main.go

FullPath("C:abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is C:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("C:\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is C:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("U:abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is U:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("U:\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is U:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is U:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

c:\Users\Alex>

and it seems to be working. It works on both Windows 10 and Windows XP.

Alex

[networkimprov]

@gopherbot add "help wanted"

[networkimprov]

@quentinmit what do you think of making os.fixLongPath() return an error for a path over the max length that doesn't start with \\?\ ?

EDIT: this says we can prepend \\?\ to the result of GetFullPathNameW() to create a valid long path: https://stackoverflow.com/questions/38036943/getfullpathnamew-and-long-windows-file-paths

cc @iwdgo

[sandreas]

@networkimprov
See https://github.com/sandreas/afero/blob/master/os_windows.go for an implementation that works for me.

[networkimprov]

@sandreas, according to #21782 (comment) syscall.FullPath doesn't prefix paths on Windows with \\?\ (required for long paths). Has that changed? If not, your code only fixes UNC paths.

Your file is missing import btw.
EDIT: And it's not what you posted in #21782 (comment)

[sandreas]

@networkimprov

according to #21782 (comment) syscall.FullPath doesn't prefix paths on Windows with \?\ (required for long paths). Has that changed? If not, your code only fixes UNC paths.
Your file is missing import btw.
EDIT: And it's not what you posted in #21782 (comment)

You're absolutely right... The code i posted has (accidentally) not been tested on a windows system, where it would belong to - so it is incomplete. I'm sorry that this did not help.

[networkimprov]

I've suggested a fix in #36375 (comment). Both that issue and this one should be fixed in the same CL.

[gopherbot]

Change https://golang.org/cl/263538 mentions this issue: os: on Windows, implement fixLongPath also for relative paths

[gopherbot]

Change https://golang.org/cl/291291 mentions this issue: os: support long paths without fixup on Windows 10 >= 1607

[sandreas]

I would like to mention a highly opinonated article about this topic: https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride
Although it may have its caveats its worth reading...

I would suggest a Path type with ToString and another PathMetaData type to solve this problem on another layer...

[iwdgo]

Issue was probably solved with #20829 and maybe later changes around os.fixLongPath.
The code with a small change to use filepath.Join becomes https://go.dev/play/p/A-GXBqvSxHb?v=gotip
Running the code on Windows 10 succeeds with the following output snipped for readability.
Local directory is obfuscated.

$ gotip version
go version devel go1.22-6e8caefc19 Fri Oct 6 22:31:47 2023 +0000 windows/amd64

$ gotip run longdir.go
Path length 337: C:\Users\Costa\Documents\Google\Issues\21782\node_modules\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef
node_modules/
node_modules\abcdef
node_modules\abcdef\abcdef
...
node_modules\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef
node_modules\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\package.json

$ cat 'node_modules\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\package.json'
$GOPATH\Issues\21782\node_modules\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef\abcdef

Issue seems solved.

[gopherbot]

Change https://go.dev/cl/574695 mentions this issue: os: support relative paths in fixLongPath

[gopherbot]

Change https://go.dev/cl/607437 mentions this issue: os: improve Windows fixLongPath