Skip to content

always running processes from 1.15.3 #17138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dannyzcy opened this issue Sep 24, 2021 · 42 comments
Closed

always running processes from 1.15.3 #17138

dannyzcy opened this issue Sep 24, 2021 · 42 comments
Labels
type/bug

Comments

@dannyzcy
Copy link

Gitea Version

1.15.3

Git Version

2.33.0.windows.2

Operating System

windows

How are you running Gitea?

running gitea by gitea-1.15.3-windows-4.0-amd64.exe

Database

MySQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Description

always running processes from 1.15.3

the bug from 1.15.3

not appear in 1.15.2

Screenshots

未命名1
未命名
@lunny lunny added the type/bug label Sep 24, 2021
@zeripath
Copy link
Contributor

zeripath commented Sep 24, 2021 •
edited
Loading

This is a duplicate report.

Upgrade to 1.15.4 (agh not awake!) - go to the admin monitor page and at the bottom there is a process manager.

That will then tell you what bit of code opened these processes and we'll be able to figure out where the missing close is.

@zeripath
Copy link
Contributor

zeripath commented Sep 24, 2021 •
edited
Loading

As this is from the latest version and I can't find the issue number (right now) that this duplicates - I am going to reopen.

@dannyzcy - could you please go to the admin page I mentioned above and show me the processes table at the bottom of the monitor page.

That will allow us to figure out what is opening these processes.

Then we will need to correlate the times with the logs as per https://docs.gitea.io/en-us/logging-configuration/#debugging-problems

@zeripath zeripath reopened this Sep 24, 2021
@dannyzcy
Copy link
Author

dannyzcy commented Sep 24, 2021 •
edited
Loading

The picture above may already be the processes table at the bottom of the monitor page? right?

image

logs:

2021/09/24 15:42:30 ...ters/private/serv.go:380:ServCommand() [D] Serv Results:
	IsWiki: false
	IsDeployKey: false
	KeyID: 31	KeyName: rsa-key-20210922
	UserName: 1234567
	UserID: 40
	OwnerName: test
	RepoName: test
	RepoID: 179
2021/09/24 15:42:30 ...s/context/context.go:740:1() [D] Session ID: dd55bf8aa898ea07
2021/09/24 15:42:30 ...s/context/context.go:741:1() [D] CSRF Token: cqktVIq6OCIK2gNAMb36lLv4JY86MTYzMjQ2OTM1MDE3MjI2MjUwMA
2021/09/24 15:42:30 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\test\test.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/24 15:42:30 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\test\test.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2021/09/24 15:42:31 modules/ssh/ssh.go:150:publicKeyHandler() [D] Handle Public Key: Fingerprint: SHA256:rcLnaYptHJVEGdltjzt1P0isyRNB9pi02rpJKCc41RA from 10.123.123.140:57680
2021/09/24 15:42:31 modules/ssh/ssh.go:231:publicKeyHandler() [D] Handle Public Key: 10.123.123.140:57680 Fingerprint: SHA256:rcLnaYptHJVEGdltjzt1P0isyRNB9pi02rpJKCc41RA is not a certificate
2021/09/24 15:42:31 modules/ssh/ssh.go:248:publicKeyHandler() [D] Successfully authenticated: 10.123.123.140:57680 Public Key Fingerprint: SHA256:rcLnaYptHJVEGdltjzt1P0isyRNB9pi02rpJKCc41RA

@wxiaoguang
Copy link
Contributor

wxiaoguang commented Sep 24, 2021 •
edited
Loading

I just tried, can not reproduce this problem on my side (Gitea 1.15.3 + Windows)

May you get a full stack trace? It would help much.

Steps:

  1. Set app.ini:
[server]
ENABLE_PPROF = true

  1. Restart Gitea

  2. Try to trigger the bug, when the processes get stuck for a while, use curl or browser to visit: http://127.0.0.1:6060/debug/pprof/goroutine?debug=1

  3. Report the output stack trace here (the stack trace doesn't contain sensitive data).

@zeripath
Copy link
Contributor

That process list indicates to me that there's a deadlock somewhere in the hook.

I'm confused as to how or where.

I guess we're going to need to add some extra logging into:

gitea/routers/private/hook.go

Lines 133 to 399 in 7ce938b

gitRepo, err := git.OpenRepository(repo.RepoPath())
if err != nil {
log.Error("Unable to get git repository for: %s/%s Error: %v", ownerName, repoName, err)
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: err.Error(),
})
return
}
defer gitRepo.Close()
// Generate git environment for checking commits
env := os.Environ()
if opts.GitAlternativeObjectDirectories != "" {
env = append(env,
private.GitAlternativeObjectDirectories+"="+opts.GitAlternativeObjectDirectories)
}
if opts.GitObjectDirectory != "" {
env = append(env,
private.GitObjectDirectory+"="+opts.GitObjectDirectory)
}
if opts.GitQuarantinePath != "" {
env = append(env,
private.GitQuarantinePath+"="+opts.GitQuarantinePath)
}
protectedTags, err := repo.GetProtectedTags()
if err != nil {
log.Error("Unable to get protected tags for %-v Error: %v", repo, err)
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: err.Error(),
})
return
}
// Iterate across the provided old commit IDs
for i := range opts.OldCommitIDs {
oldCommitID := opts.OldCommitIDs[i]
newCommitID := opts.NewCommitIDs[i]
refFullName := opts.RefFullNames[i]
if strings.HasPrefix(refFullName, git.BranchPrefix) {
branchName := strings.TrimPrefix(refFullName, git.BranchPrefix)
if branchName == repo.DefaultBranch && newCommitID == git.EmptySHA {
log.Warn("Forbidden: Branch: %s is the default branch in %-v and cannot be deleted", branchName, repo)
ctx.JSON(http.StatusForbidden, private.Response{
Err: fmt.Sprintf("branch %s is the default branch and cannot be deleted", branchName),
})
return
}
protectBranch, err := models.GetProtectedBranchBy(repo.ID, branchName)
if err != nil {
log.Error("Unable to get protected branch: %s in %-v Error: %v", branchName, repo, err)
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: err.Error(),
})
return
}
// Allow pushes to non-protected branches
if protectBranch == nil || !protectBranch.IsProtected() {
continue
}
// This ref is a protected branch.
//
// First of all we need to enforce absolutely:
//
// 1. Detect and prevent deletion of the branch
if newCommitID == git.EmptySHA {
log.Warn("Forbidden: Branch: %s in %-v is protected from deletion", branchName, repo)
ctx.JSON(http.StatusForbidden, private.Response{
Err: fmt.Sprintf("branch %s is protected from deletion", branchName),
})
return
}
// 2. Disallow force pushes to protected branches
if git.EmptySHA != oldCommitID {
output, err := git.NewCommand("rev-list", "--max-count=1", oldCommitID, "^"+newCommitID).RunInDirWithEnv(repo.RepoPath(), env)
if err != nil {
log.Error("Unable to detect force push between: %s and %s in %-v Error: %v", oldCommitID, newCommitID, repo, err)
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: fmt.Sprintf("Fail to detect force push: %v", err),
})
return
} else if len(output) > 0 {
log.Warn("Forbidden: Branch: %s in %-v is protected from force push", branchName, repo)
ctx.JSON(http.StatusForbidden, private.Response{
Err: fmt.Sprintf("branch %s is protected from force push", branchName),
})
return
}
}
// 3. Enforce require signed commits
if protectBranch.RequireSignedCommits {
err := verifyCommits(oldCommitID, newCommitID, gitRepo, env)
if err != nil {
if !isErrUnverifiedCommit(err) {
log.Error("Unable to check commits from %s to %s in %-v: %v", oldCommitID, newCommitID, repo, err)
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: fmt.Sprintf("Unable to check commits from %s to %s: %v", oldCommitID, newCommitID, err),
})
return
}
unverifiedCommit := err.(*errUnverifiedCommit).sha
log.Warn("Forbidden: Branch: %s in %-v is protected from unverified commit %s", branchName, repo, unverifiedCommit)
ctx.JSON(http.StatusForbidden, private.Response{
Err: fmt.Sprintf("branch %s is protected from unverified commit %s", branchName, unverifiedCommit),
})
return
}
}
// Now there are several tests which can be overridden:
//
// 4. Check protected file patterns - this is overridable from the UI
changedProtectedfiles := false
protectedFilePath := ""
globs := protectBranch.GetProtectedFilePatterns()
if len(globs) > 0 {
_, err := pull_service.CheckFileProtection(oldCommitID, newCommitID, globs, 1, env, gitRepo)
if err != nil {
if !models.IsErrFilePathProtected(err) {
log.Error("Unable to check file protection for commits from %s to %s in %-v: %v", oldCommitID, newCommitID, repo, err)
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: fmt.Sprintf("Unable to check file protection for commits from %s to %s: %v", oldCommitID, newCommitID, err),
})
return
}
changedProtectedfiles = true
protectedFilePath = err.(models.ErrFilePathProtected).Path
}
}
// 5. Check if the doer is allowed to push
canPush := false
if opts.IsDeployKey {
canPush = !changedProtectedfiles && protectBranch.CanPush && (!protectBranch.EnableWhitelist || protectBranch.WhitelistDeployKeys)
} else {
canPush = !changedProtectedfiles && protectBranch.CanUserPush(opts.UserID)
}
// 6. If we're not allowed to push directly
if !canPush {
// Is this is a merge from the UI/API?
if opts.PullRequestID == 0 {
// 6a. If we're not merging from the UI/API then there are two ways we got here:
//
// We are changing a protected file and we're not allowed to do that
if changedProtectedfiles {
log.Warn("Forbidden: Branch: %s in %-v is protected from changing file %s", branchName, repo, protectedFilePath)
ctx.JSON(http.StatusForbidden, private.Response{
Err: fmt.Sprintf("branch %s is protected from changing file %s", branchName, protectedFilePath),
})
return
}
// Or we're simply not able to push to this protected branch
log.Warn("Forbidden: User %d is not allowed to push to protected branch: %s in %-v", opts.UserID, branchName, repo)
ctx.JSON(http.StatusForbidden, private.Response{
Err: fmt.Sprintf("Not allowed to push to protected branch %s", branchName),
})
return
}
// 6b. Merge (from UI or API)
// Get the PR, user and permissions for the user in the repository
pr, err := models.GetPullRequestByID(opts.PullRequestID)
if err != nil {
log.Error("Unable to get PullRequest %d Error: %v", opts.PullRequestID, err)
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: fmt.Sprintf("Unable to get PullRequest %d Error: %v", opts.PullRequestID, err),
})
return
}
user, err := models.GetUserByID(opts.UserID)
if err != nil {
log.Error("Unable to get User id %d Error: %v", opts.UserID, err)
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: fmt.Sprintf("Unable to get User id %d Error: %v", opts.UserID, err),
})
return
}
perm, err := models.GetUserRepoPermission(repo, user)
if err != nil {
log.Error("Unable to get Repo permission of repo %s/%s of User %s", repo.OwnerName, repo.Name, user.Name, err)
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: fmt.Sprintf("Unable to get Repo permission of repo %s/%s of User %s: %v", repo.OwnerName, repo.Name, user.Name, err),
})
return
}
// Now check if the user is allowed to merge PRs for this repository
allowedMerge, err := pull_service.IsUserAllowedToMerge(pr, perm, user)
if err != nil {
log.Error("Error calculating if allowed to merge: %v", err)
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: fmt.Sprintf("Error calculating if allowed to merge: %v", err),
})
return
}
if !allowedMerge {
log.Warn("Forbidden: User %d is not allowed to push to protected branch: %s in %-v and is not allowed to merge pr #%d", opts.UserID, branchName, repo, pr.Index)
ctx.JSON(http.StatusForbidden, private.Response{
Err: fmt.Sprintf("Not allowed to push to protected branch %s", branchName),
})
return
}
// If we're an admin for the repository we can ignore status checks, reviews and override protected files
if perm.IsAdmin() {
continue
}
// Now if we're not an admin - we can't overwrite protected files so fail now
if changedProtectedfiles {
log.Warn("Forbidden: Branch: %s in %-v is protected from changing file %s", branchName, repo, protectedFilePath)
ctx.JSON(http.StatusForbidden, private.Response{
Err: fmt.Sprintf("branch %s is protected from changing file %s", branchName, protectedFilePath),
})
return
}
// Check all status checks and reviews are ok
if err := pull_service.CheckPRReadyToMerge(pr, true); err != nil {
if models.IsErrNotAllowedToMerge(err) {
log.Warn("Forbidden: User %d is not allowed push to protected branch %s in %-v and pr #%d is not ready to be merged: %s", opts.UserID, branchName, repo, pr.Index, err.Error())
ctx.JSON(http.StatusForbidden, private.Response{
Err: fmt.Sprintf("Not allowed to push to protected branch %s and pr #%d is not ready to be merged: %s", branchName, opts.PullRequestID, err.Error()),
})
return
}
log.Error("Unable to check if mergable: protected branch %s in %-v and pr #%d. Error: %v", opts.UserID, branchName, repo, pr.Index, err)
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: fmt.Sprintf("Unable to get status of pull request %d. Error: %v", opts.PullRequestID, err),
})
return
}
}
} else if strings.HasPrefix(refFullName, git.TagPrefix) {
tagName := strings.TrimPrefix(refFullName, git.TagPrefix)
isAllowed, err := models.IsUserAllowedToControlTag(protectedTags, tagName, opts.UserID)
if err != nil {
ctx.JSON(http.StatusInternalServerError, private.Response{
Err: err.Error(),
})
return
}
if !isAllowed {
log.Warn("Forbidden: Tag %s in %-v is protected", tagName, repo)
ctx.JSON(http.StatusForbidden, private.Response{
Err: fmt.Sprintf("Tag %s is protected", tagName),
})
return
}
}
}
ctx.PlainText(http.StatusOK, []byte("ok"))
}

@zeripath
Copy link
Contributor

@dannyzcy if I were able to provide patches would you be able to recompile gitea and report the logs? The PPROF results that @wxiaoguang mentioned would also be helpful.

@dannyzcy
Copy link
Author

dannyzcy commented Sep 24, 2021 •
edited
Loading

@wxiaoguang The logs is as follows:

http://127.0.0.1:6060/debug/pprof/goroutine?debug=1

goroutine profile: total 45
4 @ 0x10e041a 0x10f1468 0x26f7cb9 0x1116541
#	0x26f7cb8	github.com/blevesearch/bleve_index_api.AnalysisWorker+0x98	/source/vendor/github.com/blevesearch/bleve_index_api/analysis.go:46

3 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119c176 0x11d8fd6 0x11ebed8 0x146d7ff 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64		/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b		/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112			/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119c175	internal/poll.(*FD).Read+0x2f5			/usr/local/go/src/internal/poll/fd_windows.go:441
#	0x11d8fd5	net.(*netFD).Read+0x55				/usr/local/go/src/net/fd_posix.go:55
#	0x11ebed7	net.(*conn).Read+0x97				/usr/local/go/src/net/net.go:183
#	0x146d7fe	net/http.(*connReader).backgroundRead+0x5e	/usr/local/go/src/net/http/server.go:672

3 @ 0x10e041a 0x10f1468 0x2f6b4d1 0x2d409dc 0x14756cb 0x23d432b 0x14756cb 0x2d410ca 0x14756cb 0x2d410ca 0x14756cb 0x2d410ca 0x14756cb 0x2d64782 0x14756cb 0x2d410ca 0x14756cb 0x23f55ca 0x14756cb 0x23d1e2b 0x23d82dc 0x14756cb 0x23d432b 0x14756cb 0x2f76c58 0x14756cb 0x21d5f2f 0x14756cb 0x23474f1 0x14756cb 0x23d1e2b 0x23d82dc
#	0x2f6b4d0	code.gitea.io/gitea/routers/web/events.Events+0x7b0		/source/routers/web/events/events.go:77
#	0x2d409db	code.gitea.io/gitea/modules/web.Wrap.func1+0x1fb		/source/modules/web/route.go:64
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a			/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2d410c9	code.gitea.io/gitea/modules/web.Middle.func1.1+0x149		/source/modules/web/route.go:103
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2d410c9	code.gitea.io/gitea/modules/web.Middle.func1.1+0x149		/source/modules/web/route.go:103
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2d410c9	code.gitea.io/gitea/modules/web.Middle.func1.1+0x149		/source/modules/web/route.go:103
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2d64781	github.com/go-chi/chi/middleware.GetHead.func1+0x181		/source/vendor/github.com/go-chi/chi/middleware/get_head.go:37
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2d410c9	code.gitea.io/gitea/modules/web.Middle.func1.1+0x149		/source/modules/web/route.go:103
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23f55c9	code.gitea.io/gitea/modules/context.Contexter.func1.1+0x1a69	/source/modules/context/context.go:776
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23d1e2a	github.com/go-chi/chi.(*Mux).ServeHTTP+0x5aa			/source/vendor/github.com/go-chi/chi/mux.go:70
#	0x23d82db	github.com/go-chi/chi.(*Mux).Mount.func1+0x17b			/source/vendor/github.com/go-chi/chi/mux.go:311
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a			/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2f76c57	code.gitea.io/gitea/routers/web.Recovery.func1.1+0x97		/source/routers/web/base.go:186
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x21d5f2e	gitea.com/go-chi/session.Sessioner.func1.1+0x24e		/source/vendor/gitea.com/go-chi/session/session.go:256
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23474f0	code.gitea.io/gitea/modules/public.AssetsHandler.func1.1+0x90	/source/modules/public/public.go:42
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23d1e2a	github.com/go-chi/chi.(*Mux).ServeHTTP+0x5aa			/source/vendor/github.com/go-chi/chi/mux.go:70
#	0x23d82db	github.com/go-chi/chi.(*Mux).Mount.func1+0x17b			/source/vendor/github.com/go-chi/chi/mux.go:311

2 @ 0x10a4d55 0x1112e12 0x116e1c5 0x1165eaf 0x119c0a5 0x11a5b05 0x11a5ad1 0x115c0cc 0x12f432a 0x12f42fa 0x12f43ce 0x1116541
#	0x1112e11	syscall.Syscall6+0xf1				/usr/local/go/src/runtime/syscall_windows.go:343
#	0x116e1c4	syscall.ReadFile+0x104				/usr/local/go/src/syscall/zsyscall_windows.go:1006
#	0x1165eae	syscall.Read+0x6e				/usr/local/go/src/syscall/syscall_windows.go:369
#	0x119c0a4	internal/poll.(*FD).Read+0x224			/usr/local/go/src/internal/poll/fd_windows.go:427
#	0x11a5b04	os.(*File).read+0x84				/usr/local/go/src/os/file_posix.go:31
#	0x11a5ad0	os.(*File).Read+0x50				/usr/local/go/src/os/file.go:117
#	0x115c0cb	io.copyBuffer+0x12b				/usr/local/go/src/io/io.go:423
#	0x12f4329	io.Copy+0x69					/usr/local/go/src/io/io.go:382
#	0x12f42f9	os/exec.(*Cmd).writerDescriptor.func1+0x39	/usr/local/go/src/os/exec/exec.go:311
#	0x12f43cd	os/exec.(*Cmd).Start.func1+0x2d			/usr/local/go/src/os/exec/exec.go:441

2 @ 0x10e041a 0x10f1468 0x14b7dec 0x1116541
#	0x14b7deb	code.gitea.io/gitea/modules/log.(*ChannelledLog).Start+0xab	/source/modules/log/event.go:70

2 @ 0x10e041a 0x10f1468 0x14b93a5 0x1116541
#	0x14b93a4	code.gitea.io/gitea/modules/log.(*MultiChannelledLog).Start+0x364	/source/modules/log/event.go:300

2 @ 0x10e041a 0x10f1468 0x192a74a 0x1116541
#	0x192a749	code.gitea.io/gitea/modules/graceful.(*Server).awaitShutdown+0xe9	/source/modules/graceful/server_hooks.go:16

1 @ 0x10ae5d3 0x11118ef 0x19252c9 0x1116541
#	0x11118ee	os/signal.signal_recv+0xae	/usr/local/go/src/runtime/sigqueue.go:168
#	0x19252c8	os/signal.loop+0x28		/usr/local/go/src/os/signal/signal_unix.go:23

1 @ 0x10e041a 0x10a7f3a 0x10a7b6b 0x12f21cc 0x16be80a 0x16bf893 0x16bf7f7 0x16bf7f8 0x1700f0f 0x1116541
#	0x12f21cb	os/exec.(*Cmd).Wait+0x14b								/usr/local/go/src/os/exec/exec.go:515
#	0x16be809	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutEnvFullPipelineFunc+0x4e9	/source/modules/git/command.go:168
#	0x16bf892	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutEnvFullPipeline+0xd2		/source/modules/git/command.go:107
#	0x16bf7f6	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutFullPipeline+0x36		/source/modules/git/command.go:184
#	0x16bf7f7	code.gitea.io/gitea/modules/git.(*Command).RunInDirFullPipeline+0x37			/source/modules/git/command.go:220
#	0x1700f0e	code.gitea.io/gitea/modules/git.CatFileBatchCheck.func2+0x3ae				/source/modules/git/batch_reader.go:52

1 @ 0x10e041a 0x10a7f3a 0x10a7b6b 0x15b2b51 0x15b1f14 0x1116541
#	0x15b2b50	golang.org/x/crypto/ssh.(*handshakeTransport).readOnePacket+0x1d0	/source/vendor/golang.org/x/crypto/ssh/handshake.go:410
#	0x15b1f13	golang.org/x/crypto/ssh.(*handshakeTransport).readLoop+0x53		/source/vendor/golang.org/x/crypto/ssh/handshake.go:197

1 @ 0x10e041a 0x10a7f3a 0x10a7b6b 0x1700b46 0x16dc6f4 0x2eb31d4 0x2d40b9c 0x14756cb 0x23d432b 0x14756cb 0x2ec0aab 0x14756cb 0x23f6f30 0x14756cb 0x23d1e2b 0x23d82dc 0x14756cb 0x23d432b 0x14756cb 0x2d69305 0x14756cb 0x2d68529 0x14756cb 0x2d650c8 0x14756cb 0x2d605f4 0x14756cb 0x2d68e48 0x14756cb 0x23d1bb1 0x2d3fefb 0x1ede55b
#	0x1700b45	code.gitea.io/gitea/modules/git.CatFileBatchCheck.func1+0xe5		/source/modules/git/batch_reader.go:42
#	0x16dc6f3	code.gitea.io/gitea/modules/git.(*Repository).Close+0x53		/source/modules/git/repo_base_nogogit.go:86
#	0x2eb31d3	code.gitea.io/gitea/routers/private.HookPreReceive+0x4553		/source/routers/private/hook.go:399
#	0x2d40b9b	code.gitea.io/gitea/modules/web.Wrap.func1+0x3bb			/source/modules/web/route.go:76
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a				/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2ec0aaa	code.gitea.io/gitea/routers/private.CheckInternalToken.func1+0x22a	/source/routers/private/internal.go:31
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23f6f2f	code.gitea.io/gitea/modules/context.PrivateContexter.func1.1+0x16f	/source/modules/context/private.go:42
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d1e2a	github.com/go-chi/chi.(*Mux).ServeHTTP+0x5aa				/source/vendor/github.com/go-chi/chi/mux.go:70
#	0x23d82db	github.com/go-chi/chi.(*Mux).Mount.func1+0x17b				/source/vendor/github.com/go-chi/chi/mux.go:311
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a				/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d69304	code.gitea.io/gitea/routers/common.Middlewares.func2.1+0x84		/source/routers/common/middleware.go:72
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d68528	code.gitea.io/gitea/routers/common.LoggerHandler.func1.1+0x368		/source/routers/common/logger.go:23
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d650c7	github.com/go-chi/chi/middleware.StripSlashes.func1+0x107		/source/vendor/github.com/go-chi/chi/middleware/strip.go:30
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d605f3	github.com/chi-middleware/proxy.ForwardedHeaders.func1.1+0xf3		/source/vendor/github.com/chi-middleware/proxy/middleware.go:37
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d68e47	code.gitea.io/gitea/routers/common.Middlewares.func1.1+0x67		/source/routers/common/middleware.go:25
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d1bb0	github.com/go-chi/chi.(*Mux).ServeHTTP+0x330				/source/vendor/github.com/go-chi/chi/mux.go:87
#	0x2d3fefa	code.gitea.io/gitea/modules/web.(*Route).ServeHTTP+0x5a			/source/modules/web/route.go:318
#	0x1ede55a	github.com/gorilla/context.ClearHandler.func1+0x7a			/source/vendor/github.com/gorilla/context/context.go:141

1 @ 0x10e041a 0x10a7f3a 0x10a7bab 0x15b1d15 0x15b1ce9 0x15cbed0 0x15cb86e 0x2d2fcc9 0x1116541
#	0x15b1d14	golang.org/x/crypto/ssh.(*handshakeTransport).readPacket+0x54	/source/vendor/golang.org/x/crypto/ssh/handshake.go:187
#	0x15b1ce8	golang.org/x/crypto/ssh.(*handshakeTransport).waitSession+0x28	/source/vendor/golang.org/x/crypto/ssh/handshake.go:154
#	0x15cbecf	golang.org/x/crypto/ssh.(*connection).serverHandshake+0x28f	/source/vendor/golang.org/x/crypto/ssh/server.go:251
#	0x15cb86d	golang.org/x/crypto/ssh.NewServerConn+0x18d			/source/vendor/golang.org/x/crypto/ssh/server.go:206
#	0x2d2fcc8	github.com/gliderlabs/ssh.(*Server).HandleConn+0x188		/source/vendor/github.com/gliderlabs/ssh/server.go:281

1 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119c176 0x11d8fd6 0x11ebed8 0x146dd7f 0x12daf6d 0x12dbd65 0x12dbfbb 0x13faa0b 0x1467f31 0x1467f32 0x146f26b 0x1473fc5 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64		/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b		/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112			/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119c175	internal/poll.(*FD).Read+0x2f5			/usr/local/go/src/internal/poll/fd_windows.go:441
#	0x11d8fd5	net.(*netFD).Read+0x55				/usr/local/go/src/net/fd_posix.go:55
#	0x11ebed7	net.(*conn).Read+0x97				/usr/local/go/src/net/net.go:183
#	0x146dd7e	net/http.(*connReader).Read+0x1be		/usr/local/go/src/net/http/server.go:780
#	0x12daf6c	bufio.(*Reader).fill+0x10c			/usr/local/go/src/bufio/bufio.go:101
#	0x12dbd64	bufio.(*Reader).ReadSlice+0x44			/usr/local/go/src/bufio/bufio.go:360
#	0x12dbfba	bufio.(*Reader).ReadLine+0x3a			/usr/local/go/src/bufio/bufio.go:389
#	0x13faa0a	net/textproto.(*Reader).readLineSlice+0xea	/usr/local/go/src/net/textproto/reader.go:57
#	0x1467f30	net/textproto.(*Reader).ReadLine+0xb0		/usr/local/go/src/net/textproto/reader.go:38
#	0x1467f31	net/http.readRequest+0xb1			/usr/local/go/src/net/http/request.go:1027
#	0x146f26a	net/http.(*conn).readRequest+0x1aa		/usr/local/go/src/net/http/server.go:966
#	0x1473fc4	net/http.(*conn).serve+0x704			/usr/local/go/src/net/http/server.go:1858

1 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119c176 0x11d8fd6 0x11ebed8 0x2d2d7ca 0x12db658 0x115bb4e 0x15abb47 0x15abaf7 0x15d272f 0x15d2645 0x15b867b 0x15b458d 0x15b3e0f 0x15b244f 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64					/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b					/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112						/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119c175	internal/poll.(*FD).Read+0x2f5						/usr/local/go/src/internal/poll/fd_windows.go:441
#	0x11d8fd5	net.(*netFD).Read+0x55							/usr/local/go/src/net/fd_posix.go:55
#	0x11ebed7	net.(*conn).Read+0x97							/usr/local/go/src/net/net.go:183
#	0x2d2d7c9	github.com/gliderlabs/ssh.(*serverConn).Read+0x69			/source/vendor/github.com/gliderlabs/ssh/conn.go:28
#	0x12db657	bufio.(*Reader).Read+0x237						/usr/local/go/src/bufio/bufio.go:227
#	0x115bb4d	io.ReadAtLeast+0x8d							/usr/local/go/src/io/io.go:328
#	0x15abb46	io.ReadFull+0x86							/usr/local/go/src/io/io.go:347
#	0x15abaf6	golang.org/x/crypto/ssh.(*streamPacketCipher).readCipherPacket+0x36	/source/vendor/golang.org/x/crypto/ssh/cipher.go:153
#	0x15d272e	golang.org/x/crypto/ssh.(*connectionState).readPacket+0x6e		/source/vendor/golang.org/x/crypto/ssh/transport.go:130
#	0x15d2644	golang.org/x/crypto/ssh.(*transport).readPacket+0x44			/source/vendor/golang.org/x/crypto/ssh/transport.go:114
#	0x15b867a	golang.org/x/crypto/ssh.(*curve25519sha256).Server+0x5a			/source/vendor/golang.org/x/crypto/ssh/kex.go:490
#	0x15b458c	golang.org/x/crypto/ssh.(*handshakeTransport).server+0x1ac		/source/vendor/golang.org/x/crypto/ssh/handshake.go:622
#	0x15b3e0e	golang.org/x/crypto/ssh.(*handshakeTransport).enterKeyExchange+0x38e	/source/vendor/golang.org/x/crypto/ssh/handshake.go:585
#	0x15b244e	golang.org/x/crypto/ssh.(*handshakeTransport).kexLoop+0x1ae		/source/vendor/golang.org/x/crypto/ssh/handshake.go:301

1 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119e69c 0x119e968 0x11da9c5 0x11f3c39 0x11f282c 0x1929fc7 0x14790fc 0x1929adc 0x1928b7a 0x192af65 0x30a896f 0x30a88f9 0x30a79b9 0x241f0d0 0x241ceb5 0x30e4dca 0x10dfff6 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64					/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b					/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112						/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119e69b	internal/poll.(*FD).acceptOne+0x9b					/usr/local/go/src/internal/poll/fd_windows.go:810
#	0x119e967	internal/poll.(*FD).Accept+0x167					/usr/local/go/src/internal/poll/fd_windows.go:844
#	0x11da9c4	net.(*netFD).accept+0x84						/usr/local/go/src/net/fd_windows.go:139
#	0x11f3c38	net.(*TCPListener).accept+0x38						/usr/local/go/src/net/tcpsock_posix.go:139
#	0x11f282b	net.(*TCPListener).AcceptTCP+0x6b					/usr/local/go/src/net/tcpsock.go:248
#	0x1929fc6	code.gitea.io/gitea/modules/graceful.(*wrappedListener).Accept+0x66	/source/modules/graceful/server.go:215
#	0x14790fb	net/http.(*Server).Serve+0x29b						/usr/local/go/src/net/http/server.go:2961
#	0x1929adb	code.gitea.io/gitea/modules/graceful.(*Server).Serve+0x17b		/source/modules/graceful/server.go:168
#	0x1928b79	code.gitea.io/gitea/modules/graceful.(*Server).ListenAndServe+0x1f9	/source/modules/graceful/server.go:96
#	0x192af64	code.gitea.io/gitea/modules/graceful.HTTPListenAndServe+0xa4		/source/modules/graceful/server_http.go:33
#	0x30a896e	code.gitea.io/gitea/cmd.runHTTP+0x58e					/source/cmd/web_graceful.go:20
#	0x30a88f8	code.gitea.io/gitea/cmd.listen+0x518					/source/cmd/web.go:208
#	0x30a79b8	code.gitea.io/gitea/cmd.runWeb+0x4f8					/source/cmd/web.go:159
#	0x241f0cf	github.com/urfave/cli.HandleAction+0x10f				/source/vendor/github.com/urfave/cli/app.go:524
#	0x241ceb4	github.com/urfave/cli.(*App).Run+0x674					/source/vendor/github.com/urfave/cli/app.go:286
#	0x30e4dc9	main.main+0xa69								/source/main.go:115
#	0x10dfff5	runtime.main+0x255							/usr/local/go/src/runtime/proc.go:225

1 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119e69c 0x119e968 0x11da9c5 0x11f3c39 0x11f282c 0x1929fc7 0x2d2f893 0x1929adc 0x1928b7a 0x2d39038 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64					/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b					/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112						/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119e69b	internal/poll.(*FD).acceptOne+0x9b					/usr/local/go/src/internal/poll/fd_windows.go:810
#	0x119e967	internal/poll.(*FD).Accept+0x167					/usr/local/go/src/internal/poll/fd_windows.go:844
#	0x11da9c4	net.(*netFD).accept+0x84						/usr/local/go/src/net/fd_windows.go:139
#	0x11f3c38	net.(*TCPListener).accept+0x38						/usr/local/go/src/net/tcpsock_posix.go:139
#	0x11f282b	net.(*TCPListener).AcceptTCP+0x6b					/usr/local/go/src/net/tcpsock.go:248
#	0x1929fc6	code.gitea.io/gitea/modules/graceful.(*wrappedListener).Accept+0x66	/source/modules/graceful/server.go:215
#	0x2d2f892	github.com/gliderlabs/ssh.(*Server).Serve+0x152				/source/vendor/github.com/gliderlabs/ssh/server.go:237
#	0x1929adb	code.gitea.io/gitea/modules/graceful.(*Server).Serve+0x17b		/source/modules/graceful/server.go:168
#	0x1928b79	code.gitea.io/gitea/modules/graceful.(*Server).ListenAndServe+0x1f9	/source/modules/graceful/server.go:96
#	0x2d39037	code.gitea.io/gitea/modules/ssh.listen+0xb7				/source/modules/ssh/ssh_graceful.go:20

1 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119e69c 0x119e968 0x11da9c5 0x11f3c39 0x11f29eb 0x14790fc 0x1478e05 0x30aaf1c 0x30aaf1d 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64		/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b		/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112			/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119e69b	internal/poll.(*FD).acceptOne+0x9b		/usr/local/go/src/internal/poll/fd_windows.go:810
#	0x119e967	internal/poll.(*FD).Accept+0x167		/usr/local/go/src/internal/poll/fd_windows.go:844
#	0x11da9c4	net.(*netFD).accept+0x84			/usr/local/go/src/net/fd_windows.go:139
#	0x11f3c38	net.(*TCPListener).accept+0x38			/usr/local/go/src/net/tcpsock_posix.go:139
#	0x11f29ea	net.(*TCPListener).Accept+0x6a			/usr/local/go/src/net/tcpsock.go:261
#	0x14790fb	net/http.(*Server).Serve+0x29b			/usr/local/go/src/net/http/server.go:2961
#	0x1478e04	net/http.(*Server).ListenAndServe+0xc4		/usr/local/go/src/net/http/server.go:2890
#	0x30aaf1b	net/http.ListenAndServe+0x9b			/usr/local/go/src/net/http/server.go:3144
#	0x30aaf1c	code.gitea.io/gitea/cmd.runWeb.func2+0x9c	/source/cmd/web.go:142

1 @ 0x10e041a 0x10f1468 0x150381b 0x1116541
#	0x150381a	database/sql.(*DB).connectionCleaner+0x11a	/usr/local/go/src/database/sql/sql.go:1005

1 @ 0x10e041a 0x10f1468 0x15042bc 0x1116541
#	0x15042bb	database/sql.(*DB).connectionOpener+0xbb	/usr/local/go/src/database/sql/sql.go:1133

1 @ 0x10e041a 0x10f1468 0x18243ca 0x1116541
#	0x18243c9	github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain+0x109	/source/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206

1 @ 0x10e041a 0x10f1468 0x184afe5 0x1116541
#	0x184afe4	github.com/syndtr/goleveldb/leveldb.(*DB).compactionError+0xc4	/source/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90

1 @ 0x10e041a 0x10f1468 0x18505cb 0x1116541
#	0x18505ca	github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction+0x12a	/source/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:762

1 @ 0x10e041a 0x10f1468 0x18508cf 0x1116541
#	0x18508ce	github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction+0x22e	/source/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:825

1 @ 0x10e041a 0x10f1468 0x1854a7b 0x1116541
#	0x1854a7a	github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain+0xda	/source/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101

1 @ 0x10e041a 0x10f1468 0x192595b 0x1116541
#	0x192595a	golang.org/x/sys/windows/svc/debug.Run.func1+0xba	/source/vendor/golang.org/x/sys/windows/svc/debug/service.go:32

1 @ 0x10e041a 0x10f1468 0x19277bf 0x1925837 0x192c2fb 0x1116541
#	0x19277be	code.gitea.io/gitea/modules/graceful.(*Manager).Execute+0x23e		/source/modules/graceful/manager_windows.go:132
#	0x1925836	golang.org/x/sys/windows/svc/debug.Run+0x156				/source/vendor/golang.org/x/sys/windows/svc/debug/service.go:40
#	0x192c2fa	code.gitea.io/gitea/modules/graceful.(*Manager).start.func1+0x5a	/source/modules/graceful/manager_windows.go:99

1 @ 0x10e041a 0x10f1468 0x1f36627 0x1116541
#	0x1f36626	github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1+0xa6	/source/vendor/github.com/go-sql-driver/mysql/connection.go:614

1 @ 0x10e041a 0x10f1468 0x25d040a 0x1925ed0 0x1116541
#	0x25d0409	code.gitea.io/gitea/services/webhook.DeliverHooks+0x269				/source/services/webhook/deliver.go:221
#	0x1925ecf	code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownContext+0xaf	/source/modules/graceful/manager.go:139

1 @ 0x10e041a 0x10f1468 0x2b502e8 0x2b4fba5 0x1925c38 0x1116541
#	0x2b502e7	code.gitea.io/gitea/modules/queue.(*ByteFIFOQueue).readToChan+0x427		/source/modules/queue/queue_bytefifo.go:152
#	0x2b4fba4	code.gitea.io/gitea/modules/queue.(*ByteFIFOQueue).Run+0x2a4			/source/modules/queue/queue_bytefifo.go:121
#	0x1925c37	code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownFns+0x117	/source/modules/graceful/manager.go:83

1 @ 0x10e041a 0x10f1468 0x2d13a7e 0x1116541
#	0x2d13a7d	github.com/gogs/cron.(*Cron).run+0x29d	/source/vendor/github.com/gogs/cron/cron.go:197

1 @ 0x10e041a 0x10f1468 0x2d18950 0x1925ed0 0x1116541
#	0x2d1894f	code.gitea.io/gitea/services/mirror.syncMirrors+0xef				/source/services/mirror/mirror.go:72
#	0x1925ecf	code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownContext+0xaf	/source/modules/graceful/manager.go:139

1 @ 0x10e041a 0x10f1468 0x2d29f1a 0x1925ed0 0x1116541
#	0x2d29f19	code.gitea.io/gitea/modules/eventsource.(*Manager).Run+0xf9			/source/modules/eventsource/manager_run.go:32
#	0x1925ecf	code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownContext+0xaf	/source/modules/graceful/manager.go:139

1 @ 0x110f345 0x2cd88d5 0x2cd8687 0x2cd5158 0x2d10925 0x2d121a9 0x14756cb 0x1477597 0x1478d0a 0x147418d 0x1116541
#	0x110f344	runtime/pprof.runtime_goroutineProfileWithLabels+0x64	/usr/local/go/src/runtime/mprof.go:716
#	0x2cd88d4	runtime/pprof.writeRuntimeProfile+0xd4			/usr/local/go/src/runtime/pprof/pprof.go:724
#	0x2cd8686	runtime/pprof.writeGoroutine+0xa6			/usr/local/go/src/runtime/pprof/pprof.go:684
#	0x2cd5157	runtime/pprof.(*Profile).WriteTo+0x3f7			/usr/local/go/src/runtime/pprof/pprof.go:331
#	0x2d10924	net/http/pprof.handler.ServeHTTP+0x384			/usr/local/go/src/net/http/pprof/pprof.go:253
#	0x2d121a8	net/http/pprof.Index+0x8e8				/usr/local/go/src/net/http/pprof/pprof.go:371
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a			/usr/local/go/src/net/http/server.go:2049
#	0x1477596	net/http.(*ServeMux).ServeHTTP+0x1b6			/usr/local/go/src/net/http/server.go:2428
#	0x1478d09	net/http.serverHandler.ServeHTTP+0xa9			/usr/local/go/src/net/http/server.go:2867
#	0x147418c	net/http.(*conn).serve+0x8cc				/usr/local/go/src/net/http/server.go:1932

1 @ 0x146d7a1 0x1116541
#	0x146d7a0	net/http.(*connReader).backgroundRead+0x0	/usr/local/go/src/net/http/server.go:671

@dannyzcy
Copy link
Author

@dannyzcy if I were able to provide patches would you be able to recompile gitea and report the logs?

I can't always reproduce this bug

sometime appear sometime disappear

and not always the same repository

I don't know how to recompile gitea

sorry

@lunny
Copy link
Member

lunny commented Sep 24, 2021

Is it blocked by internal push update router?

@dannyzcy
Copy link
Author

未命名

New discovery different source: services/pull/pull.go:212

PPROF results:

goroutine profile: total 55
8 @ 0x10a4d55 0x1112e12 0x116e1c5 0x1165eaf 0x119c0a5 0x11a5b05 0x11a5ad1 0x115c0cc 0x12f432a 0x12f42fa 0x12f43ce 0x1116541
#	0x1112e11	syscall.Syscall6+0xf1				/usr/local/go/src/runtime/syscall_windows.go:343
#	0x116e1c4	syscall.ReadFile+0x104				/usr/local/go/src/syscall/zsyscall_windows.go:1006
#	0x1165eae	syscall.Read+0x6e				/usr/local/go/src/syscall/syscall_windows.go:369
#	0x119c0a4	internal/poll.(*FD).Read+0x224			/usr/local/go/src/internal/poll/fd_windows.go:427
#	0x11a5b04	os.(*File).read+0x84				/usr/local/go/src/os/file_posix.go:31
#	0x11a5ad0	os.(*File).Read+0x50				/usr/local/go/src/os/file.go:117
#	0x115c0cb	io.copyBuffer+0x12b				/usr/local/go/src/io/io.go:423
#	0x12f4329	io.Copy+0x69					/usr/local/go/src/io/io.go:382
#	0x12f42f9	os/exec.(*Cmd).writerDescriptor.func1+0x39	/usr/local/go/src/os/exec/exec.go:311
#	0x12f43cd	os/exec.(*Cmd).Start.func1+0x2d			/usr/local/go/src/os/exec/exec.go:441

4 @ 0x10e041a 0x10a7f3a 0x10a7b6b 0x12f21cc 0x16be80a 0x16bf893 0x16bf7f7 0x16bf7f8 0x1700f0f 0x1116541
#	0x12f21cb	os/exec.(*Cmd).Wait+0x14b								/usr/local/go/src/os/exec/exec.go:515
#	0x16be809	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutEnvFullPipelineFunc+0x4e9	/source/modules/git/command.go:168
#	0x16bf892	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutEnvFullPipeline+0xd2		/source/modules/git/command.go:107
#	0x16bf7f6	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutFullPipeline+0x36		/source/modules/git/command.go:184
#	0x16bf7f7	code.gitea.io/gitea/modules/git.(*Command).RunInDirFullPipeline+0x37			/source/modules/git/command.go:220
#	0x1700f0e	code.gitea.io/gitea/modules/git.CatFileBatchCheck.func2+0x3ae				/source/modules/git/batch_reader.go:52

4 @ 0x10e041a 0x10f1468 0x26f7cb9 0x1116541
#	0x26f7cb8	github.com/blevesearch/bleve_index_api.AnalysisWorker+0x98	/source/vendor/github.com/blevesearch/bleve_index_api/analysis.go:46

3 @ 0x10e041a 0x10a7f3a 0x10a7b6b 0x1700b46 0x16dc6f4 0x2eb31d4 0x2d40b9c 0x14756cb 0x23d432b 0x14756cb 0x2ec0aab 0x14756cb 0x23f6f30 0x14756cb 0x23d1e2b 0x23d82dc 0x14756cb 0x23d432b 0x14756cb 0x2d69305 0x14756cb 0x2d68529 0x14756cb 0x2d650c8 0x14756cb 0x2d605f4 0x14756cb 0x2d68e48 0x14756cb 0x23d1bb1 0x2d3fefb 0x1ede55b
#	0x1700b45	code.gitea.io/gitea/modules/git.CatFileBatchCheck.func1+0xe5		/source/modules/git/batch_reader.go:42
#	0x16dc6f3	code.gitea.io/gitea/modules/git.(*Repository).Close+0x53		/source/modules/git/repo_base_nogogit.go:86
#	0x2eb31d3	code.gitea.io/gitea/routers/private.HookPreReceive+0x4553		/source/routers/private/hook.go:399
#	0x2d40b9b	code.gitea.io/gitea/modules/web.Wrap.func1+0x3bb			/source/modules/web/route.go:76
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a				/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2ec0aaa	code.gitea.io/gitea/routers/private.CheckInternalToken.func1+0x22a	/source/routers/private/internal.go:31
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23f6f2f	code.gitea.io/gitea/modules/context.PrivateContexter.func1.1+0x16f	/source/modules/context/private.go:42
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d1e2a	github.com/go-chi/chi.(*Mux).ServeHTTP+0x5aa				/source/vendor/github.com/go-chi/chi/mux.go:70
#	0x23d82db	github.com/go-chi/chi.(*Mux).Mount.func1+0x17b				/source/vendor/github.com/go-chi/chi/mux.go:311
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a				/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d69304	code.gitea.io/gitea/routers/common.Middlewares.func2.1+0x84		/source/routers/common/middleware.go:72
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d68528	code.gitea.io/gitea/routers/common.LoggerHandler.func1.1+0x368		/source/routers/common/logger.go:23
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d650c7	github.com/go-chi/chi/middleware.StripSlashes.func1+0x107		/source/vendor/github.com/go-chi/chi/middleware/strip.go:30
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d605f3	github.com/chi-middleware/proxy.ForwardedHeaders.func1.1+0xf3		/source/vendor/github.com/chi-middleware/proxy/middleware.go:37
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d68e47	code.gitea.io/gitea/routers/common.Middlewares.func1.1+0x67		/source/routers/common/middleware.go:25
#	0x14756ca

@zeripath
Copy link
Contributor

zeripath commented Sep 24, 2021 •
edited
Loading

I think we might need to pass the outputs of the profiles through pprof ourselves.

if you give us the files obtained from:

curl 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1' -o goroutine.out
curl 'http://127.0.0.1:6060/debug/pprof/profile?seconds=30' -o profile.out
curl 'http://127.0.0.1:6060/debug/pprof/trace?seconds=30' -o traceprofile.out
curl 'http://127.0.0.1:6060/debug/pprof/block?debug=1' -o block.out
curl 'http://127.0.0.1:6060/debug/pprof/mutex?debug=1' -o mutex.out

We may be able to divine where the deadlock is occurring.

you may need to zip these up.

@dannyzcy
Copy link
Author

Gitea.zip

5 files are in zip file, thanks

@wxiaoguang
Copy link
Contributor

wxiaoguang commented Sep 24, 2021 •
edited
Loading

From the goroutine list, I guess:

  1. HookPreReceive has reached return and gitRepo.Close has been called
  2. the cancel of CatFileBatch has been called, the stdin/stdout pipes are closed, but chan closed is not closed
  3. why chan closed is not closed: the git process doesn't exit after the pipes are closed, so it gets stuck in RunInDirTimeoutEnvFullPipelineFunc : cmd.Wait of modules/git/command.go:168

I am not sure about pipe behaviors in Windows. If the pipes in Linux are closed, the git process will exit. But the above stack trace shows that even the pipes in Windows are closed, the git process still exists. Or maybe the pipes are not really closed?

@zeripath
Copy link
Contributor

  1. the cancel of CatFileBatch has been called, the stdin/stdout pipes are closed, but chan closed is not closed

So that cancel will cancel the parent context of the running git process which should cause it to be killed.

@zeripath
Copy link
Contributor

There is clearly some problem with the git process not being killed or terminated despite us cancelling everything.

I don't think that cat-file usually has an associated autogc - I guess that could be a cause?!

We could add some belt and braces cmd.Process.Release() and ...Kill() ?

@wxiaoguang
Copy link
Contributor

I have no idea right now.

Release won't stop the child process, if there is a bug on our side or on git side, the git process is still there, it still wastes system resources.

Kill calls TerminateProcess, which is not graceful.

Maybe we can try Kill (with error logs) when we can confirm that the child process gets stuck:

close pipes
select {
   case <-closed:
   case <-time.After(10s):
       log.Error(...)
       cmd.Process.Kill()
}

And maybe we need more logs when running git cli.

@zeripath
Copy link
Contributor

git cat-file --batch will not report any of this stuff over stdout and stderr as it would break the format.

@zeripath
Copy link
Contributor

zeripath commented Sep 25, 2021 •
edited
Loading

I just can't think of why windows would keep the process around - could it be that there's unread content in one of the pipes and that's doing it?!

I think TerminateProcess is fine in this case because cat-file --batch and cat-file --batch-check are supposed to be read-only - the only q is if they're doing an autogc and that's responsible for the issue. (They don't.)

@zeripath
Copy link
Contributor

#17138 (comment)

This would be fine for the cat-file --batch and batch-check commands but without more information as to what's causing the problem I can't help but worry that other processes are going to hang too.

I do wonder if the issue is something to do with the pipes. One thing we do is we use io.Pipe() in the command because I thought that this would avoid an os.Pipe() however looking at the code for exec this does not appear to be the case - and instead the os.Pipe is just hidden deeper in the system.

So @dannyzcy are you able to run a few tests for us?

I can provide two PRs (and provide builds of):

  • One with the above cmd.Process.Kill() on a timeout to see if that actually solves problem
  • But also one with the os.Pipe() used directly to see if directly closing these prevents them problem from happening.

@wxiaoguang
Copy link
Contributor

I can provide two PRs (and provide builds of):

* One with the above `cmd.Process.Kill()` on a timeout to see if that actually solves problem

* But also one with the `os.Pipe()` used directly to see if directly closing these prevents them problem from happening.

I think these two patches can be tried in one build, just add some error logs before Kill.

ps: I haven't noticed the difference between 'io.Pipe' and 'os.Pipe', it surprised me ....

@dannyzcy
Copy link
Author

So @dannyzcy are you able to run a few tests for us?

I can provide two PRs (and provide builds of):

  • One with the above cmd.Process.Kill() on a timeout to see if that actually solves problem
  • But also one with the os.Pipe() used directly to see if directly closing these prevents them problem from happening.

Of course, my pleasure.

zeripath added a commit to zeripath/gitea that referenced this issue Sep 25, 2021
@zeripath
Prevent dangling cat-files [16:34:43]
481b030 
In go-gitea#17138 it appears that on Windows a git cat-file can fail to be killed.

This PR attempts to prevent this by:

1. Using os.Pipes for the input - this should mean that the stdin is definitely closed.
2. If the cat-file is doesn't close properly it will attempt to kill it repeatedly logging this.

Fix go-gitea#17138

Signed-off-by: Andrew Thornton <[email protected]>
@zeripath zeripath mentioned this issue Sep 25, 2021
Closed
zeripath added a commit to zeripath/gitea that referenced this issue Sep 25, 2021
@zeripath
Prevent dangling cat-files (go-gitea#17154)
7d871cc 
Backport go-gitea#17154

In go-gitea#17138 it appears that on Windows a git cat-file can fail to be killed.

This PR attempts to prevent this by:

1. Using os.Pipes for the input - this should mean that the stdin is definitely closed.
2. If the cat-file is doesn't close properly it will attempt to kill it repeatedly logging this.

Fix go-gitea#17138

Signed-off-by: Andrew Thornton <[email protected]>
@zeripath zeripath mentioned this issue Sep 25, 2021
Closed
@zeripath
Copy link
Contributor

https://eldritchkitty.com/~andrew/gitea-v1.15.3-5-ga31966ed8-pr-17155-windows-4.0-amd64.exe.xz

@dannyzcy
Copy link
Author

The first push is the same problem, but I tried to push the second time and it was successful

未命名
2

logs:


2021/09/26 06:53:33 modules/ssh/ssh.go:150:publicKeyHandler() [D] Handle Public Key: Fingerprint: SHA256:lu+r7jlRkAvUNPoiBJset0gjVR1xDW6qpFBvkVB/Tpc from 10.1.123.456:49537
2021/09/26 06:53:33 modules/ssh/ssh.go:231:publicKeyHandler() [D] Handle Public Key: 10.1.123.456:49537 Fingerprint: SHA256:lu+r7jlRkAvUNPoiBJset0gjVR1xDW6qpFBvkVB/Tpc is not a certificate
2021/09/26 06:53:33 modules/ssh/ssh.go:248:publicKeyHandler() [D] Successfully authenticated: 10.1.123.456:49537 Public Key Fingerprint: SHA256:lu+r7jlRkAvUNPoiBJset0gjVR1xDW6qpFBvkVB/Tpc
2021/09/26 06:53:39 ...ters/private/serv.go:380:ServCommand() [D] Serv Results:
	IsWiki: false
	IsDeployKey: false
	KeyID: 14	KeyName: [email protected]
	UserName: danny
	UserID: 2
	OwnerName: s
	RepoName: s
	RepoID: 28
2021/09/26 06:53:40 ...dules/git/command.go:121:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\s\s.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/26 06:53:40 ...dules/git/command.go:121:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\s\s.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2021/09/26 06:53:50 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:53:50 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:54:00 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:54:00 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:54:10 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:54:10 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:54:20 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:54:20 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:54:30 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:54:30 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:54:40 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:54:40 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:54:50 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:54:50 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:54:51 ...s/context/context.go:740:1() [D] Session ID: edccbd0acb8edcba
2021/09/26 06:54:51 ...s/context/context.go:741:1() [D] CSRF Token: tepTdbRPsT3Uo-NS2JC6uYbXAhU6MTYzMjUyNTY4NjE2NTgyNTgwMA
2021/09/26 06:54:51 ...s/context/context.go:740:1() [D] Session ID: edccbd0acb8edcba
2021/09/26 06:54:51 ...s/context/context.go:741:1() [D] CSRF Token: tepTdbRPsT3Uo-NS2JC6uYbXAhU6MTYzMjUyNTY4NjE2NTgyNTgwMA
2021/09/26 06:54:51 ...s/context/context.go:740:1() [D] Session ID: edccbd0acb8edcba
2021/09/26 06:54:51 ...s/context/context.go:741:1() [D] CSRF Token: DXW6QH0igkYue30BQ1qJXmur00g6MTYzMjYxMDQ5MTIwNTAzMTAwMA
2021/09/26 06:54:51 ...s/context/context.go:185:HTML() [D] Template: user/dashboard/dashboard
2021/09/26 06:54:52 ...s/context/context.go:740:1() [D] Session ID: edccbd0acb8edcba
2021/09/26 06:54:52 ...s/context/context.go:741:1() [D] CSRF Token: DXW6QH0igkYue30BQ1qJXmur00g6MTYzMjYxMDQ5MTIwNTAzMTAwMA
2021/09/26 06:54:54 ...s/context/context.go:740:1() [D] Session ID: edccbd0acb8edcba
2021/09/26 06:54:54 ...s/context/context.go:741:1() [D] CSRF Token: DXW6QH0igkYue30BQ1qJXmur00g6MTYzMjYxMDQ5MTIwNTAzMTAwMA
2021/09/26 06:54:54 ...s/context/context.go:185:HTML() [D] Template: admin/dashboard
2021/09/26 06:54:54 ...s/context/context.go:740:1() [D] Session ID: edccbd0acb8edcba
2021/09/26 06:54:54 ...s/context/context.go:741:1() [D] CSRF Token: DXW6QH0igkYue30BQ1qJXmur00g6MTYzMjYxMDQ5MTIwNTAzMTAwMA
2021/09/26 06:54:56 ...s/context/context.go:740:1() [D] Session ID: edccbd0acb8edcba
2021/09/26 06:54:56 ...s/context/context.go:741:1() [D] CSRF Token: DXW6QH0igkYue30BQ1qJXmur00g6MTYzMjYxMDQ5MTIwNTAzMTAwMA
2021/09/26 06:54:56 ...s/context/context.go:185:HTML() [D] Template: admin/monitor
2021/09/26 06:54:56 ...s/context/context.go:740:1() [D] Session ID: edccbd0acb8edcba
2021/09/26 06:54:56 ...s/context/context.go:741:1() [D] CSRF Token: DXW6QH0igkYue30BQ1qJXmur00g6MTYzMjYxMDQ5MTIwNTAzMTAwMA
2021/09/26 06:55:00 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:55:00 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:55:10 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:55:10 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:55:20 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:55:20 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:55:30 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:55:30 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:55:40 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:55:40 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:55:50 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:55:50 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:56:00 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:56:00 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:56:10 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:56:10 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:56:20 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:56:20 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument
2021/09/26 06:56:30 .../git/batch_reader.go:56:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133) still running 10s after cancellation. Reattempt Kill
2021/09/26 06:56:30 .../git/batch_reader.go:59:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: E:\gitea\gitea-repositories\s\s.git] (routers/private/hook.go:133). Error: invalid argument

pprof files:
210926.zip

@zeripath
Copy link
Contributor

OK the Error: invalid argument suggests that the handle is nil - which is perhaps as a consequence of the inherent raciness of the current implementation or may actually be true.

I'm just not sure what's going on or why these processes aren't shutting down.

@zeripath
Copy link
Contributor

OK lets try:

https://eldritchkitty.com/~andrew/gitea-v1.15.3-10-gd8a318a80-pr-17155-windows-4.0-amd64.exe.xz

@dannyzcy
Copy link
Author

OK lets try:

https://eldritchkitty.com/~andrew/gitea-v1.15.3-10-gd8a318a80-pr-17155-windows-4.0-amd64.exe.xz

I have run that version for 3 hours
No previous problems found

I will continue to observe and report back
thanks

@zeripath
Copy link
Contributor

Did you get any logs with: "still running 10s after cancellation. Reattempt Kill" in them?

@dannyzcy
Copy link
Author

Did you get any logs with: "still running 10s after cancellation. Reattempt Kill" in them?

Running till now (7 hours), i am not get that log.

@zeripath
Copy link
Contributor

I just don't understand it. There's nothing really changed to help stop that from happening. The only thing I can think of is some weird race that the changes have prevented.

I'm gonna make one more change to the PR - dropping the Kill goroutine.

@zeripath
Copy link
Contributor

https://eldritchkitty.com/~andrew/gitea-v1.15.3-14-g68100a6f2-pr-17155-windows-4.0-amd64.exe.xz

This closes just one end of the pipe and removes the terminate go-routine.

@dannyzcy
Copy link
Author

好的，讓我們試試：
https://eldritchkitty.com/~andrew/gitea-v1.15.3-10-gd8a318a80-pr-17155-windows-4.0-amd64.exe.xz

i get error with 'still running 10s after cancellation. Reattempt Kill'

2021/09/28 17:23:39 ...dules/git/command.go:167:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch [repo_path: E:\gitea\gitea-repositories\r\data.git] (services/pull/pull.go:714) still running 10s after cancellation. Reattempt Kill
2021/09/28 17:23:39 ...dules/git/command.go:170:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch [repo_path: E:\gitea\gitea-repositories\r\data.git] (services/pull/pull.go:714). Error: invalid argument

2
未命名

logs

2021/09/28 17:23:37 modules/ssh/ssh.go:150:publicKeyHandler() [D] Handle Public Key: Fingerprint: SHA256:wdRzmPz2yqLFAeJhc573sKrQmfpqJcqz1tME9jGgIJk from 10.123.10.123:64228
2021/09/28 17:23:37 modules/ssh/ssh.go:231:publicKeyHandler() [D] Handle Public Key: 10.123.10.123:64228 Fingerprint: SHA256:wdRzmPz2yqLFAeJhc573sKrQmfpqJcqz1tME9jGgIJk is not a certificate
2021/09/28 17:23:37 modules/ssh/ssh.go:248:publicKeyHandler() [D] Successfully authenticated: 10.123.10.123:64228 Public Key Fingerprint: SHA256:wdRzmPz2yqLFAeJhc573sKrQmfpqJcqz1tME9jGgIJk
2021/09/28 17:23:37 ...ters/private/serv.go:380:ServCommand() [D] Serv Results:
	IsWiki: false
	IsDeployKey: false
	KeyID: 16	KeyName: dev_12/22
	UserName: user
	UserID: 22
	OwnerName: mit
	RepoName: mit_skill_manager
	RepoID: 230
2021/09/28 17:23:39 ...dules/git/command.go:167:func1() [W] D:\Program Files\Git\cmd\git.exe cat-file --batch [repo_path: E:\gitea\gitea-repositories\r\data.git] (services/pull/pull.go:714) still running 10s after cancellation. Reattempt Kill
2021/09/28 17:23:39 ...dules/git/command.go:170:func1() [E] Error whilst trying to kill: D:\Program Files\Git\cmd\git.exe cat-file --batch [repo_path: E:\gitea\gitea-repositories\r\data.git] (services/pull/pull.go:714). Error: invalid argument
2021/09/28 17:23:40 ...s/context/context.go:740:1() [D] Session ID: 0f6f16a2758fca47
2021/09/28 17:23:40 ...s/context/context.go:741:1() [D] CSRF Token: 2h0ezDWkkfEb2otbjEbbtDvSJ8c6MTYzMjc4Njc2MzI3ODE0NTYwMA
2021/09/28 17:23:40 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/28 17:23:40 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= show-ref --tags
2021/09/28 17:23:40 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2021/09/28 17:23:40 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= show-ref --heads
2021/09/28 17:23:40 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= log --name-status -c --format=commit%x00%H %P%x00 --parents --no-renames -t -z d673e7b71168f1cdae0dea0b6c816d29f412e294 -- config.json controller includes profile.json public
2021/09/28 17:23:40 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= log --name-status -c --format=commit%x00%H %P%x00 --parents --no-renames -t -z d673e7b71168f1cdae0dea0b6c816d29f412e294 -- config.json controller includes
2021/09/28 17:23:40 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= log --name-status -c --format=commit%x00%H %P%x00 --parents --no-renames -t -z 41ed736a4e58fe593271fef8a268662d9e794c6c -- config.json includes
2021/09/28 17:23:40 ...s/context/context.go:185:HTML() [D] Template: repo/home
2021/09/28 17:23:41 ...s/context/context.go:740:1() [D] Session ID: 0f6f16a2758fca47
2021/09/28 17:23:41 ...s/context/context.go:741:1() [D] CSRF Token: 2h0ezDWkkfEb2otbjEbbtDvSJ8c6MTYzMjc4Njc2MzI3ODE0NTYwMA
2021/09/28 17:23:41 ...s/context/context.go:740:1() [D] Session ID: 0f6f16a2758fca47
2021/09/28 17:23:41 ...s/context/context.go:741:1() [D] CSRF Token: 2h0ezDWkkfEb2otbjEbbtDvSJ8c6MTYzMjc4Njc2MzI3ODE0NTYwMA
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= show-ref --tags
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= show-ref --heads
2021/09/28 17:23:41 ...s/web/repo/branch.go:71:Branches() [D] Branches: skip: 0 limit: 20
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= show-ref --heads
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/28 17:23:41 ...repo_base_nogogit.go:69:CatFileBatchCheck() [D] Opening temporary cat file batch-check: E:\gitea\gitea-repositories\mit\drafter.git
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2021/09/28 17:23:41 ...repo_base_nogogit.go:60:CatFileBatch() [D] Opening temporary cat file batch for: E:\gitea\gitea-repositories\mit\drafter.git
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-list --count master..refs/heads/1.0.2-rc
2021/09/28 17:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-list --count refs/heads/1.0.2-rc..master
2021/09/28 17:23:42 ...s/web/repo/branch.go:208:loadBranches() [D] loadOneBranch: load default: 'master'
2021/09/28 17:23:42 ...repo_base_nogogit.go:69:CatFileBatchCheck() [D] Opening temporary cat file batch-check: E:\gitea\gitea-repositories\mit\drafter.git
2021/09/28 17:23:42 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2021/09/28 17:23:42 ...repo_base_nogogit.go:60:CatFileBatch() [D] Opening temporary cat file batch for: E:\gitea\gitea-repositories\mit\drafter.git
2021/09/28 17:23:42 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/28 17:23:42 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-list --count master..refs/heads/master
2021/09/28 17:23:42 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-list --count refs/heads/master..master
2021/09/28 17:23:42 ...s/context/context.go:185:HTML() [D] Template: repo/branch/list
2021/09/28 17:23:42 ...s/context/context.go:740:1() [D] Session ID: 0f6f16a2758fca47
2021/09/28 17:23:42 ...s/context/context.go:741:1() [D] CSRF Token: 2h0ezDWkkfEb2otbjEbbtDvSJ8c6MTYzMjc4Njc2MzI3ODE0NTYwMA
2021/09/28 17:23:43 ...s/context/context.go:740:1() [D] Session ID: 0f6f16a2758fca47
2021/09/28 17:23:43 ...s/context/context.go:741:1() [D] CSRF Token: 2h0ezDWkkfEb2otbjEbbtDvSJ8c6MTYzMjc4Njc2MzI3ODE0NTYwMA
2021/09/28 17:23:44 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/28 17:23:44 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= show-ref --tags
2021/09/28 17:23:44 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2021/09/28 17:23:44 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= show-ref --heads
2021/09/28 17:23:44 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\mit\drafter.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= show-ref --tags
2021/09/28 17:23:44 ...s/context/context.go:185:HTML() [D] Template: repo/release/list
2021/09/28 17:23:44 ...s/context/context.go:740:1() [D] Session ID: 0f6f16a2758fca47
2021/09/28 17:23:44 ...s/context/context.go:741:1() [D] CSRF Token: 2h0ezDWkkfEb2otbjEbbtDvSJ8c6MTYzMjc4Njc2MzI3ODE0NTYwMA
2021/09/28 17:23:44 ...s/context/context.go:740:1() [D] Session ID: 3a3dc64e8c1d115e
2021/09/28 17:23:44 ...s/context/context.go:741:1() [D] CSRF Token: 1Lxc4e1HSmIIqJm_r5X2OQLu7NU6MTYzMjgyMTAyNDcyNzE4MjkwMA
2021/09/28 17:23:44 modules/ssh/ssh.go:150:publicKeyHandler() [D] Handle Public Key: Fingerprint: SHA256:wdRzmPz2yqLFAeJhc573sKrQmfpqJcqz1tME9jGgIJk from 10.123.10.123:55042
2021/09/28 17:23:44 modules/ssh/ssh.go:231:publicKeyHandler() [D] Handle Public Key: 10.123.10.123:55042 Fingerprint: SHA256:wdRzmPz2yqLFAeJhc573sKrQmfpqJcqz1tME9jGgIJk is not a certificate

pprof:
210928.zip

@dannyzcy
Copy link
Author

https://eldritchkitty.com/~andrew/gitea-v1.15.3-14-g68100a6f2-pr-17155-windows-4.0-amd64.exe.xz

這僅關閉管道的一端並刪除終止 go-routine。

let me try the version

@zeripath
Copy link
Contributor

I'm really not sure what's going on here. This looks to be either some bug in go, git or Windows. Are you using a virus checker?

I'm really struggling for what else we can do. I don't have any real idea as to what's causing Windows to keep the process around and clearly from your invalid argument error we can't just kill the process again as the handle becomes invalid. I guess we could try to get the handle straight after starting but that is likely racy. It might be that closing both ends of the pipe is the problem so maybe this last version solves that?

Otherwise: Are you sure that there isn't some bug in your git version? Could this be an antivirus issue?

(The last version I uploaded just ensures that only one end of the pipes gets closed but the reattempt closer has been removed so you're stuck checking for blocked processes manually.)

@dannyzcy
Copy link
Author

My gitea upgrade from 1.14.5 to 1.15.0 on 8/24, and found #17007

Then I upgraded git from 2.32.0 to 2.33.0 to try to fix

This problem was discovered when upgrading to 1.15.3 on 9/22

No blocked process was found before 1.14.5

Except for git & gitea to upgrade, other server environments have not been adjusted

I checked my virus checker without any log

@dannyzcy
Copy link
Author

(The last version I uploaded just ensures that only one end of the pipes gets closed but the reattempt closer has been removed so you're stuck checking for blocked processes manually.)

The last version still have a blocked process

753

logs & pprof:
gitea_210929.zip

maybe i should install git 2.32.0 to try

@zeripath
Copy link
Contributor

May be worth trying 2.32 - the other option is that the problem is related to go 1.17 I guess

@dannyzcy
Copy link
Author

i installed Git 2.32.0 have the same problem.

未命名
pprof_logs.zip

@zeripath
Copy link
Contributor

OK I'm stumped. Genuinely stumped. Some more questions?

  • Are you using MSYS2 git or git-for-windows?
  • I wonder if this is a go 1.16 bug?
  • Were you previously running the gogit variant in 1.14?

I don't see any reason why our code is causing this and I can't think of anymore ways of attempting to workaround the issue.

@zeripath
Copy link
Contributor

To that end I'm going to push up a build of 1.15.4 with go 1.17 in case the issue is simply a go1.16 bug.

https://www.eldritchkitty.com/~andrew/gitea-v1.15.4-go1.17-windows-4.0-amd64.exe.xz

@dannyzcy
Copy link
Author

dannyzcy commented Oct 12, 2021 •
edited
Loading

  • Are you using MSYS2 git or git-for-windows?

maybe MSYS2 git
I downloaded from the git official website (https://git-scm.com/download/win)

  • I wonder if this is a go 1.16 bug?
    To that end I'm going to push up a build of 1.15.4 with go 1.17 in case the issue is simply a go1.16 bug.

I still have the same problem, can you help me try it with go 1.15?
My server is windows7, maybe it's time to be eliminated.
I am try to dump all gitea data to windows 10 server, these problems disappeared.

  • Were you previously running the gogit variant in 1.14?

No, I always download gitea from the official blog

@zeripath zeripath mentioned this issue Oct 20, 2021
Merged
@dannyzcy
Copy link
Author

dannyzcy commented Nov 3, 2021

I run 1.15.6 with git 2.33.1 for 2 days,
I didn't find this bug anymore

@zeripath
Copy link
Contributor

zeripath commented Nov 3, 2021

Ok I'm closing - I don't think we'll ever find out what was the problem.

@zeripath zeripath closed this as completed Nov 3, 2021
@wxiaoguang wxiaoguang mentioned this issue Dec 3, 2021
Merged
@atom0s atom0s mentioned this issue Feb 12, 2022
Closed
@go-gitea go-gitea locked and limited conversation to collaborators Apr 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prevent dangling cat-files … zeripath/gitea
4 participants
@lunny @zeripath @wxiaoguang @dannyzcy