Push failed. Can't delete temp index file. Warning on service stop #18691
Description
Gitea Version
1.16.1
Git Version
2.35.1.windows.2
Operating System
Windows server 2012
How are you running Gitea?
Gitea is running as a windows service, it's installed with the package gitea-1.16.1-windows-4.0-amd64.exe
Database
SQLite
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Description
Often the push from a client to the gitea remote get stuck for a lot of time.
Looking at the log I found this
2022/02/09 15:57:49 ...es/git/repo_index.go:59:func1() [E] failed to remove tmp index file: remove C:\Windows\TEMP\index3937789675: Impossibile accedere al file. Il file è utilizzato da un altro processo.
2022/02/09 16:55:04 ...es/git/repo_index.go:59:func1() [E] failed to remove tmp index file: remove C:\Windows\TEMP\index1770678566: Impossibile accedere al file. Il file è utilizzato da un altro processo.
2022/02/09 16:56:59 ...es/git/repo_index.go:59:func1() [E] failed to remove tmp index file: remove C:\Windows\TEMP\index3990406363: Impossibile accedere al file. Il file è utilizzato da un altro processo.
2022/02/09 17:00:57 ...eue/queue_channel.go:98:func1() [W] ChannelQueue: push_update-channel Terminated before completed flushing
2022/02/09 17:01:57 .../graceful/manager.go:226:doHammerTime() [W] Setting Hammer condition
This temp index folders are empty, I've tried to remove that but I can't until I stop the gitea service. This take a lot of time and sometime even fail and I have to terminate the gitea process from the task manager.
During the stop of the service this lines are added to the log, even after a restart without any remote push
2022/02/09 17:11:01 ...eue/queue_channel.go:98:func1() [W] ChannelQueue: task-channel Terminated before completed flushing
2022/02/09 17:11:01 ...eue/queue_channel.go:98:func1() [W] ChannelQueue: push_update-channel Terminated before completed flushing
2022/02/09 17:11:01 ...eue/queue_channel.go:98:func1() [W] ChannelQueue: notification-service-channel Terminated before completed flushing
2022/02/09 17:11:01 .../graceful/manager.go:226:doHammerTime() [W] Setting Hammer condition
2022/02/09 17:11:02 .../graceful/manager.go:243:doTerminate() [W] Terminating
Screenshots
No response
Metadata
Metadata
Assignees
Type
Projects
Milestone
Relationships
Development
No branches or pull requests
Participants
Activity
zeripath commentedon Feb 9, 2022
I'm having difficulty understanding what it is you're reporting.
I think you might be reporting a possible db deadlock. In which case it would be helpful to give us more logs.
bota87 commentedon Feb 9, 2022
I posted all the log that I have. I try to set a more verbouse log level
bota87 commentedon Feb 10, 2022
I set the log level to info, restarted the service, do a fetch from a client, stopped the service. Here the log
I tried the same without the fetch. Here the log
zeripath commentedon Feb 10, 2022
Logs
When you are reporting bugs or issues it is helpful to give us DEBUG logs.
In fact the issue proforma expressly asks you to do that and links to a helpful section of the documentation that to do that: https://docs.gitea.io/en-us/logging-configuration/#debugging-problems
I will copy that below:
Debugging problems
When submitting logs in Gitea issues it is often helpful to submit
merged logs obtained by either by redirecting the console log to a file or
copying and pasting it. To that end it is recommended to set your logging to:
Sometimes it will be helpful get some specific
TRACElevel logging restrictedto messages that match a specific
EXPRESSION. Adjusting theMODEin the[log]section toMODE = console,traceconsoleto add a new logger outputtraceconsoleand then adding its corresponding section would be helpful:(It's worth noting that log messages that match the expression at or above debug
level will get logged twice so don't worry about that.)
STACKTRACE_LEVELshould generally be left unconfigured (and hence kept atnone). There are only very specific occasions when it useful.zeripath commentedon Feb 10, 2022
From the small amount of logs you've given me it appears that Gitea is being shutdown externally.
This is likely by the windows service.
There was an older issue regarding this: #15762 but I'm not sure what's going on here.
bota87 commentedon Feb 10, 2022
Hi @zeripath,
thank's for your help.
Here you can find the debug level log
zeripath commentedon Feb 10, 2022
Yup. Gitea is being shutdown by your windows service.
bota87 commentedon Feb 10, 2022
I know that, I start and then stop the windows service.
I'm worried about this warnings on service stop, I think that are related to my issue with push get stuck, I'm wrong?
bota87 commentedon Feb 10, 2022
Here the push client side, as you can see it fail after 2 minutes.
And the related gitea log
bota87 commentedon Feb 10, 2022
Hi tried the same push again without doing nothing client or server side and this time worked.
gitea log
zeripath commentedon Feb 10, 2022
[log]section? I do not see any router logs in here. I need to see these.bota87 commentedon Feb 10, 2022
There is a package for the latest 1.16 head?
I missed to add the router entry on the app.ini, now I think it's right
In the mean time I got the failed to remove tmp index file again. Here the log
bota87 commentedon Feb 10, 2022
Here the log with the router entry, got the failed to remove tmp index file again
bota87 commentedon Feb 10, 2022
If can be usefull the client that cause this info message is windows 11, with windows 10 this dosen't appear
[I] http: TLS handshake error from 192.168.213.3:52009: tls: no cipher suite supported by both client and serverzeripath commentedon Feb 10, 2022
So I need to see the router.log file too. (This is why the documentation says use
MODE=consoleso that the router logs will be interleaved...)Anyway... I think that's telling you what the problem is...
https://docs.gitea.io/en-us/config-cheat-sheet/#server-server
Take a look at
[server]section and the following settings:It would be useful to know if you've set a LOCAL_ROOT_URL and to know what the rest of your app.ini looks like.
At present I don't know if the problem is that Gitea can't talk to itself or if it's git that can't talk to Gitea. The Router logs would help a little with that.
4 remaining items
zeripath commentedon Feb 15, 2022
hmm... I wonder if this is related to #18734 would it be possible to try the latest 1.16 head https://dl.gitea.io/gitea/1.16?
bota87 commentedon Feb 15, 2022
ok I installed the latest 1.16 head, let see how it go.
Before the upgrade I made a backup and got this warning
Complete logs here
bota87 commentedon Feb 16, 2022
Unfortunately even with the latest version the server sometime is unresponding.
This time I tried a push, in the mean time I tried to access to the repository website but this times out, after 2 minutes even the push times out, then I tried the push again and as usual it was completed in seconds. Here the logs
lunny commentedon Feb 16, 2022
Looks like it's still related with #18754 .
bota87 commentedon Feb 23, 2022
Hi @lunny,
today I got again the timeout on the push, after the timeout I tried to push again and worked in few seconds
2022/02/23 10:09:40 ...c/net/http/server.go:3160:logf() [I] http: TLS handshake error from [::1]:48498: read tcp [::1]:3000->[::1]:48498: wsarecv: Connessione interrotta dal software del computer host.Here the complete logs
bota87 commentedon Feb 23, 2022
Got again the open checker error and then the push timeouts
Logs
bota87 commentedon Apr 8, 2022
Any news on this?
@lunny @zeripath
wxiaoguang commentedon Oct 7, 2022
Many different issues are mixed here. I would like to guess:
[E] failed to remove tmp index file: remove C:\Windows\TEMP\index3937789675is that Gitea tried to delete some temp files but these files are not closed properly. On Windows, the not-closed files might not be deleted.[E] Unable to open checker for 7259fc298bcd5d32dfb69fd6c615ca8cb5aa9b23. Error: failed to run attr-check., it might be a bug, might not. Not sure whether it is related to the push failure.[I] http: TLS handshake error from [::1]:49008: read tcp [::1]:3000->[::1]:49008:ps: 1.16 is out dated.
As a suggestion, running Gitea on Windows doesn't seem to be a good idea. If you can use docker, you wouldn't have so many troubles.
If you can try 1.17 and provide reproducible steps from scratch (a fresh installation, all configs, all operations), it would be easier for others to help.
bota87 commentedon Oct 7, 2022
Hi @wxiaoguang,
I've already updated to 1.17.2 some weeks ago but the issue still persist.
One of the reasons I chose Gitea was that it's easy to install on windows, anyway I tought to move on docker and probably I'll do that once I get really annoyed by this bug.
wxiaoguang commentedon Oct 7, 2022
Thanks for understanding.
Windows server 2012 seems quite old nowadays 😂 , while Git is a POSIX-based system, I have seen many issues of running Gitea on old Windows for large scale use (git compatibility problem, performance problem). And another challenge is that nowadays few Gitea contributors use Windows for production, so if Windows users can not provide details about bugs, it's really difficult to improve.
bota87 commentedon Oct 7, 2022
Just moved the setup to another server with windows server 2019, let's see if there works without problems 🤞
bota87 commentedon Oct 14, 2022
haven't had any problem on the last week since the server migration, seems fixed! 🥳
Thank's wxiaoguang
twisted-nematic57 commentedon Nov 12, 2022
Lol so is this some kind of Windows 7 era specific bug, I can't afford newer hardware is there no way to fix the
attr-checkproblem??