UNIX sockets not cleaned up on exit

[Daniel15]

I'm using Supervisord to run Kestrel for my site. This is my Supervisor config:

[program:dan]
command=/var/www/.dnx/runtimes/dnx-mono.1.0.0-rc1-final/bin/dnx --appbase /var/www/dan.cx/live/site/approot/packages/Daniel15.Web/1.0.0/root Microsoft.Dnx.ApplicationHost --configuration Release web-live
environment=ASPNET_ENV=Production
user=www-data
directory=/var/www/dan.cx/live/site/

Command in project.json:

"web-live": "Microsoft.AspNet.Server.Kestrel --server.urls http://unix:/run/dan-live.sock",

This is working fine, but when I stop Kestrel (supervisorctl stop dan, "dan" being the name of the site in the Supervisor config), it does not clean up the UNIX socket (/run/dan-live.sock). The next time I try to start it, I get an "address already in use" error:

Microsoft.AspNet.Server.Kestrel.Networking.UvException: Error -98 EADDRINUSE address already in use
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.Check (Int32 statusCode) <0x41d3a2a0 + 0x00063> in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.pipe_bind (Microsoft.AspNet.Server.Kestrel.Networking.UvPipeHandle handle, System.String name) <0x41d3e0e0 + 0x00053> in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.UvPipeHandle.Bind (System.String name) <0x41d3e0a0 + 0x00027> in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Http.PipeListener.CreateListenSocket () <0x41d3d9c0 + 0x00087> in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Http.Listener+<>c__DisplayClass5_0.<StartAsync>b__0 (System.Object _) <0x41d3d8c0 + 0x00030> in <filename unknown>:0

I need to manually rm /run/dan-live.sock before I can start Kestrel again. I think Kestrel simply needs to delete the socket on exit.

[Daniel15]

As a guess, PipeListener and PipeListenerPrimary should delete the socket in their Dispose implementation.

[cesarblum]

Supervisor sends SIGTERM by default to stop a process. If you set stopsignal to INT, that should gracefully stop kestrel and the UNIX socket file will be deleted. Could you try that and get back to us with the results?

[Daniel15]

@CesarBS - unfortunately it looks like that didn't work. This is my full config:

[program:dan]
command=/var/www/.dnx/runtimes/dnx-mono.1.0.0-rc1-final/bin/dnx --appbase /var/www/dan.cx/live/site/approot/packages/Daniel15.Web/1.0.0/root Microsoft.Dnx.ApplicationHost --configuration Release web-live
environment=ASPNET_ENV=Production
user=www-data
directory=/var/www/dan.cx/live/site/
stopsignal=INT

Here's what I was doing:

root@dan:~# supervisorctl status dan
dan                              RUNNING    pid 16159, uptime 0:00:27

root@dan:~# ls -lah /run/dan-live.sock
srwxr-xr-x 1 www-data www-data 0 Dec  8 19:55 /run/dan-live.sock

root@dan:~# supervisorctl stop dan
dan: stopped

root@dan:~# ls -lah /run/dan-live.sock
srwxr-xr-x 1 www-data www-data 0 Dec  8 19:55 /run/dan-live.sock

SIGTERM should still clean up the sockets, right? I didn't know much about the different signals, but I found this useful post on Quora that explains them. Emphasis mine:

SIGTERM is the termination signal. The default behavior is to terminate the process, but it also can be caught or ignored. The intention is to kill the process, gracefully or not, but to first allow it a chance to cleanup.

[cesarblum]

My suggestion was actually an educated guess. I didn't really run Kestrel behind Supervisor - what I actually did was to check the behavior difference between killing kestrel with SIGTERM and SIGINT and I noticed that the socket lingered around with SIGTERM but not with SIGINT.

Just to make sure: did you restart Supervisor after setting stopsignal to INT?

[Daniel15]

Yeah, I restarted Supervisor.

[cesarblum]

When I try running an app with supervisor, I get

kestrel: ERROR (abnormal termination)

Have you ever run into that?

[Daniel15]

Try checking the output. supervisorctl tail appname for stdout and supervisorctl tail appname stderr for stderr, or look at the log files for the process (they're in /var/log/supervisord/ on my system)

[cesarblum]

Can your app run on CoreCLR? I'm now able to repro your issue, but I noticed it only happens on Mono. If I use the CoreCLR DNX the socket is gone once Supervisor stops the process.

[Daniel15]

Unfortunately I can't use CoreCLR yet as I'm using some third-party libraries that don't support it and also some stuff in the System.Drawing namespace.

[cesarblum]

Can you run your app manually, kill it with SIGINT and check if the socket file is still there after that? I'd like to check that you're seeing the same results as I'm seeing. When I kill the process manually, the socket is deleted.

[Daniel15]

Interesting... It does appear to correctly clean up the socket when I run the app manually and use SIGINT. Is Supervisor doing something differently?