CouchbaseHealthIndicator can hang when Couchbase goes away

[HarryEMartland]

When couchbase is stoped completely the health endpoint does not return and the below logs continuously spew out.

Steps to reproduce:

• Start couchbase
• Boot up spring boot app with couchbase and couchbase health indicator enabled
• Kill couchbase
• Request health endpoint /actuator/health

Versions:
spring-data-couchbase:3.0.8.RELEASE
spring-boot-gradle-plugin:2.0.3.RELEASE

2018-07-24 22:52:14.877  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:14.877  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:14.877  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:14.980  WARN 5945 --- [      cb-io-1-1] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:14.980  WARN 5945 --- [      cb-io-1-2] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:14.980  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:14.980  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:14.980  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:14.980  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.083  WARN 5945 --- [      cb-io-1-5] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.083  WARN 5945 --- [      cb-io-1-6] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.084  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.084  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.084  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.084  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.185  WARN 5945 --- [      cb-io-1-1] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.186  WARN 5945 --- [      cb-io-1-2] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.186  WARN 5945 --- [      cb-io-1-2] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.189  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.190  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.190  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.286  WARN 5945 --- [      cb-io-1-5] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.286  WARN 5945 --- [      cb-io-1-6] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.286  WARN 5945 --- [      cb-io-1-6] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.294  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.295  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.295  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.392  WARN 5945 --- [      cb-io-1-1] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.393  WARN 5945 --- [      cb-io-1-2] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.394  WARN 5945 --- [      cb-io-1-2] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.395  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.395  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.395  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.498  WARN 5945 --- [      cb-io-1-5] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.498  WARN 5945 --- [      cb-io-1-6] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.499  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.499  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.499  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.499  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.603  WARN 5945 --- [      cb-io-1-1] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.603  WARN 5945 --- [      cb-io-1-2] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.604  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.604  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.604  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.604  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.705  WARN 5945 --- [      cb-io-1-6] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.705  WARN 5945 --- [      cb-io-1-5] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.706  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.706  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.706  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.706  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.809  WARN 5945 --- [      cb-io-1-1] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.809  WARN 5945 --- [      cb-io-1-2] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.810  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.810  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.810  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.810  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.913  WARN 5945 --- [      cb-io-1-6] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.913  WARN 5945 --- [      cb-io-1-5] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.913  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.913  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.913  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:15.913  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.017  WARN 5945 --- [      cb-io-1-2] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.017  WARN 5945 --- [      cb-io-1-1] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.018  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.018  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.018  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.018  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.119  WARN 5945 --- [      cb-io-1-5] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.119  WARN 5945 --- [      cb-io-1-6] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.120  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.120  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.120  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.120  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.221  WARN 5945 --- [      cb-io-1-1] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.221  WARN 5945 --- [      cb-io-1-2] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.222  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.222  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.222  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.222  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.325  WARN 5945 --- [      cb-io-1-5] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.325  WARN 5945 --- [      cb-io-1-6] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.327  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.327  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.327  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.327  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.430  WARN 5945 --- [      cb-io-1-1] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.430  WARN 5945 --- [      cb-io-1-2] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.431  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.431  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.431  WARN 5945 --- [      cb-io-1-4] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.431  WARN 5945 --- [      cb-io-1-3] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.532  WARN 5945 --- [      cb-io-1-5] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.532  WARN 5945 --- [      cb-io-1-6] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.533  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.533  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : [localhost:8091][ConfigEndpoint]: Could not connect to remote socket: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.533  WARN 5945 --- [      cb-io-1-7] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091
2018-07-24 22:52:16.533  WARN 5945 --- [      cb-io-1-8] c.c.client.core.endpoint.Endpoint        : Error during reconnect: com.couchbase.client.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8091

[snicoll]

Similar to #10809 with hopefully a way to configure the client not to do that.

[dnl50]

I digged a bit into it and the reason why this happens is because the CouchBaseHealthIndicator invokes
the info(timeout, timeUnit) method of the DefaultBucketManager of couchbase, which in return retries sending a BucketConfigRequest to the couchbase server. It returns after the request was successfor or the given timeout threshold is exceeded.

The default timeout is set to 75 seconds (specified in DefaultCouchbaseEnvironment by the MANAGEMENT_TIMEOUT field), which is why it takes forever for the health actuator to return when the server is not online.

I will try to find a way to make the timeout configurable, because 75 seconds seems a bit much :P

Edit:

So the configuration class creating the DefaultCouchbaseEnvironment is AbstractCouchbaseConfiguration of spring data couchbase.

Edit 2:

Calling DefaultBucketManager#info(timeout, timeunit) with a configurable timeout will solve this problem.

[snicoll]

so this issue can be closed I think.

@dnl50 what makes you think this issue can be closed? As you've indicated we should call info with a configurable timeout probably

[dnl50]

I thought the info(timeout, timeunit) method was protected, so yeah it can be called with a configurable timeout of course.