Add Reactive retries for SC LoadBalancer (#847)

* Implement retry logic.

* Fix retrying on next instance when RetryExhausted in same instance.

* Fix retrying on next instance when RetryExhausted in same instance.

* Fix retrying on next instance when RetryExhausted in same instance.

* Move duplicated methods to utility class. Fix checkstyle.

* Fix test.

* Add more tests.

* Fix test.

* Add autoConfiguration.

* Refactor and add javadocs.

* Add javadocs.

* Use RetryAwareServiceInstanceListSupplier with reactive retries.

* Update properties.

* Fix the docs.

* Rename utility class.

* Verify interactions in order.

Author: OlgaMaciaszek

docs/src/main/asciidoc/_configprops.adoc

@@ -32,6 +32,10 @@
 |spring.cloud.loadbalancer.health-check.path |  | 
 |spring.cloud.loadbalancer.hint |  | Allows setting the value of <code>hint</code> that is passed on to the LoadBalancer request and can subsequently be used in {@link ReactiveLoadBalancer} implementations.
 |spring.cloud.loadbalancer.retry.avoid-previous-instance | true | Enables wrapping ServiceInstanceListSupplier beans with `RetryAwareServiceInstanceListSupplier` if Spring-Retry is in the classpath.
+|spring.cloud.loadbalancer.retry.backoff.enabled | false | Indicates whether Reactor Retry backoffs should be applied.
+|spring.cloud.loadbalancer.retry.backoff.jitter | 0.5 | Used to set {@link RetryBackoffSpec#jitter}.
+|spring.cloud.loadbalancer.retry.backoff.max-backoff |  | Used to set {@link RetryBackoffSpec#maxBackoff}.
+|spring.cloud.loadbalancer.retry.backoff.min-backoff | 5ms | Used to set {@link RetryBackoffSpec#minBackoff}.
 |spring.cloud.loadbalancer.retry.enabled | true | 
 |spring.cloud.loadbalancer.retry.max-retries-on-next-service-instance | 1 | Number of retries to be executed on the next <code>ServiceInstance</code>. A <code>ServiceInstance</code> is chosen before each retry call.
 |spring.cloud.loadbalancer.retry.max-retries-on-same-service-instance | 0 | Number of retries to be executed on the same <code>ServiceInstance</code>.

docs/src/main/asciidoc/spring-cloud-commons.adoc

@@ -437,18 +437,27 @@ Then, `ReactiveLoadBalancer` is used underneath.
 
 A load-balanced `RestTemplate` can be configured to retry failed requests.
 By default, this logic is disabled.
-You can enable it by adding link:https://github.com/spring-projects/spring-retry[Spring Retry] to your application's classpath.
+For the non-reactive version (with `RestTemplate`), you can enable it by adding link:https://github.com/spring-projects/spring-retry[Spring Retry] to your application's classpath. For the reactive version (with `WebTestClient), you need to set `spring.cloud.loadbalancer.retry.enabled=false`.
 
-If you would like to disable the retry logic with Spring Retry on the classpath, you can set `spring.cloud.loadbalancer.retry.enabled=false`.
+If you would like to disable the retry logic with Spring Retry or Reactive Retry on the classpath, you can set `spring.cloud.loadbalancer.retry.enabled=false`.
 
-If you would like to implement a `BackOffPolicy` in your retries, you need to create a bean of type `LoadBalancedRetryFactory` and override the `createBackOffPolicy()` method.
+For the non-reactive implementation, if you would like to implement a `BackOffPolicy` in your retries, you need to create a bean of type `LoadBalancedRetryFactory` and override the `createBackOffPolicy()` method.
+
+For the reactive implementation, you just need to enable it by setting `spring.cloud.loadbalancer.retry.backoff.enabled` to `false`.
 
 You can set:
 
 - `spring.cloud.loadbalancer.retry.maxRetriesOnSameServiceInstance` - indicates how many times a request should be retried on the same `ServiceInstance` (counted separately for every selected instance)
 - `spring.cloud.loadbalancer.retry.maxRetriesOnNextServiceInstance` - indicates how many times a request should be retried a newly selected `ServiceInstance`
 - `spring.cloud.loadbalancer.retry.retryableStatusCodes` - the status codes on which to always retry a failed request.
 
+For the reactive implementation, you can additionally set:
+ - `spring.cloud.loadbalancer.retry.backoff.minBackoff` - Sets the minimum backoff duration (by default, 5 milliseconds)
+ - `spring.cloud.loadbalancer.retry.backoff.maxBackoff` - Sets the maximum backoff duration (by default, max long value of milliseconds)
+ - `spring.cloud.loadbalancer.retry.backoff.jitter` - Sets the jitter used for calculationg the actual backoff duration for each call (by default, 0.5).
+
+For the reactive implementation, you can also implement your own `LoadBalancerRetryPolicy` to have more detailed control over the load-balanced call retries.
+
 NOTE: For load-balanced retries, by default, we wrap the `ServiceInstanceListSupplier` bean with `RetryAwareServiceInstanceListSupplier` to select a different instance from the one previously chosen, if available. You can disable this behavior by setting the value of `spring.cloud.loadbalancer.retry.avoidPreviousInstance` to `false`.
 
 ====

spring-cloud-commons/src/main/java/org/springframework/cloud/client/loadbalancer/ClientRequestContext.java

@@ -16,6 +16,7 @@
 
 package org.springframework.cloud.client.loadbalancer;
 
+import org.springframework.http.HttpMethod;
 import org.springframework.web.reactive.function.client.ClientRequest;
 
 /**
@@ -36,4 +37,8 @@ public ClientRequest getClientRequest() {
 		return (ClientRequest) super.getClientRequest();
 	}
 
+	public HttpMethod method() {
+		return ((ClientRequest) super.getClientRequest()).method();
+	}
+
 }

spring-cloud-commons/src/main/java/org/springframework/cloud/client/loadbalancer/reactive/ExchangeFilterFunctionUtils.java

@@ -0,0 +1,53 @@
+/*
+ * Copyright 2012-2020 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.springframework.cloud.client.loadbalancer.reactive;
+
+import java.net.URI;
+import java.util.Map;
+
+import org.springframework.web.reactive.function.client.ClientRequest;
+import org.springframework.web.reactive.function.client.ExchangeFilterFunction;
+
+/**
+ * A utility class for load-balanced {@link ExchangeFilterFunction} instances.
+ *
+ * @author Olga Maciaszek-Sharma
+ * @since 3.0.0
+ */
+public final class ExchangeFilterFunctionUtils {
+
+	private ExchangeFilterFunctionUtils() {
+		throw new IllegalStateException("Can't instantiate a utility class.");
+	}
+
+	static String getHint(String serviceId, Map<String, String> hints) {
+		String defaultHint = hints.getOrDefault("default", "default");
+		String hintPropertyValue = hints.get(serviceId);
+		return hintPropertyValue != null ? hintPropertyValue : defaultHint;
+	}
+
+	static ClientRequest buildClientRequest(ClientRequest request, URI uri) {
+		return ClientRequest.create(request.method(), uri).headers(headers -> headers.addAll(request.headers()))
+				.cookies(cookies -> cookies.addAll(request.cookies()))
+				.attributes(attributes -> attributes.putAll(request.attributes())).body(request.body()).build();
+	}
+
+	static String serviceInstanceUnavailableMessage(String serviceId) {
+		return "LoadBalancer does not contain an instance for the service " + serviceId;
+	}
+
+}

spring-cloud-commons/src/main/java/org/springframework/cloud/client/loadbalancer/reactive/LoadBalancedExchangeFilterFunction.java

@@ -0,0 +1,29 @@
+/*
+ * Copyright 2012-2020 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.springframework.cloud.client.loadbalancer.reactive;
+
+import org.springframework.web.reactive.function.client.ExchangeFilterFunction;
+
+/**
+ * A marker interface for load-balanced {@link ExchangeFilterFunction} instances.
+ *
+ * @author Olga Maciaszek-Sharma
+ * @since 3.0.0
+ */
+public interface LoadBalancedExchangeFilterFunction extends ExchangeFilterFunction {
+
+}

spring-cloud-commons/src/main/java/org/springframework/cloud/client/loadbalancer/reactive/LoadBalancerBeanPostProcessorAutoConfiguration.java

@@ -58,8 +58,8 @@ protected static class ReactorDeferringLoadBalancerFilterConfig {
 
 		@Bean
 		@Primary
-		DeferringLoadBalancerExchangeFilterFunction<ReactorLoadBalancerExchangeFilterFunction> reactorDeferringLoadBalancerExchangeFilterFunction(
-				ObjectProvider<ReactorLoadBalancerExchangeFilterFunction> exchangeFilterFunctionProvider) {
+		DeferringLoadBalancerExchangeFilterFunction<LoadBalancedExchangeFilterFunction> reactorDeferringLoadBalancerExchangeFilterFunction(
+				ObjectProvider<LoadBalancedExchangeFilterFunction> exchangeFilterFunctionProvider) {
 			return new DeferringLoadBalancerExchangeFilterFunction<>(exchangeFilterFunctionProvider);
 		}
 

spring-cloud-commons/src/main/java/org/springframework/cloud/client/loadbalancer/reactive/LoadBalancerProperties.java

@@ -21,6 +21,8 @@
 import java.util.Map;
 import java.util.Set;
 
+import reactor.util.retry.RetryBackoffSpec;
+
 import org.springframework.boot.context.properties.ConfigurationProperties;
 import org.springframework.http.HttpMethod;
 import org.springframework.util.LinkedCaseInsensitiveMap;
@@ -47,7 +49,7 @@ public class LoadBalancerProperties {
 	private Map<String, String> hint = new LinkedCaseInsensitiveMap<>();
 
 	/**
-	 * Properties for Spring-Retry support in Spring Cloud LoadBalancer.
+	 * Properties for Spring-Retry and Reactor Retry support in Spring Cloud LoadBalancer.
 	 */
 	private Retry retry = new Retry();
 
@@ -141,6 +143,11 @@ public static class Retry {
 		 */
 		private Set<Integer> retryableStatusCodes = new HashSet<>();
 
+		/**
+		 * Properties for Reactor Retry backoffs in Spring Cloud LoadBalancer.
+		 */
+		private Backoff backoff = new Backoff();
+
 		/**
 		 * Returns true if the load balancer should retry failed requests.
 		 * @return True if the load balancer should retry failed requests; false
@@ -190,6 +197,70 @@ public void setRetryableStatusCodes(Set<Integer> retryableStatusCodes) {
 			this.retryableStatusCodes = retryableStatusCodes;
 		}
 
+		public Backoff getBackoff() {
+			return backoff;
+		}
+
+		public void setBackoff(Backoff backoff) {
+			this.backoff = backoff;
+		}
+
+		public static class Backoff {
+
+			/**
+			 * Indicates whether Reactor Retry backoffs should be applied.
+			 */
+			private boolean enabled = false;
+
+			/**
+			 * Used to set {@link RetryBackoffSpec#minBackoff}.
+			 */
+			private Duration minBackoff = Duration.ofMillis(5);
+
+			/**
+			 * Used to set {@link RetryBackoffSpec#maxBackoff}.
+			 */
+			private Duration maxBackoff = Duration.ofMillis(Long.MAX_VALUE);
+
+			/**
+			 * Used to set {@link RetryBackoffSpec#jitter}.
+			 */
+			private double jitter = 0.5d;
+
+			public Duration getMinBackoff() {
+				return minBackoff;
+			}
+
+			public void setMinBackoff(Duration minBackoff) {
+				this.minBackoff = minBackoff;
+			}
+
+			public Duration getMaxBackoff() {
+				return maxBackoff;
+			}
+
+			public void setMaxBackoff(Duration maxBackoff) {
+				this.maxBackoff = maxBackoff;
+			}
+
+			public double getJitter() {
+				return jitter;
+			}
+
+			public void setJitter(double jitter) {
+				this.jitter = jitter;
+			}
+
+			public boolean isEnabled() {
+				return enabled;
+			}
+
+			public void setEnabled(boolean enabled) {
+				this.enabled = enabled;
+			}
+
+		}
+
 	}
 
 }

spring-cloud-commons/src/main/java/org/springframework/cloud/client/loadbalancer/reactive/LoadBalancerRetryContext.java

@@ -0,0 +1,83 @@
+/*
+ * Copyright 2012-2020 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.springframework.cloud.client.loadbalancer.reactive;
+
+import org.springframework.http.HttpMethod;
+import org.springframework.web.reactive.function.client.ClientRequest;
+import org.springframework.web.reactive.function.client.ClientResponse;
+
+/**
+ * Stores the data for a load-balanced call that is being retried.
+ *
+ * @author Olga Maciaszek-Sharma
+ * @since 3.0.0
+ */
+class LoadBalancerRetryContext {
+
+	private final ClientRequest request;
+
+	private ClientResponse clientResponse;
+
+	private Integer retriesSameServiceInstance = 0;
+
+	private Integer retriesNextServiceInstance = 0;
+
+	LoadBalancerRetryContext(ClientRequest request) {
+		this.request = request;
+	}
+
+	ClientRequest getRequest() {
+		return request;
+	}
+
+	ClientResponse getClientResponse() {
+		return clientResponse;
+	}
+
+	void setClientResponse(ClientResponse clientResponse) {
+		this.clientResponse = clientResponse;
+	}
+
+	Integer getRetriesSameServiceInstance() {
+		return retriesSameServiceInstance;
+	}
+
+	void incrementRetriesSameServiceInstance() {
+		retriesSameServiceInstance++;
+	}
+
+	void resetRetriesSameServiceInstance() {
+		retriesSameServiceInstance = 0;
+	}
+
+	Integer getRetriesNextServiceInstance() {
+		return retriesNextServiceInstance;
+	}
+
+	void incrementRetriesNextServiceInstance() {
+		retriesNextServiceInstance++;
+	}
+
+	Integer getResponseStatusCode() {
+		return clientResponse.statusCode().value();
+	}
+
+	HttpMethod getRequestMethod() {
+		return request.method();
+	}
+
+}

spring-cloud-commons/src/main/java/org/springframework/cloud/client/loadbalancer/reactive/LoadBalancerRetryPolicy.java

@@ -0,0 +1,58 @@
+/*
+ * Copyright 2012-2020 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.springframework.cloud.client.loadbalancer.reactive;
+
+import org.springframework.http.HttpMethod;
+
+/**
+ * Pluggable policy used to establish whether a given load-balanced call should be
+ * retried.
+ *
+ * @author Olga Maciaszek-Sharma
+ * @since 3.0.0
+ */
+public interface LoadBalancerRetryPolicy {
+
+	/**
+	 * Return <code>true</code> to retry on the same service instance.
+	 * @param context the context for the retry operation
+	 * @return true to retry on the same service instance
+	 */
+	boolean canRetrySameServiceInstance(LoadBalancerRetryContext context);
+
+	/**
+	 * Return <code>true</code> to retry on the next service instance.
+	 * @param context the context for the retry operation
+	 * @return true to retry on the same service instance
+	 */
+	boolean canRetryNextServiceInstance(LoadBalancerRetryContext context);
+
+	/**
+	 * Return <code>true</code> to retry on the provided HTTP status code.
+	 * @param statusCode the HTTP status code
+	 * @return true to retry on the provided HTTP status code
+	 */
+	boolean retryableStatusCode(int statusCode);
+
+	/**
+	 * Return <code>true</code> to retry on the provided HTTP method.
+	 * @param method the HTTP request method
+	 * @return true to retry on the provided HTTP method
+	 */
+	boolean canRetryOnMethod(HttpMethod method);
+
+}