OutOfMemoryError - issue with metrics

[EdgarsM]

Application fails with OutOfMemoryError after several days without restart. Issue is caused by Spring Boot issue #13895 - there is no possibility to disable request auto timing using property management.metrics.web.server.auto-time-requests. Value of this property should be set to false in case of spring-cloud-gateway otherwise it might be an issue if URIs contains path variables with large cardinality. Instead of recording single metric with placeholders multiple metrics will be created.
Seems like issue #171 was related to same root cause - in case of #171 removing micrometer dependency removed MeterRegistry from class-path and as a result org.springframework.boot.actuate.autoconfigure.metrics.web.servlet.WebMvcMetricsAutoConfiguration got disabled by rule @ConditionalOnBean(MeterRegistry.class).

[EdgarsM]

Current workaround is to remove Micrometer dependency just as in #171 but in case you would still like to collect metrics for your application WebFluxMetricsAutoConfiguration auto configuration can be excluded so that you can still benefit from all metrics except HTTP request auto timing.

[ryanjbaxter]

I dont see what there is to do in the Gateway if the problem is with Boot.

[EdgarsM]

I think it would be good to disable auto time requests in spring-cloud-gateway (perhaps override defaults of management.metrics.web.server.auto-time-requests) or clearly state that by not disabling auto request timing it could blow up metric cardinality. What are your thoughts?

[spencergibb]

Ping @jkschneider

[jkschneider]

I think we should instead introduce a MeterFilter#maximumAllowableTags for this metric much like we do for RestTemplate metrics where we can't guarantee that folks don't incorrectly use RestTemplate with path variable substitution.

If you'd like, you can create a variation on this that does something else like logging periodically when the maximum is reached and pointing folks to probably the better solution which is to use their domain knowledge of the structure of their URIs to construct a MeterFilter that rewrites tag values to reduce cardinality.

[ivelkov]

I believe we can use the following WebFluxTagsProvider implementation as a workaround until this gets a proper fix.

@Component
@Primary
public class CustomWebMVcTagsProvider implements WebFluxTagsProvider {

  private static final Tag URI_UNKNOWN = Tag.of("uri", "UNKNOWN");

  @Override
  public Iterable<Tag> httpRequestTags(ServerWebExchange exchange, Throwable ex) {
    return Arrays.asList(WebFluxTags.method(exchange), uri(exchange),
        WebFluxTags.exception(ex), WebFluxTags.status(exchange));
  }

  private Tag uri(ServerWebExchange exchange) {
    Tag tag = URI_UNKNOWN;
    if (exchange != null) {
      final Route route = exchange.getAttribute(GATEWAY_ROUTE_ATTR);
      if (route != null) {
        tag = Tag.of("uri", route.getUri().toString());
      }
    }
    return tag;
  }
}

Cheers

[ryanjbaxter]

@EdgarsM is this still a problem since the Boot issue was fixed in Boot 2.0.6?