Expose Spring Data Repository metrics

[mp911de]

Spring Data repository invocations are not metered yet and it would make sense to expose invocation metrics to improve observability from a repository perspective.

Ideally, repository metrics measure the number of invocations and the timing for each repository method. The metering should use a Micrometer Timer with the following tags per method invocation:

• repository: Simple name of the repository interface class
• method: Method name of the invoked method
• invocation: Invocation type: SYNC (Person, List, …), ASYNC (CompletionStage, ListenableFuture), Stream (Java 8 Stream), REACTIVE (a supported Reactive type according to ReactiveAdapterRegistry)
• result: Whether the method invocation yielded a result (NULL, EMPTY (for Optional.empty() or an empty Collection), PRESENT (for Optional.of(…)))
• exception: Simple name of the exception if an exception was thrown

Specifics to consider:

• Asynchronous calls: Repository queries may offload calls to a worker thread and return CompletionStage or ListenableFuture
• Reactive repositories: Return a reactive type, metrics get collected on success or on error
• Stream queries: Metrics get collected when the Stream is fully consumed/Stream gets closed

Metrics can be collected through a MethodInterceptor as repositories are pure proxy objects that internally dispatch method calls, so from an outside an interceptor seems appropriate. The interceptor can be attached through a RepositoryProxyPostProcessor which needs to be configured on repository factory beans (RepositoryFactoryBeanSupport). That change needs to be done in Spring Data Commons (see DATACMNS-1688).

I have a PoC that uses BeanPostProcessor.postProcessBeforeInitialization(…) so we can turn that into a pull request.

Likely, this feature would require a bit of auto-configuration since metrics so we would need to find a good spot for configuration properties.

Limitations:
JPA runs some activity that happens outside of repository calls (e.g. lazy loading, defer activities until transaction cleanup). These activities would not be included in these metrics.

[wilkinsona]

Thanks for the suggestion, @mp911de.

On first impression it sounds to me like much of the code should go in Spring Data with Spring Boot then auto-configuring things when both Micrometer and Spring Data are on the classpath. I'm imagining Spring Data providing a MicrometerMetricsMethodInterceptor or the like that we can configure when appropriate.

This would be similar to the approach that's been taken in other projects such as Spring Kafka that provides MicrometerConsumerListener and MicrometerProducerListener that we then auto-configure:

spring-boot/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/metrics/KafkaMetricsAutoConfiguration.java

Lines 56 to 72 in ecbc8ea

	@Bean
	public DefaultKafkaProducerFactoryCustomizer kafkaProducerMetrics(MeterRegistry meterRegistry) {
	return (producerFactory) -> addListener(producerFactory, meterRegistry);
	}
	@Bean
	public DefaultKafkaConsumerFactoryCustomizer kafkaConsumerMetrics(MeterRegistry meterRegistry) {
	return (consumerFactory) -> addListener(consumerFactory, meterRegistry);
	}
	private <K, V> void addListener(DefaultKafkaConsumerFactory<K, V> factory, MeterRegistry meterRegistry) {
	factory.addListener(new MicrometerConsumerListener<>(meterRegistry));
	}
	private <K, V> void addListener(DefaultKafkaProducerFactory<K, V> factory, MeterRegistry meterRegistry) {
	factory.addListener(new MicrometerProducerListener<>(meterRegistry));
	}

[mp911de]

I’m quite reluctant to introduce Micrometer to Spring Data as we would introduce another dependency constraint on our release process. Metrics and statistics aren’t things that Spring Data had as domain and I’d like to keep it that way. We’d rather go for a similar arrangement as with WebMVC/WebFlux metrics.

[philwebb]

Whilst I think this would be a nice addition, I don't think that we should try to add too much custom code in Spring Boot to do this. We've already been trying to move some of our MVC metrics code into Spring Framework where it has a more natural home. The specific complexities around asynchronous calls make me quite wary that we'll be maintaining a lot of code without the real expertise required to support it. It's still worth looking at the prototype code that's already been developed, just to see if my concerns are ill-founded or not.

My initial feeling is that we should look at something similar to Hikari. They have a MetricsTrackerFactory interface that creates a IMetricsTracker instance. This provides some nice high-level metrics hook points but doesn't directly tie anything to Micrometer. In that example, there's a MicrometerMetricsTracker IMetricsTracker implementation that eventually bridges to Micrometer. Spring Boot simply configures the datasource to use it.

I like that model a lot because it makes keeps metrics in the domain of the code that understands it best, but doesn't create any hard dependency on any one library.

@mp911de can you share what you have so far (it doesn't need to be a pull-request). Does your prototype code deal with the specific concerns you mentioned in the top issue comment?

[mp911de]

Thanks, Phil. Your words frame exactly the issue. I'm also concerned about the amount of code if all of this functionality would be handled by Spring Boot. We should come up with an approach that doesn't require any outside library to make the same assumptions over execution details as Spring Data does. At the same time, we need something that is able to expose all required details so Spring Boot (or any other library) can collect metrics.

Let me investigate on that topic so I can come back with an API proposal. Here's my draft that uses a MethodInterceptor. The draft deals with all concerns except for Kotlin Coroutine methods.

[wilkinsona]

@mp911de How's this going on the Spring Data side?

[mp911de]

We've provided with Spring Data 2.4 an API to listen for repository invocation along with metering the invocation duration. Here's an example: https://github.com/spring-projects/spring-data-examples/tree/master/mongodb/repository-metrics

[wilkinsona]

Cool, thanks. Let's see if we can do something with this in Spring Boot 2.5.

[sanjayrawat1]

@mp911de I took the reference of your draft of using MethodInterceptor for repository metrics, which is exposing metrics but I lose the histogram config, common tags and other configurations which I configured in application.yml. Could you please guide me why this is happening. I tried annotating RepositoryMetricsConfig class with @AutoConfigureAfter(MetricsAutoConfiguration.class) but still no luck.

[sanjayrawat1]

Below is the sample (inspired by @mp911de draft) which is working in 2.4.x version without losing histogram and other configs defined in application.yml and I believe this will also work in 2.3.x:

public class JpaMetricsRepositoryFactoryBean<T extends Repository<S, ID>, S, ID> extends JpaRepositoryFactoryBean<T, S, ID> {

	private MeterRegistry registry;

	private RepositoryTagsProvider tagsProvider;

	public JpaMetricsRepositoryFactoryBean(Class<? extends T> repositoryInterface) {
		super(repositoryInterface);
	}

	@Autowired
	public void setRegistry(ObjectProvider<MeterRegistry> registry) {
		this.registry = registry.getIfAvailable(() -> Metrics.globalRegistry);
	}

	@Autowired
	public void setTagsProvider(ObjectProvider<RepositoryTagsProvider> tagsProvider) {
		this.tagsProvider = tagsProvider.getIfAvailable(DefaultRepositoryTagsProvider::new);
	}

	@Override
	protected RepositoryFactorySupport createRepositoryFactory(EntityManager entityManager) {
		RepositoryFactorySupport repositoryFactory = super.createRepositoryFactory(entityManager);
		repositoryFactory.addRepositoryProxyPostProcessor(new RepositoryMetricsConfiguration.RepositoryMetricsProxyPostProcessor(registry, tagsProvider));
		return repositoryFactory;
	}
}

@Slf4j
@Configuration
@EnableJpaRepositories(repositoryFactoryBeanClass = JpaMetricsRepositoryFactoryBean.class)
@EnableTransactionManagement
public class DatabaseConfiguration {}

public class RepositoryMetricsConfiguration {

	@Bean
	@ConditionalOnMissingBean(RepositoryTagsProvider.class)
	public DefaultRepositoryTagsProvider repositoryTagsProvider(ObjectProvider<RepositoryTagsContributor> contributors) {
		return new DefaultRepositoryTagsProvider(contributors.orderedStream().collect(Collectors.toList()));
	}

	static class RepositoryMetricsProxyPostProcessor implements RepositoryProxyPostProcessor {

		private final MeterRegistry registry;

		private final RepositoryTagsProvider tagsProvider;

		public RepositoryMetricsProxyPostProcessor(MeterRegistry registry, RepositoryTagsProvider tagsProvider) {
			this.registry = registry;
			this.tagsProvider = tagsProvider;
		}

		@Override
		public void postProcess(ProxyFactory factory, RepositoryInformation repositoryInformation) {
			MetricsMethodInterceptor interceptor = new MetricsMethodInterceptor(
				registry,
				repositoryInformation.getRepositoryInterface(),
				tagsProvider,
				"spring.data.repositories",
				AutoTimer.ENABLED
			);
			factory.addAdvice(interceptor);
		}
	}

	static class MetricsMethodInterceptor implements MethodInterceptor {

		private final MeterRegistry registry;

		private final Class<?> repositoryInterface;

		private final RepositoryTagsProvider tagsProvider;

		private final String metricName;

		private final AutoTimer autoTimer;

		// prettier-ignore
		public MetricsMethodInterceptor(MeterRegistry registry, Class<?> repositoryInterface, RepositoryTagsProvider tagsProvider,
										String metricName, AutoTimer autoTimer) {
			this.registry = registry;
			this.repositoryInterface = repositoryInterface;
			this.tagsProvider = tagsProvider;
			this.metricName = metricName;
			this.autoTimer = autoTimer;
		}

		@Override
		public Object invoke(MethodInvocation invocation) throws Throwable {
			Timer.Sample timerSample = Timer.start(this.registry);
			try {
				Object result = invocation.proceed();
				// record
				return result;
			} catch (Throwable throwable) {
				// record
				throw throwable;
			}
		}
	}
}