Supporting multiple LLMs in auto-config

[ddobrin]

Currently, proprietary or open models running in Google Vertex can be accessed through the Vertex Gemini module, respectively using the OpenAI API.

#################################
# Google Vertex AI Gemini
#################################
spring.ai.vertex.ai.gemini.project-id=<project>
spring.ai.vertex.ai.gemini.location=us-central1
spring.ai.vertex.ai.gemini.chat.options.model=gemini-2.0-flash-001
spring.ai.vertex.ai.gemini.transport=grpc

#################################
# OpenAI API VertexAI
#################################
spring.ai.openai.api-key=abc123
spring.ai.openai.vertex.ai.gemini.project-id=<project>
spring.ai.openai.vertex.ai.gemini.location=us-central1
spring.ai.openai.vertex.ai.chat.options.model=meta/llama3-405b-instruct-maas
spring.ai.openai.vertex.ai.chat.base-url=<baseURL>
spring.ai.openai.vertex.ai.chat.completions-path=/chat/completions
spring.ai.openai.vertex.ai.chat.options.max-tokens=1024

The problem - there is no ability to auto-config more than 1 Gemini or open model in Vertex at this time.

The proposal - introduce the ability to add a in the auto-config, to allow for multiple configurations

spring.ai.vertex.ai.gemini.<gemini-flash>.project-id=<project>
spring.ai.vertex.ai.gemini.<gemini-flash>.location=us-central1
spring.ai.vertex.ai.gemini.<gemini-flash>.chat.options.model=gemini-2.0-flash-001
spring.ai.vertex.ai.gemini.<gemini-flash>.transport=grpc

spring.ai.vertex.ai.gemini.<gemini-pro>.project-id=<project>
spring.ai.vertex.ai.gemini.<gemini-pro>.location=us-central1
spring.ai.vertex.ai.gemini.<gemini-pro>.chat.options.model=gemini-1.5-001
spring.ai.vertex.ai.gemini.<gemini-pro>.transport=grpc

Workaround - not using auto-config and manually configuring each model in use

CC @tzolov

[ThomasVitale]

I would really like to have this possibility in Spring AI. Unfortunately, I'm afraid it's up to the Spring Boot project to support registering multiple beans of the same type via auto-configuration, a capability that so far has not been supported by Spring Boot.

I hope things will change in Spring Boot 4, based on the new changes introduced in Spring Framework 7 for programmatically registering beans, that would enable such functionality (https://docs.spring.io/spring-framework/reference/7.0/core/beans/java/programmatic-bean-registration.html). See spring-projects/spring-boot#15732.

[markpollack]

I think that isn't really a deal breaker, there can be work around. Need an epic to research and discuss.

[ThomasVitale]

@markpollack I created this issue to be used as an epic to track the discussion: #3518

[paulbakker]

I would really like to see this supported as well. I think this could be done by adding a qualifier to each bean created, so that, if there's more than a single bean, the user can selectively inject the correct one based on the qualifier. The qualifier value could be the model name.
This likely means the beans need to be created through a BeanFactoryPostProcessor instead of regular auto configuration.

[paulbakker]

I found a workaround until this is fully supported. It's easy to mutate an existing ChatModel instance to switch the underlying model:

Let's assume I have:

@Autowired
OpenAiChatModel openAiChatModel;

I can create another instance with a different model by mutating a builder based on the original:

var evalModel = openAiChatModel.mutate().defaultOptions(OpenAiChatOptions.builder().model("another-model").build()).build();

[ThomasVitale]

@paulbakker did you find a way to use a BeanFactoryPostProcessor for this purpose? I tried that, but didn't succeed to build multiple ChatModel beans from configuration properties due to the dependency injection features not available yet at this stage of the bean lifecycle. I added some more context in #3518