Skip to content

Commit 4503623

Browse files
fmbenhassinefmbenhassine
committed
Add initial support for MongoDB as job repository
Issue spring-projects/spring-batch#877
1 parent 68fd366 commit 4503623

23 files changed

+1951
-1
lines changed

‎README.md

Lines changed: 71 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,74 @@
22

33
This repository contains experimental features in Spring Batch. Experimental features are not intended to be used in production.
44

5-
They are shared here to be explored by the community and to gather feedback.
5+
They are shared here to be explored by the community and to gather feedback.
6+
7+
# Enabling experimental features
8+
9+
Experimental features are not released to Maven Central, but they will be available from the Spring Milestones repository soon.
10+
11+
For now, you can build the project and install it in your local Maven repository with the following command:
12+
13+
```shell
14+
$>./mvnw clean install
15+
```
16+
17+
The experimental features are based on the latest Spring Batch 5 release, which requires Java 17+.
18+
19+
To test experimental features, you need to add the following dependency in your project:
20+
21+
```xml
22+
<dependency>
23+
<groupId>org.springframework.batch</groupId>
24+
<artifactId>spring-batch-experimental</artifactId>
25+
<version>0.1.0-SNAPSHOT</version>
26+
</dependency>
27+
```
28+
29+
Depending on the feature you are testing, other dependencies might be required.
30+
31+
# MongoDB as data store for batch meta-data
32+
33+
This feature introduces new implementations of `JobRepository` and `JobExplorer` for MongoDB.
34+
35+
To test this feature, first import the `spring-batch-experimental` jar as described in the [Enabling experimental features](#enabling-experimental-features) section.
36+
37+
Then, add the following dependencies as well:
38+
39+
```xml
40+
<dependencies>
41+
<dependency>
42+
<groupId>org.springframework.data</groupId>
43+
<artifactId>spring-data-mongodb</artifactId>
44+
<version>4.1.5</version>
45+
</dependency>
46+
<dependency>
47+
<groupId>org.mongodb</groupId>
48+
<artifactId>mongodb-driver-sync</artifactId>
49+
<version>4.9.1</version>
50+
</dependency>
51+
</dependencies>
52+
```
53+
54+
After that, you need to create the collections and sequences required by the MongoDB `JobRepository` implementation in your MongoDB server instance.
55+
Similar to the DDL scripts provided for relational databases, the MongoShell scripts for MongoDB are provided in [schema-drop-mongodb.js](src/main/resources/org/springframework/batch/core/schema-drop-mongodb.js) and [schema-mongodb.js](src/main/resources/org/springframework/batch/core/schema-mongodb.js).
56+
57+
Finally, you can define the MongoDB-based `JobRepository` and use it in your Spring Batch application as a regular `JobRepository`:
58+
59+
```java
60+
@Bean
61+
public JobRepository jobRepository(MongoTemplate mongoTemplate, MongoTransactionManager transactionManager) throws Exception {
62+
MongoJobRepositoryFactoryBean jobRepositoryFactoryBean = new MongoJobRepositoryFactoryBean();
63+
jobRepositoryFactoryBean.setMongoOperations(mongoTemplate);
64+
jobRepositoryFactoryBean.setTransactionManager(transactionManager);
65+
jobRepositoryFactoryBean.afterPropertiesSet();
66+
return jobRepositoryFactoryBean.getObject();
67+
}
68+
```
69+
70+
The implementation requires a [MongoTemplate](https://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#mongo-template) to interact with MongoDB and a [MongoTransactionManager](https://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#mongo.transactions.tx-manager) to drive Spring Batch transactions.
71+
Those can be defined as Spring beans in the application context as described in Spring Data MongoDB documentation.
72+
73+
You can find a complete example in the [MongoDBJobRepositoryIntegrationTests](./src/test/java/org/springframework/batch/experimental/core/repository/support/MongoDBJobRepositoryIntegrationTests.java) file.
74+
75+
If you find an issue, please report it on the [issue tracker](https://github.com/spring-projects-experimental/spring-batch-experimental/issues).

‎pom.xml

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,13 @@
4747
<java.version>17</java.version>
4848
<spring-batch.version>5.0.3</spring-batch.version>
4949
<maven-compiler-plugin.version>3.11.0</maven-compiler-plugin.version>
50+
<mongodb-driver-sync.version>4.9.1</mongodb-driver-sync.version>
51+
<spring-data-mongodb.version>4.1.5</spring-data-mongodb.version>
52+
53+
<junit-jupiter.version>5.10.0</junit-jupiter.version>
54+
<spring-test.version>6.0.13</spring-test.version>
55+
<testcontainers.version>1.19.1</testcontainers.version>
56+
<slf4j.version>2.0.9</slf4j.version>
5057
</properties>
5158

5259
<dependencies>
@@ -55,6 +62,49 @@
5562
<artifactId>spring-batch-core</artifactId>
5663
<version>${spring-batch.version}</version>
5764
</dependency>
65+
<dependency>
66+
<groupId>org.springframework.data</groupId>
67+
<artifactId>spring-data-mongodb</artifactId>
68+
<version>${spring-data-mongodb.version}</version>
69+
<optional>true</optional>
70+
</dependency>
71+
<dependency>
72+
<groupId>org.mongodb</groupId>
73+
<artifactId>mongodb-driver-sync</artifactId>
74+
<version>${mongodb-driver-sync.version}</version>
75+
<optional>true</optional>
76+
</dependency>
77+
78+
<dependency>
79+
<groupId>org.junit.jupiter</groupId>
80+
<artifactId>junit-jupiter</artifactId>
81+
<version>${junit-jupiter.version}</version>
82+
<scope>test</scope>
83+
</dependency>
84+
<dependency>
85+
<groupId>org.springframework</groupId>
86+
<artifactId>spring-test</artifactId>
87+
<version>${spring-test.version}</version>
88+
<scope>test</scope>
89+
</dependency>
90+
<dependency>
91+
<groupId>org.testcontainers</groupId>
92+
<artifactId>junit-jupiter</artifactId>
93+
<version>${testcontainers.version}</version>
94+
<scope>test</scope>
95+
</dependency>
96+
<dependency>
97+
<groupId>org.testcontainers</groupId>
98+
<artifactId>mongodb</artifactId>
99+
<version>${testcontainers.version}</version>
100+
<scope>test</scope>
101+
</dependency>
102+
<dependency>
103+
<groupId>org.slf4j</groupId>
104+
<artifactId>slf4j-simple</artifactId>
105+
<version>${slf4j.version}</version>
106+
<scope>test</scope>
107+
</dependency>
58108
</dependencies>
59109

60110
<build>

‎src/main/java/org/springframework/batch/experimental/core/explore/support/MongoJobExplorerFactoryBean.java

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
/*
2+
* Copyright 2023 the original author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* https://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
package org.springframework.batch.experimental.core.explore.support;
17+
18+
import org.springframework.batch.core.explore.support.AbstractJobExplorerFactoryBean;
19+
import org.springframework.batch.core.repository.dao.ExecutionContextDao;
20+
import org.springframework.batch.core.repository.dao.JobExecutionDao;
21+
import org.springframework.batch.core.repository.dao.JobInstanceDao;
22+
import org.springframework.batch.core.repository.dao.StepExecutionDao;
23+
import org.springframework.batch.experimental.core.repository.dao.MongoExecutionContextDao;
24+
import org.springframework.batch.experimental.core.repository.dao.MongoJobExecutionDao;
25+
import org.springframework.batch.experimental.core.repository.dao.MongoJobInstanceDao;
26+
import org.springframework.batch.experimental.core.repository.dao.MongoStepExecutionDao;
27+
import org.springframework.beans.factory.InitializingBean;
28+
import org.springframework.data.mongodb.core.MongoOperations;
29+
import org.springframework.util.Assert;
30+
31+
/**
32+
* @author Mahmoud Ben Hassine
33+
*/
34+
public class MongoJobExplorerFactoryBean extends AbstractJobExplorerFactoryBean implements InitializingBean {
35+
36+
private MongoOperations mongoOperations;
37+
38+
public void setMongoOperations(MongoOperations mongoOperations) {
39+
this.mongoOperations = mongoOperations;
40+
}
41+
42+
@Override
43+
protected JobInstanceDao createJobInstanceDao() {
44+
return new MongoJobInstanceDao(this.mongoOperations);
45+
}
46+
47+
@Override
48+
protected JobExecutionDao createJobExecutionDao() {
49+
return new MongoJobExecutionDao(this.mongoOperations);
50+
}
51+
52+
@Override
53+
protected StepExecutionDao createStepExecutionDao() {
54+
return new MongoStepExecutionDao(this.mongoOperations);
55+
}
56+
57+
@Override
58+
protected ExecutionContextDao createExecutionContextDao() {
59+
return new MongoExecutionContextDao(this.mongoOperations);
60+
}
61+
62+
@Override
63+
public void afterPropertiesSet() throws Exception {
64+
super.afterPropertiesSet();
65+
Assert.notNull(this.mongoOperations, "MongoOperations must not be null.");
66+
}
67+
68+
}

‎src/main/java/org/springframework/batch/experimental/core/repository/dao/MongoExecutionContextDao.java

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
/*
2+
* Copyright 2023 the original author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* https://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
package org.springframework.batch.experimental.core.repository.dao;
17+
18+
import java.util.Collection;
19+
import java.util.Map;
20+
21+
import org.springframework.batch.core.JobExecution;
22+
import org.springframework.batch.core.StepExecution;
23+
import org.springframework.batch.core.repository.dao.ExecutionContextDao;
24+
import org.springframework.batch.experimental.core.repository.persistence.converter.JobExecutionConverter;
25+
import org.springframework.batch.item.ExecutionContext;
26+
import org.springframework.data.mongodb.core.MongoOperations;
27+
import org.springframework.data.mongodb.core.query.Query;
28+
import org.springframework.data.mongodb.core.query.Update;
29+
30+
import static org.springframework.data.mongodb.core.query.Criteria.where;
31+
import static org.springframework.data.mongodb.core.query.Query.query;
32+
33+
/**
34+
* @author Mahmoud Ben Hassine
35+
*/
36+
public class MongoExecutionContextDao implements ExecutionContextDao {
37+
38+
private static final String STEP_EXECUTIONS_COLLECTION_NAME = "BATCH_STEP_EXECUTION";
39+
40+
private static final String JOB_EXECUTIONS_COLLECTION_NAME = "BATCH_JOB_EXECUTION";
41+
42+
private final JobExecutionConverter jobExecutionConverter = new JobExecutionConverter();
43+
44+
private final MongoOperations mongoOperations;
45+
46+
public MongoExecutionContextDao(MongoOperations mongoOperations) {
47+
this.mongoOperations = mongoOperations;
48+
}
49+
50+
@Override
51+
public ExecutionContext getExecutionContext(JobExecution jobExecution) {
52+
org.springframework.batch.experimental.core.repository.persistence.JobExecution execution = this.mongoOperations.findById(
53+
jobExecution.getId(), org.springframework.batch.experimental.core.repository.persistence.JobExecution.class,
54+
JOB_EXECUTIONS_COLLECTION_NAME);
55+
if (execution == null) {
56+
return new ExecutionContext();
57+
}
58+
return new ExecutionContext(execution.getExecutionContext().map());
59+
}
60+
61+
@Override
62+
public ExecutionContext getExecutionContext(StepExecution stepExecution) {
63+
org.springframework.batch.experimental.core.repository.persistence.StepExecution execution = this.mongoOperations.findById(
64+
stepExecution.getId(), org.springframework.batch.experimental.core.repository.persistence.StepExecution.class,
65+
STEP_EXECUTIONS_COLLECTION_NAME);
66+
if (execution == null) {
67+
return new ExecutionContext();
68+
}
69+
return new ExecutionContext(execution.getExecutionContext().map());
70+
}
71+
72+
@Override
73+
public void saveExecutionContext(JobExecution jobExecution) {
74+
ExecutionContext executionContext = jobExecution.getExecutionContext();
75+
Query query = query(where("_id").is(jobExecution.getId()));
76+
77+
// TODO use ExecutionContext#toMap introduced in v5.1
78+
Map<String, Object> map = Map.ofEntries(executionContext.entrySet().toArray(new Map.Entry[0]));
79+
80+
Update update = Update.update("executionContext",
81+
new org.springframework.batch.experimental.core.repository.persistence.ExecutionContext(map, executionContext.isDirty()));
82+
this.mongoOperations.updateFirst(query, update,
83+
org.springframework.batch.experimental.core.repository.persistence.JobExecution.class,
84+
JOB_EXECUTIONS_COLLECTION_NAME);
85+
}
86+
87+
@Override
88+
public void saveExecutionContext(StepExecution stepExecution) {
89+
ExecutionContext executionContext = stepExecution.getExecutionContext();
90+
Query query = query(where("_id").is(stepExecution.getId()));
91+
92+
// TODO use ExecutionContext#toMap introduced in v5.1
93+
Map<String, Object> map = Map.ofEntries(executionContext.entrySet().toArray(new Map.Entry[0]));
94+
95+
Update update = Update.update("executionContext",
96+
new org.springframework.batch.experimental.core.repository.persistence.ExecutionContext(map, executionContext.isDirty()));
97+
this.mongoOperations.updateFirst(query, update,
98+
org.springframework.batch.experimental.core.repository.persistence.StepExecution.class,
99+
STEP_EXECUTIONS_COLLECTION_NAME);
100+
101+
}
102+
103+
@Override
104+
public void saveExecutionContexts(Collection<StepExecution> stepExecutions) {
105+
for (StepExecution stepExecution : stepExecutions) {
106+
saveExecutionContext(stepExecution);
107+
}
108+
}
109+
110+
@Override
111+
public void updateExecutionContext(JobExecution jobExecution) {
112+
saveExecutionContext(jobExecution);
113+
}
114+
115+
@Override
116+
public void updateExecutionContext(StepExecution stepExecution) {
117+
saveExecutionContext(stepExecution);
118+
}
119+
120+
}

0 commit comments

Comments
 (0)