forked from delta-io/delta-sharing
-
Notifications
You must be signed in to change notification settings - Fork 0
Test the server with the delta sharing client #129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
tmnd1991
merged 21 commits into
main
from
feature/128-test-the-server-with-the-delta-sharing-client
Dec 1, 2023
Merged
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
065171d
rebased
1d180fc
remove delta and aws dependencies from server:app
a02298d
trying to autogenerate the http client
434b5e6
Fix build
agilelab-tmnd1991 009bfe1
use the openapi autogen client
54a0256
apply spotless
701b96d
run IT tests in CI/CD
ab7f78f
fix gradle build: Task ':client-spark:spotlessJava' uses this output …
a6de41f
Don't run spark-client:test early
agilelab-tmnd1991 204f010
fix jar path
agilelab-tmnd1991 8d6949e
Run --no-daemon
agilelab-tmnd1991 702e4f4
tag ITDeltaSharingClient as an integrationTest
976d0e5
define ClientSparkTest tag
fa2b721
exclude ClientSparkTest tag from Test
534bb2c
remove exclusion
2879876
remove group and version
2ea0873
add starting version
cb511f4
throw an exception when getting starting/ending version
3157de3
move up version checks
79d2388
check also endingVersion
8db51e3
Update server/app/src/main/java/io/whitefox/api/deltasharing/DeltaMap…
agilelab-tmnd1991 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,86 @@ | ||
import org.openapitools.generator.gradle.plugin.tasks.GenerateTask | ||
|
||
plugins { | ||
java | ||
id("com.diffplug.spotless") | ||
id("whitefox.java-conventions") | ||
} | ||
|
||
repositories { | ||
mavenCentral() | ||
} | ||
|
||
dependencies { | ||
// OPENAPI | ||
implementation("org.eclipse.microprofile.openapi:microprofile-openapi-api:3.1.1") | ||
implementation("org.openapitools:jackson-databind-nullable:0.2.6") | ||
testImplementation("com.fasterxml.jackson.datatype:jackson-datatype-jsr310") | ||
testImplementation("jakarta.annotation:jakarta.annotation-api:2.1.1") | ||
|
||
// DELTA | ||
testImplementation("org.apache.hadoop:hadoop-common:3.3.6") | ||
testImplementation("io.delta:delta-sharing-spark_2.12:1.0.2") | ||
|
||
//SPARK | ||
testImplementation("org.apache.spark:spark-core_2.12:3.3.2") | ||
testImplementation("org.apache.spark:spark-sql_2.12:3.3.2") | ||
testImplementation("com.github.mrpowers:spark-fast-tests_2.12:1.3.0") | ||
|
||
//JUNIT | ||
testImplementation("org.junit.jupiter:junit-jupiter:5.8.1") | ||
} | ||
|
||
|
||
tasks.getByName<Test>("test") { | ||
useJUnitPlatform { | ||
excludeTags.add("clientSparkTest") | ||
} | ||
} | ||
|
||
tasks.withType<Test> { | ||
environment = env.allVariables | ||
systemProperty ("java.util.logging.manager", "java.util.logging.LogManager") //TODO modularize the whitefox-conventions plugin | ||
} | ||
|
||
tasks.register<Test>("clientSparkTest") { | ||
useJUnitPlatform { | ||
includeTags.add("clientSparkTest") | ||
} | ||
} | ||
|
||
val openApiCodeGenDir = "generated/openapi" | ||
val generatedCodeDirectory = generatedCodeDirectory(layout, openApiCodeGenDir) | ||
|
||
val whiteFoxGenerate = tasks.register<GenerateTask>("openapiGenerateClientApi") { | ||
dependsOn(tasks.spotlessApply) | ||
generatorName.set("java") | ||
inputSpec.set("$rootDir/protocol/whitefox-protocol-api.yml") | ||
library.set("native") | ||
outputDir.set(generatedCodeDirectory) | ||
additionalProperties.set(mapOf( | ||
"apiPackage" to "io.whitefox.api.client", | ||
"invokerPackage" to "io.whitefox.api.utils", | ||
"modelPackage" to "io.whitefox.api.client.model", | ||
"dateLibrary" to "java8", | ||
"sourceFolder" to "src/gen/java", | ||
"openApiNullable" to "true", | ||
"annotationLibrary" to "none", | ||
"serializationLibrary" to "jackson", | ||
"useJakartaEe" to "true", | ||
"useRuntimeException" to "true" | ||
)) | ||
} | ||
|
||
sourceSets { | ||
getByName("test") { | ||
java { | ||
srcDir("${generatedCodeDirectory(layout, openApiCodeGenDir)}/src/gen/java") | ||
} | ||
} | ||
} | ||
|
||
tasks.withType<JavaCompile> { | ||
options.encoding = "UTF-8" | ||
options.compilerArgs.add("-parameters") | ||
dependsOn(whiteFoxGenerate) | ||
} |
61 changes: 61 additions & 0 deletions
61
client-spark/src/test/java/io/whitefox/api/client/ITDeltaSharingClient.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
package io.whitefox.api.client; | ||
|
||
import static org.junit.jupiter.api.Assertions.assertEquals; | ||
|
||
import com.github.mrpowers.spark.fast.tests.DatasetComparer; | ||
import io.whitefox.api.models.MrFoxDeltaTableSchema; | ||
import io.whitefox.api.utils.StorageManagerInitializer; | ||
import java.util.List; | ||
import org.apache.spark.sql.SparkSession; | ||
import org.apache.spark.sql.types.DataType; | ||
import org.apache.spark.sql.types.Metadata; | ||
import org.apache.spark.sql.types.StructField; | ||
import org.apache.spark.sql.types.StructType; | ||
import org.junit.jupiter.api.BeforeAll; | ||
import org.junit.jupiter.api.Tag; | ||
import org.junit.jupiter.api.Test; | ||
import scala.collection.GenMap; | ||
|
||
@Tag("clientSparkTest") | ||
public class ITDeltaSharingClient implements DatasetComparer { | ||
|
||
private final String tablePath = String.format( | ||
"%s#%s.%s.%s", | ||
getClass().getClassLoader().getResource("MrFoxProfile.json"), | ||
"s3share", | ||
"s3schema", | ||
"s3Table1"); | ||
|
||
private final SparkSession spark = SparkSession.builder() | ||
.appName("delta sharing client test") | ||
.config("spark.driver.host", "localhost") | ||
.master("local[1, 4]") | ||
.getOrCreate(); | ||
|
||
@BeforeAll | ||
static void initStorageManager() { | ||
new StorageManagerInitializer().initStorageManager(); | ||
} | ||
|
||
@Test | ||
void showS3Table1withQueryTableApi() { | ||
var ds = spark.read().format("deltaSharing").load(tablePath); | ||
var expectedSchema = new StructType(new StructField[] { | ||
new StructField("id", DataType.fromDDL("long"), true, new Metadata(GenMap.empty())) | ||
}); | ||
var expectedData = spark | ||
.createDataFrame( | ||
List.of( | ||
new MrFoxDeltaTableSchema(0), | ||
new MrFoxDeltaTableSchema(3), | ||
new MrFoxDeltaTableSchema(2), | ||
new MrFoxDeltaTableSchema(1), | ||
new MrFoxDeltaTableSchema(4)), | ||
MrFoxDeltaTableSchema.class) | ||
.toDF(); | ||
|
||
assertEquals(expectedSchema.json(), ds.schema().json()); | ||
assertEquals(5, ds.count()); | ||
assertSmallDatasetEquality(ds, expectedData, true, false, false, 500); | ||
} | ||
} |
13 changes: 13 additions & 0 deletions
13
client-spark/src/test/java/io/whitefox/api/models/MrFoxDeltaTableSchema.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
package io.whitefox.api.models; | ||
|
||
public class MrFoxDeltaTableSchema { | ||
private final long id; | ||
|
||
public MrFoxDeltaTableSchema(long id) { | ||
this.id = id; | ||
} | ||
|
||
public long getId() { | ||
return id; | ||
} | ||
} |
32 changes: 32 additions & 0 deletions
32
client-spark/src/test/java/io/whitefox/api/utils/S3TestConfig.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
package io.whitefox.api.utils; | ||
|
||
public class S3TestConfig { | ||
private final String region; | ||
private final String accessKey; | ||
private final String secretKey; | ||
|
||
public String getRegion() { | ||
return region; | ||
} | ||
|
||
public String getAccessKey() { | ||
return accessKey; | ||
} | ||
|
||
public String getSecretKey() { | ||
return secretKey; | ||
} | ||
|
||
public S3TestConfig(String region, String accessKey, String secretKey) { | ||
this.region = region; | ||
this.accessKey = accessKey; | ||
this.secretKey = secretKey; | ||
} | ||
|
||
public static S3TestConfig loadFromEnv() { | ||
return new S3TestConfig( | ||
System.getenv().get("WHITEFOX_TEST_AWS_REGION"), | ||
System.getenv().get("WHITEFOX_TEST_AWS_ACCESS_KEY_ID"), | ||
System.getenv().get("WHITEFOX_TEST_AWS_SECRET_ACCESS_KEY")); | ||
} | ||
} |
77 changes: 77 additions & 0 deletions
77
client-spark/src/test/java/io/whitefox/api/utils/StorageManagerInitializer.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
package io.whitefox.api.utils; | ||
|
||
import io.whitefox.api.client.*; | ||
import io.whitefox.api.client.model.*; | ||
import java.util.List; | ||
import java.util.Map; | ||
|
||
public class StorageManagerInitializer { | ||
private final S3TestConfig s3TestConfig; | ||
private final StorageV1Api storageV1Api; | ||
private final ProviderV1Api providerV1Api; | ||
private final TableV1Api tableV1Api; | ||
private final ShareV1Api shareV1Api; | ||
private final SchemaV1Api schemaV1Api; | ||
|
||
public StorageManagerInitializer() { | ||
var apiClient = new ApiClient(); | ||
this.s3TestConfig = S3TestConfig.loadFromEnv(); | ||
this.storageV1Api = new StorageV1Api(apiClient); | ||
this.providerV1Api = new ProviderV1Api(apiClient); | ||
this.tableV1Api = new TableV1Api(apiClient); | ||
this.shareV1Api = new ShareV1Api(apiClient); | ||
this.schemaV1Api = new SchemaV1Api(apiClient); | ||
} | ||
|
||
public void initStorageManager() { | ||
storageV1Api.createStorage(createStorageRequest(s3TestConfig)); | ||
providerV1Api.addProvider(addProviderRequest()); | ||
tableV1Api.createTableInProvider(addProviderRequest().getName(), createTableRequest()); | ||
shareV1Api.createShare(createShareRequest()); | ||
schemaV1Api.createSchema(createShareRequest().getName(), createSchemaRequest()); | ||
schemaV1Api.addTableToSchema( | ||
createShareRequest().getName(), createSchemaRequest(), addTableToSchemaRequest()); | ||
} | ||
|
||
private String createSchemaRequest() { | ||
return "s3schema"; | ||
} | ||
|
||
private AddTableToSchemaRequest addTableToSchemaRequest() { | ||
return new AddTableToSchemaRequest() | ||
.name("s3Table1") | ||
.reference(new TableReference().providerName("MrFoxProvider").name("s3Table1")); | ||
} | ||
|
||
private CreateShareInput createShareRequest() { | ||
return new CreateShareInput().name("s3share").recipients(List.of("Mr.Fox")).schemas(List.of()); | ||
} | ||
|
||
private CreateTableInput createTableRequest() { | ||
return new CreateTableInput() | ||
.name("s3Table1") | ||
.skipValidation(true) | ||
.properties(Map.of( | ||
"type", "delta", | ||
"location", "s3a://whitefox-s3-test-bucket/delta/samples/delta-table")); | ||
} | ||
|
||
private ProviderInput addProviderRequest() { | ||
return new ProviderInput() | ||
.name("MrFoxProvider") | ||
.storageName("MrFoxStorage") | ||
.metastoreName(null); | ||
} | ||
|
||
private CreateStorage createStorageRequest(S3TestConfig s3TestConfig) { | ||
return new CreateStorage() | ||
.name("MrFoxStorage") | ||
.type(CreateStorage.TypeEnum.S3) | ||
.properties(new StorageProperties(new S3Properties() | ||
.credentials(new SimpleAwsCredentials() | ||
.region(s3TestConfig.getRegion()) | ||
.awsAccessKeyId(s3TestConfig.getAccessKey()) | ||
.awsSecretAccessKey(s3TestConfig.getSecretKey())))) | ||
.skipValidation(true); | ||
} | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
{ | ||
"shareCredentialsVersion": 1, | ||
"endpoint": "http://localhost:8080/delta-api/v1/", | ||
"bearerToken": "fakeToken", | ||
"expirationTime": null | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.