Another Example of Running Spark on AKS
Add the following to build.sbt
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.0" % "provided"
libraryDependencies += "com.microsoft.azure" % "spark-mssql-connector_2.12" % "1.2.0"
libraryDependencies += "com.microsoft.sqlserver" % "mssql-jdbc" % "10.2.0.jre11"Example command for a spark submit job:
./bin/spark-submit \
--master k8s://http://127.0.0.1:8080 \
--deploy-mode cluster \
--name sqlserverexample \
--class sqlexample.SQLServerExample \
--conf spark.executor.instances=3 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.container.image=cjm1909jf.azurecr.io/spark:v1 \
--conf spark.hadoop.fs.azure.account.key.sparkdemo45jh.blob.core.windows.net=<storage account key> \
--packages org.apache.hadoop:hadoop-azure:3.2.0,com.microsoft.azure:azure-storage:8.6.3 \
--conf spark.driver.extraJavaOptions="-Divy.cache.dir=/tmp -Divy.home=/tmp" \
--conf spark.kubernetes.file.upload.path=wasbs://[email protected]/ file:///home/cmcdougall/SparkPi-assembly-0.1.0-SNAPSHOT.jar