Update spark version 2.4.2 #87

vishalbollu · 2019-04-25T20:25:26Z

Spark 2.4.2 was built with Scala 2.12.
Spark 2.4.1 was built with Scala 2.11.
Tensorflow connector was built with Scala 2.11.

As a result of bumping Scala version, the Scalatest version needed to be bumped as well.

The changes to spark-base Dockerfile address these changes.

Checklist:

Run make test and make lint
Test end to end manually (e.g. build/push all images, restart local operator, and run cx refresh in an example folder)
Update documentation
Update examples and cx init
Alert team if dev environment changed
Cherry-pick into release branches if it's a bugfix
Delete the branch once it's merged

…ith scala 2.12

vishalbollu added 2 commits April 25, 2019 11:32

Upgrade spark from 2.4.1 to 2.4.2

53be7be

Update tensorflow connector to be compatible with spark 2.4.2 built w…

aafd387

…ith scala 2.12

vishalbollu requested review from deliahu and 1vn April 25, 2019 20:25

vishalbollu added 2 commits April 25, 2019 20:29

Address line length in Dockerfile

a2ab7a5

Group args together

e99f442

deliahu approved these changes Apr 25, 2019

View reviewed changes

vishalbollu merged commit 0f19ef3 into master Apr 25, 2019

vishalbollu deleted the update-spark-version-2.4.2 branch April 26, 2019 23:18