System.Runtime issue

[rousebaby]

Hi All,

Might be a dumb question.

I am experienced in .net but new to the GIT and Spark. I am trying to run Spark from .net.

I tried to follow the step as shown here,

https://github.com/dotnet/spark/blob/master/README.md#get-started

After couple of struggle i was able to get in to the point of running Spark-Submit.

But when i run that

I am getting this error

Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

I know its a reference error and i know how to fix it in the VS.

But since this thing is running under Spark Interop, am not sure how to fix this.

I tried installing the System.Runtime from the nuget manager.

Thanks in advance and any help is much appreciated.

[imback82]

Hi @rousebaby, can you share the command you ran?

[rousebaby]

Hi @imback82 , Thanks for your reply.

Please find below.

spark-submit --class org.apache.spark.deploy.DotnetRunner --master local microsoft-spark-2.4.x-0.1.0.jar HelloSpark.dll

[imback82]

Can you do <full-path-dotnet.exe> HelloSpark.dll? We updated the README earlier today.

[rousebaby]

i believe when you say full path to the exe, can you please confirm is this as like below

spark-submit --class org.apache.spark.deploy.DotnetRunner --master local microsoft-spark-2.4.x-0.1.0.jar C:\XXX\HelloSpark\bin\Debug\netcoreapp2.1\HelloSpark.dll

still got the same error

[imback82]

C:\>where dotnet
C:\Program Files\dotnet\dotnet.exe

C:\Program Files\dotnet\dotnet.exe HelloSpark.dll

[rousebaby]

got it, meanwhile you posted,Thanks for that. i tried running it after finding the dotnet path. Seems i got the out put. But it was showing so many logs which i was not able to get what it is doing. Hopefully i think it ran. i was able to see this thing n the console window

+----+-------+
| age| name|
+----+-------+
|null|Michael|
| 30| Andy|
| 19| Justin|
+----+-------+

Thanks for your help.

But any luck how to suppress or customize these logs?

[rousebaby]

Also would like inputs on comparing .Net with Scala. (to note am new to Spark as well as Scala)

As shown in below image

https://devblogs.microsoft.com/dotnet/wp-content/uploads/sites/10/2019/04/dotnetsparkarchitecture-1.png

Scala is very close to the Apache Spark Core and our .Net is sitting after a Interop. (Am also seeing a SparkR dont know what it is). Yes, I have seen the metrics it said Scala is little better than .Net.

Will these interop add additional load to the process?

[imback82]

But any luck how to suppress or customize these logs?

Please refer to: https://spark.apache.org/docs/latest/configuration.html#configuring-logging

[imback82]

Will these interop add additional load to the process?

Unless you use UDFs, Scala and .NET perf should be very similar. When you have UDFs, .NET worker is launched on the executor node to process .NET UDFs and you may see perf differences depending on your UDF and workload. Please refer to https://devblogs.microsoft.com/dotnet/introducing-net-for-apache-spark/ and ".NET for Apache Spark performance" section.

[imback82]

I am closing this issue now that you resolved the issue running the sample app.

[zec-hue]

hey @imback82 , just trying to follow the readme and got all running until I got to the same path.

perhaps the documentation needs to be tidy up and that path be made more obvious?

kind regards
Zac

[imback82]

@d0ggyRul3z can you help with the documentation if something is not clear? On the other hand, we are releasing v0.2.0 soon where you don't need to provide the full path for dotnet. Will that help?

[zec-hue]

hey @imback82 , i'm happy to help. Are there any privilages that needs to be given to me?