superpmi collection: benchmark collection FSharp.dll fails

[BruceForstall]

I was looking at the "SuperPMI collect realworld run linux x64 checked" job in superpmi collection https://dev.azure.com/dnceng/internal/_build/results?buildId=2533665&view=results

Partition1:

Top command:
+ /usr/bin/python3.10 /datadisks/disk1/work/B19A097C/p/superpmi/superpmi_benchmarks.py -performance_directory /datadisks/disk1/work/B19A097C/w/AA450939/u/performance -superpmi_directory /datadisks/disk1/work/B19A097C/p/superpmi -core_root /datadisks/disk1/work/B19A097C/p/superpmi -arch x64 -partition_count 1 -partition_index 0 -benchmark_path src/benchmarks/real-world/FSharp/FSharp.fsproj -benchmark_binary FSharp.dll -output_mch_path /datadisks/disk1/work/B19A097C/w/AA450939/uploads/realworld.run..linux.x64.checked.mch -log_file /datadisks/disk1/work/B19A097C/w/AA450939/uploads/realworld.run..linux.x64.checked.log

...

/datadisks/disk1/work/B19A097C/t/tmp_xsrl0_v/run_benchmarks.sh contents:
******************************************
#!/bin/bash
export JitName=$DOTNET_JitName
unset DOTNET_JitName
unset DOTNET_TieredCompilation
unset DOTNET_TieredPGO
pushd /datadisks/disk1/work/B19A097C/w/AA450939/u/performance
/datadisks/disk1/work/B19A097C/w/AA450939/u/performance/tools/dotnet/x64/dotnet /datadisks/disk1/work/B19A097C/w/AA450939/u/performance/artifacts/FSharp.dll --corerun /datadisks/disk1/work/B19A097C/p/superpmi/corerun --filter "*" --iterationCount 1 --warmupCount 0 --invocationCount 1 --unrollFactor 1 --strategy ColdStart --logBuildOutput --envVars DOTNET_JitName:$JitName DOTNET_ReadyToRun:0 DOTNET_TieredCompilation:0
******************************************
Inside make_executable
Running: ls -l /datadisks/disk1/work/B19A097C/t/tmp_xsrl0_v/run_benchmarks.sh
-rw-r--r-- 1 helixbot helixbot 600 Sep  8 23:22 /datadisks/disk1/work/B19A097C/t/tmp_xsrl0_v/run_benchmarks.sh

Running: ls -l /datadisks/disk1/work/B19A097C/t/tmp_xsrl0_v/run_benchmarks.sh
-r-xr-xr-x 1 helixbot helixbot 600 Sep  8 23:22 /datadisks/disk1/work/B19A097C/t/tmp_xsrl0_v/run_benchmarks.sh

Running: /usr/bin/python3.10 /datadisks/disk1/work/B19A097C/p/superpmi/superpmi.py collect --clean -core_root /datadisks/disk1/work/B19A097C/p/superpmi -log_file /datadisks/disk1/work/B19A097C/w/AA450939/uploads/realworld.run..linux.x64.checked.log -output_mch_path /datadisks/disk1/work/B19A097C/w/AA450939/uploads/realworld.run..linux.x64.checked.mch_all.mch -log_level debug /datadisks/disk1/work/B19A097C/t/tmp_xsrl0_v/run_benchmarks.sh
[23:22:15] ================ Logging to /datadisks/disk1/work/B19A097C/w/AA450939/uploads/realworld.run..linux.x64.checked.log
[23:22:15] SuperPMI collect
[23:22:15] ------------------------------------------------------------
[23:22:15] Start time: 23:22:15
[23:22:15] Using superpmi from Core_Root: /datadisks/disk1/work/B19A097C/p/superpmi/superpmi
[23:22:15] Using mcs from Core_Root: /datadisks/disk1/work/B19A097C/p/superpmi/mcs
[23:22:15] SuperPMI JIT Path: /datadisks/disk1/work/B19A097C/p/superpmi/libclrjit.so
[23:22:15] Starting collection.
[23:22:15] 
[23:22:15] Starting collection using command
[23:22:15] export SuperPMIShimLogPath=/datadisks/disk1/work/B19A097C/t/tmpe1xg40qm
[23:22:15] export SuperPMIShimPath=/datadisks/disk1/work/B19A097C/p/superpmi/libclrjit.so
[23:22:15] export DOTNET_EnableExtraSuperPmiQueries=1
[23:22:15] export DOTNET_TieredCompilation=0
[23:22:15] export DOTNET_JitPath=/datadisks/disk1/work/B19A097C/p/superpmi/libsuperpmi-shim-collector.so
[23:22:15] export DOTNET_JitName=libsuperpmi-shim-collector.so
[23:22:15] Collecting using command:
[23:22:15]   /datadisks/disk1/work/B19A097C/t/tmp_xsrl0_v/run_benchmarks.sh 
[23:22:16] /datadisks/disk1/work/B19A097C/w/AA450939/u/performance /datadisks/disk1/work/B19A097C/t/tmp_xsrl0_v
[23:22:16] No benchmarks to choose from. Make sure you provided public non-sealed non-static types with public [Benchmark] methods.
[23:22:16] Done. Elapsed time: 0:00:00.340947
[23:22:16] No .mc files generated.
[23:22:16] Finish time: 23:22:16
[23:22:16] Elapsed time: 0:00:00.343393

Command failed. Exiting.
+ export _commandExitCode=1
+ exit 1
['Partition1' END OF WORK ITEM LOG: Command exited with 1]

Apparently FSharp.dll is not a benchmark binary?

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[BruceForstall]

This is due to dotnet/performance#4358

cc @TIHan -- do we have enough spmi F# coverage left after this?

[BruceForstall]

Fixed with #107588