Many tests in FSharp.Tests.FSharpSuite cannot run in parallel because they compile executables with the same name `test.exe`

[abelbraaksma]

Tests in, for example FSharp.Tests.Core.CoreTests fail if they run in parallel, even if the parallelization is achieved by different processes.

Repro steps

Run all tests in FSharp.Tests.Core.CoreTests with any test runner that can run multiple tests at once in difference processes, or just fire up two test runs (i.e., NUnit GUI or VS) in parallel.

See for instance these tests and test.exe for both:

    [<Test>]
    let ``add files with same name from different folders including signature files`` () =
        let cfg = testConfig "core/samename"

        log "== Compiling F# Code with files with same name in different folders including signature files"
        fsc cfg "%s -o:test.exe" cfg.fsc_flags ["folder1/a.fsi"; "folder1/a.fs"; "folder1/b.fsi"; "folder1/b.fs"; "folder2/a.fsi"; "folder2/a.fs"; "folder2/b.fsi"; "folder2/b.fs"]

        peverify cfg "test.exe"

        exec cfg ("." ++ "test.exe") ""

    [<Test>]
    let ``add files with same name from different folders including signature files that are not synced`` () =
        let cfg = testConfig "core/samename"

        log "== Compiling F# Code with files with same name in different folders including signature files"
        fsc cfg "%s -o:test.exe" cfg.fsc_flags ["folder1/a.fsi"; "folder1/a.fs"; "folder1/b.fs"; "folder2/a.fsi"; "folder2/a.fs"; "folder2/b.fsi"; "folder2/b.fs"]

        peverify cfg "test.exe"

        exec cfg ("." ++ "test.exe") ""

Expected behavior

Tests should be atomic and not depend on each other, or disrupt each other. Any resource that is written or requires an exclusive lock should be unique to that test.

Actual behavior

They fail with typically the following error:

error FS2014: A problem occurred writing the binary 'C:\Users\Abel\AppData\Local\NCrunch\12716\16\tests\fsharp\core\samename\test.exe': Could not open file for writing (binary mode): C:\Users\Abel\AppData\Local\NCrunch\12716\16\tests\fsharp\core\samename\test.exe
Error running command 'C:\Users\Abel\AppData\Local\NCrunch\12716\16\tests\fsharp\..\..\release\net40\bin\fsc.exe' with args '-r:System.Core.dll --nowarn:20 --define:COMPILED -o:test.exe folder1/a.fsi folder1/a.fs folder1/b.fs folder2/a.fsi folder2/a.fs folder2/b.fsi folder2/b.fs' in directory 'C:\Users\Abel\AppData\Local\NCrunch\12716\16\tests\fsharp\core\samename'. ERRORLEVEL 1
   at TestFramework.checkResult(CmdResult result) in D:\Projects\OpenSource\VisualFSharp\tests\fsharp\test-framework.fs:line 258
   at TestFramework.exec@438-22.Invoke(CmdResult result) in D:\Projects\OpenSource\VisualFSharp\tests\fsharp\test-framework.fs:line 438
   at TestFramework.exec@438-23.Invoke(String x) in D:\Projects\OpenSource\VisualFSharp\tests\fsharp\test-framework.fs:line 438
   at TestFramework.Commands.fsc[a,b](a workDir, FSharpFunc`2 exec, String dotNetExe, String fscExe, String flags, FSharpList`1 srcFiles) in D:\Projects\OpenSource\VisualFSharp\tests\fsharp\test-framework.fs:line 77
   at TestFramework.fsc@446-1.Invoke(String flags, FSharpList`1 srcFiles) in D:\Projects\OpenSource\VisualFSharp\tests\fsharp\test-framework.fs:line 446
   at FSharp-Tests-Core.CoreTests.add files with same name from different folders including signature files that are not 

Known workarounds

Run each test in isolation. But this is not really a workaround because unit tests should be parallizable and atomic. Besides, it would take much longer for the tests to complete (i.e., perhaps this is one of the reasons Jenkins is so slow, as tests are run serialized?).

Related information

Happens on current master and probably has been the situation for a while now.

Note that these tests only run in RELEASE build.

I'd be happy to fix this. One way is to have each test create a unique name for its temporary executable output (i.e., the -o commandline param in the example above). That could be achieved by adding a timestamp, or a Guid.New to the output name, or by using nameof [currentTestName] (but ironically, @vasily-kirichenko just let me know there is little interest presently in that functionality, see #2290 (comment)).

Please let me know if there's a preferred method and I will create a PR.

P.S.

Quite a few tests seem to have been written properly. For instance, the sigs tests (though these have a different issue I'll report separately) look like this (i.e., each of them uses a unique -o param)

    [<Test>]
    let ``sigs pos17`` () = 
        let cfg = testConfig "typecheck/sigs"
        fsc cfg "%s --target:exe -o:pos17.exe" cfg.fsc_flags ["pos17.fs"]
        peverify cfg "pos17.exe"
        exec cfg ("." ++ "pos17.exe") ""

    [<Test>]
    let ``sigs pos15`` () = 
        let cfg = testConfig "typecheck/sigs"
        fsc cfg "%s --target:exe -o:pos15.exe" cfg.fsc_flags ["pos15.fs"]
        peverify cfg "pos15.exe"
        exec cfg ("." ++ "pos15.exe") ""

[dsyme]

... just let me know there is little interest presently in that functionality...

This is not true

[abelbraaksma]

@dsyme, thanks. Could you answer this too?

Please let me know if there's a preferred method and I will create a PR.

As I've no problem fixing this, but would like to know if it'll have your blessing or whether you'd prefer to wait for nameof to fix it that way (in fact, in my own project I have a dynamic version of nameof, based on reflection and call hierarchy, useful for logging and which could perhaps be used here, but then, I prefer tests that aren't too complex in their logic).

[dsyme]

Please let me know if there's a preferred method and I will create a PR.

Do it any way you like but don't wait for nameof, thanks

[cartermp]

@abelbraaksma Good find. IMO, an approach like you found in ``sigs pos17`` sounds good.

[enricosada]

A way maybe is to copy the test directory in another directory, who is a temp directory for the test.
this can be done in a single place (inside testConfig )
So then asking for

let cfg = testConfig "typecheck/sigs"

this copy that directory to a temp dir, and so the rest of the test is the same (current dir is in the cfg)

this allow run the test in parallel. But nunit has some restriction about test parallelization (usually per class), so depends