Virtualize swift compiler outputs #63206

cachemeifyoucan · 2023-01-25T00:53:46Z

Step 1 of making swift compiler output cacheable. Virtualize swift compiler outputs (not everything yet, but most of the important output to show feasibility) and add a mechanics to verify the output determinism through the newly added function.

Need some changes in LLVM swiftlang/llvm-project#6112

Feedbacks about how to structurally pass through VirtualBackends in swift compiler is welcomed.

cachemeifyoucan · 2023-01-25T00:56:22Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci Please test

cachemeifyoucan · 2023-01-25T03:50:53Z

Ah, I break lldb of course. let me try smoke test first.

@swift-ci please smoke test

cachemeifyoucan · 2023-01-25T03:55:37Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

artemcm

Looks good to me.
Cool to have the frontend output determinism tests in-suite.

cachemeifyoucan · 2023-01-25T21:07:09Z

For deterministic check to be really useful, need to add an option to bypass the laziness when modhash matches.

Also thinking I should just put the output backend into ASTContext which will make the code a lot cleaner.

cachemeifyoucan · 2023-01-26T02:03:55Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

cachemeifyoucan · 2023-01-26T19:16:46Z

Update: fix the tests and remove withOutputFile helper function to encourage use of output backend.

Now properly capture swiftdeps output but it is NOT deterministic.

cachemeifyoucan · 2023-01-26T19:16:58Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

cachemeifyoucan · 2023-01-26T21:02:46Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

cachemeifyoucan · 2023-01-26T21:20:09Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

cachemeifyoucan · 2023-01-27T00:58:31Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

include/swift/AST/ASTContext.h

include/swift/AST/DiagnosticsFrontend.def

include/swift/AST/FileSystem.h

lib/Frontend/Frontend.cpp

lib/Frontend/FrontendOptions.cpp

lib/IRGen/IRGenModule.cpp

lib/FrontendTool/FrontendTool.cpp

cachemeifyoucan · 2023-04-04T04:34:00Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

cachemeifyoucan · 2023-04-04T06:26:53Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

cachemeifyoucan · 2023-04-04T07:36:38Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

benlangmuir

There were a couple of my minor comments from January that seemed to have been lost, but otherwise LGTM.

benlangmuir · 2023-04-04T16:39:48Z

include/swift/Option/FrontendOptions.td

@@ -1135,6 +1135,13 @@ def enable_emit_generic_class_ro_t_list :
  HelpText<"Enable emission of a section with references to class_ro_t of "
           "generic class patterns">;

+def enable_swift_deterministic_check :
+  Flag<["-"], "enable-swift-deterministic-check">,


This comment wasn't addressed

benlangmuir · 2023-04-04T16:39:51Z

include/swift/Option/FrontendOptions.td

@@ -1135,6 +1135,13 @@ def enable_emit_generic_class_ro_t_list :
  HelpText<"Enable emission of a section with references to class_ro_t of "
           "generic class patterns">;

+def enable_swift_deterministic_check :
+  Flag<["-"], "enable-swift-deterministic-check">,
+  HelpText<"Check swift compiler output determinisim by run it twice">;


This comment wasn't addressed

cachemeifyoucan · 2023-04-04T23:03:15Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

cachemeifyoucan · 2023-04-05T00:47:12Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

Using a virutal output backend to capture all the outputs from swift-frontend invocation. This allows future virtualization of swift compiler outputs.

Add option to test swift compiler determinism by running the job twice and compare the output produced.

Add an option to skip module hash checking that checks if swift compiler can skip object file generation because it is going to produce the same result. Always generation object file output when flag is used so we can check for output determinism.

Batch mode depscan also gets diagnostics for output errors.

Support makefile style dependency file and imported modules. Need improvements to virtual backend to allow emit loaded module trace file.

Add unit-tests for already virtualized outputs

cachemeifyoucan · 2023-04-05T07:05:09Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

Fix broken test cases on non-macos platforms.

cachemeifyoucan · 2023-04-05T10:54:51Z

Please test with following pull request:
swiftlang/llvm-project#6112

@swift-ci please smoke test

Using a virutal output backend to capture all the outputs from swift-frontend invocation. This allows redirecting and/or mirroring compiler outputs to multiple location using different OutputBackend. As an example usage for the virtual outputs, teach swift compiler to check its output determinism by running the compiler invocation twice and compare the hash of all its outputs. Virtual output will be used to enable caching in the future.

cachemeifyoucan requested review from akyrtzi, benlangmuir and jansvoboda11 January 25, 2023 00:53

cachemeifyoucan requested review from hborla, slavapestov, xedin and artemcm as code owners January 25, 2023 00:53

xedin removed their request for review January 25, 2023 18:52

artemcm previously approved these changes Jan 25, 2023

View reviewed changes

cachemeifyoucan force-pushed the eng/PR-swift-virtualize-output branch from d940dbe to fef81b9 Compare January 26, 2023 02:03

cachemeifyoucan mentioned this pull request Jan 26, 2023

VirtualOutputBackend Improvements for swift compiler swiftlang/llvm-project#6112

Merged

cachemeifyoucan force-pushed the eng/PR-swift-virtualize-output branch from fef81b9 to d3edb11 Compare January 26, 2023 19:15

cachemeifyoucan force-pushed the eng/PR-swift-virtualize-output branch from d3edb11 to 4a4fda1 Compare January 26, 2023 21:02

cachemeifyoucan force-pushed the eng/PR-swift-virtualize-output branch from 08920e3 to e73b6f3 Compare January 27, 2023 00:58

benlangmuir reviewed Jan 27, 2023

View reviewed changes

cachemeifyoucan force-pushed the eng/PR-swift-virtualize-output branch 2 times, most recently from c6cffe4 to c148737 Compare February 1, 2023 21:48

cachemeifyoucan requested review from shahmishal, zoecarver and hyp as code owners February 1, 2023 21:48

cachemeifyoucan force-pushed the eng/PR-swift-virtualize-output branch from 62b9ce1 to 6674e1b Compare April 4, 2023 06:26

cachemeifyoucan force-pushed the eng/PR-swift-virtualize-output branch from 6674e1b to 11a25b8 Compare April 4, 2023 07:36

artemcm approved these changes Apr 4, 2023

View reviewed changes

benlangmuir reviewed Apr 4, 2023

View reviewed changes

benlangmuir approved these changes Apr 4, 2023

View reviewed changes

cachemeifyoucan force-pushed the eng/PR-swift-virtualize-output branch from 11a25b8 to b15877b Compare April 4, 2023 23:01

cachemeifyoucan force-pushed the eng/PR-swift-virtualize-output branch from b15877b to bfa53ff Compare April 5, 2023 00:46

cachemeifyoucan added 10 commits April 5, 2023 00:04

Virtualize swift-frontend outputs

fb2532c

Using a virutal output backend to capture all the outputs from swift-frontend invocation. This allows future virtualization of swift compiler outputs.

Add an option to verify the deterministic output of swift compiler

f463a3c

Add option to test swift compiler determinism by running the job twice and compare the output produced.

Support deterministic checking in explicit module build

9288398

Virtualize depscan output to allow deterministic check

a98f7d9

Batch mode depscan also gets diagnostics for output errors.

Capture clang importer output in virtual output

60b06d5

Handle dependencies output

af6c515

Support makefile style dependency file and imported modules. Need improvements to virtual backend to allow emit loaded module trace file.

Virtualize swift abi description file

2aaeb37

Add test cases for virtualized output types

f70d2bb

Add unit-tests for already virtualized outputs

Address review feedback

9be198f

cachemeifyoucan force-pushed the eng/PR-swift-virtualize-output branch from bfa53ff to 9be198f Compare April 5, 2023 07:04

Fix abi-digester for non-macos platforms

ab9975f

Fix broken test cases on non-macos platforms.

cachemeifyoucan merged commit 09b8af8 into swiftlang:main Apr 5, 2023

cachemeifyoucan deleted the eng/PR-swift-virtualize-output branch April 5, 2023 15:34

kavon mentioned this pull request Apr 7, 2023

🍒 "Virtualize swift compiler outputs" #65015

Closed

Virtualize swift compiler outputs #63206

Virtualize swift compiler outputs #63206

Uh oh!

Conversation

cachemeifyoucan commented Jan 25, 2023

Uh oh!

cachemeifyoucan commented Jan 25, 2023

Uh oh!

cachemeifyoucan commented Jan 25, 2023

Uh oh!

cachemeifyoucan commented Jan 25, 2023

Uh oh!

artemcm left a comment

Choose a reason for hiding this comment

Uh oh!

cachemeifyoucan commented Jan 25, 2023

Uh oh!

cachemeifyoucan commented Jan 26, 2023

Uh oh!

cachemeifyoucan commented Jan 26, 2023

Uh oh!

cachemeifyoucan commented Jan 26, 2023

Uh oh!

cachemeifyoucan commented Jan 26, 2023

Uh oh!

cachemeifyoucan commented Jan 26, 2023

Uh oh!

cachemeifyoucan commented Jan 27, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cachemeifyoucan commented Apr 4, 2023

Uh oh!

cachemeifyoucan commented Apr 4, 2023

Uh oh!

cachemeifyoucan commented Apr 4, 2023

Uh oh!

benlangmuir left a comment

Choose a reason for hiding this comment

Uh oh!

benlangmuir Apr 4, 2023

Choose a reason for hiding this comment

Uh oh!

benlangmuir Apr 4, 2023

Choose a reason for hiding this comment

Uh oh!

cachemeifyoucan commented Apr 4, 2023

Uh oh!

cachemeifyoucan commented Apr 5, 2023

Uh oh!

cachemeifyoucan commented Apr 5, 2023

Uh oh!

cachemeifyoucan commented Apr 5, 2023

Uh oh!

Uh oh!