Skip to content

[Windows] Support Nightly Toolchains for main and release/5.10 #69484

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shahmishal opened this issue Oct 28, 2023 · 83 comments
Open

[Windows] Support Nightly Toolchains for main and release/5.10 #69484

shahmishal opened this issue Oct 28, 2023 · 83 comments
Labels

Comments

@shahmishal
Copy link
Member

shahmishal commented Oct 28, 2023

Windows toolchain needs to be built on ci-external.swift.org for main and release/5.10 branch.

@shahmishal
Copy link
Member Author

Currently blocked on #69483

@shahmishal
Copy link
Member Author

cc: @grynspan @compnerd

@shahmishal
Copy link
Member Author

Currently blocked on getting this PR merged #69555

@shahmishal
Copy link
Member Author

Job created for main branch: https://ci-external.swift.org/job/swift-main-windows-toolchain/20/

@shahmishal
Copy link
Member Author

Job failed to find zlib

CMake Error: The source directory "C:/Users/swift-ci/jenkins/workspace/swift-main-windows-toolchain/zlib" does not exist.

Found the issue, and started new build: https://ci-external.swift.org/job/swift-main-windows-toolchain/21/console

@shahmishal
Copy link
Member Author

@compnerd We are seeing following test failing, it might be config issue. However, if you can look into this would be helpful. Thanks!

https://ci-external.swift.org/job/swift-main-windows-toolchain/21/console

Failed Tests (65):
  Swift(windows-x86_64) :: Driver/android-link.swift
  Swift(windows-x86_64) :: Driver/driver_mode.swift
  Swift(windows-x86_64) :: Driver/help.swift
  Swift(windows-x86_64) :: Driver/options-repl.swift
  Swift(windows-x86_64) :: Driver/print_target_info.swift
  Swift(windows-x86_64) :: Driver/sdk.swift
  Swift(windows-x86_64) :: Driver/working-directory.swift
  Swift(windows-x86_64) :: IDE/complete_macros_expanded.swift
  Swift(windows-x86_64) :: IDE/complete_optionset.swift
  Swift(windows-x86_64) :: Index/index_macros.swift
  Swift(windows-x86_64) :: Interop/CxxToSwiftToCxx/bridge-cxx-struct-back-to-cxx-execution.cpp
  Swift(windows-x86_64) :: Interop/SwiftToCxx/stdlib/swift-stdlib-in-cxx.swift
  Swift(windows-x86_64) :: Macros/accessor_macros.swift
  Swift(windows-x86_64) :: Macros/attached_macros_diags.swift
  Swift(windows-x86_64) :: Macros/composed_macro.swift
  Swift(windows-x86_64) :: Macros/extension_macro_plugin.swift
  Swift(windows-x86_64) :: Macros/if_expr.swift
  Swift(windows-x86_64) :: Macros/imported_conformance_macro.swift
  Swift(windows-x86_64) :: Macros/init_accessor.swift
  Swift(windows-x86_64) :: Macros/lazy_parsing.swift
  Swift(windows-x86_64) :: Macros/macro_and_typealias.swift
  Swift(windows-x86_64) :: Macros/macro_expand.swift
  Swift(windows-x86_64) :: Macros/macro_expand_attributes.swift
  Swift(windows-x86_64) :: Macros/macro_expand_closure.swift
  Swift(windows-x86_64) :: Macros/macro_expand_codeitems.swift
  Swift(windows-x86_64) :: Macros/macro_expand_conformances.swift
  Swift(windows-x86_64) :: Macros/macro_expand_conformances_multi.swift
  Swift(windows-x86_64) :: Macros/macro_expand_extensions.swift
  Swift(windows-x86_64) :: Macros/macro_expand_member_with_conformances.swift
  Swift(windows-x86_64) :: Macros/macro_expand_other.swift
  Swift(windows-x86_64) :: Macros/macro_expand_peers.swift
  Swift(windows-x86_64) :: Macros/macro_expand_primary.swift
  Swift(windows-x86_64) :: Macros/macro_expand_synthesized_members.swift
  Swift(windows-x86_64) :: Macros/macro_expand_throwing.swift
  Swift(windows-x86_64) :: Macros/macro_expand_variadic.swift
  Swift(windows-x86_64) :: Macros/macro_fixits.swift
  Swift(windows-x86_64) :: Macros/macro_invalid.swift
  Swift(windows-x86_64) :: Macros/macro_keywordname.swift
  Swift(windows-x86_64) :: Macros/macro_mixedpluginpath.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_basic.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_error.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_searchorder.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_server.swift
  Swift(windows-x86_64) :: Macros/macro_shadowing.swift
  Swift(windows-x86_64) :: Macros/macro_swiftdeps.swift
  Swift(windows-x86_64) :: Macros/macro_vfs.swift
  Swift(windows-x86_64) :: Macros/magic_literal_expansion.swift
  Swift(windows-x86_64) :: Macros/top_level_freestanding.swift
  Swift(windows-x86_64) :: ModuleInterface/Observable.swift
  Swift(windows-x86_64) :: ModuleInterface/macros.swift
  Swift(windows-x86_64) :: Profiler/coverage_macros.swift
  Swift(windows-x86_64) :: SIL/Serialization/deserialize_stdlib.sil
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map-clang-implicit.swift
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map-forwarding-module.swift
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map-typecheck-module-from-interface.swift
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map.swift
  Swift(windows-x86_64) :: Serialization/macros.swift
  Swift(windows-x86_64) :: SourceKit/Macros/diags.swift
  Swift(windows-x86_64) :: SourceKit/Macros/macro_across_modules.swift
  Swift(windows-x86_64) :: SourceKit/Macros/macro_formatting.swift
  Swift(windows-x86_64) :: SourceKit/Macros/macro_semantic_token.swift
  Swift(windows-x86_64) :: SourceKit/Macros/syntactic_expansion.swift
  Swift(windows-x86_64) :: SymbolGraph/Symbols/Macro.swift
  Swift(windows-x86_64) :: SymbolGraph/Symbols/Mixins/DeclarationFragments/Full/Macros.swift
  Swift(windows-x86_64) :: stdlib/llvm-support-odr-violation.test-sh


Testing Time: 759.15s
  Skipped          :    1
  Unsupported      : 2459
  Passed           : 7591
  Expectedly Failed:   47
  Failed           :   65

@compnerd
Copy link
Member

compnerd commented Nov 2, 2023

@shahmishal super exciting!

Just looking through a couple of things stand out:

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\vsdevcmd.bat" -arch=amd64 -host_arch=amd64
'C:\Program' is not recognized as an internal or external command,
operable program or batch file.

With the switch to build.ps1, this shouldn't be needed (build.ps1 will properly setup the VS Dev environment to support building the SDK for all the architectures). Something that might help simplify the Jenkins configuration.

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>d /s /q C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\build
'd' is not recognized as an internal or external command,
operable program or batch file.

Odd, I'm not sure where this is coming from. We should investigate this later.

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>powershell.exe -ExecutionPolicy RemoteSigned -File C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\utils\build.ps1 -SourceCache C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain -BinaryCache T: -ImageRoot T: -BuildType Release -Test swift,dispatch,foundation,xctest, -Stage T:\artifacts

This looks right ... and if this is failing, I doubt it's a configuration issue and more likely this is an issue in build.ps1. I'll need a little bit to dig into this a bit further to try to understand what is going on.

@compnerd
Copy link
Member

compnerd commented Nov 2, 2023

There seem to be a few failures due to %swift_driver_plain being expanded to main. This would mean that config.swift = inferSwiftBinary('swift') returned main? I don't see that in the log, so I expect that it found it in the path.

@shahmishal
Copy link
Member Author

If it helps, here is the build step:

            "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\Tools\\vsdevcmd.bat" -arch=amd64 -host_arch=amd64"

            rd /s /q %LocalAppData%\\clang\\ModuleCache


            set CMAKE_BUILD_TYPE=Release
            set PYTHON_HOME="%ProgramFiles(x86)%\\Microsoft Visual Studio\\Shared\\Python37_64"

            set SourceRoot=%WORKSPACE%
            set BuildRoot=%SourceRoot%\\build

            md %BuildRoot%
            subst T: /d
            subst T: %BuildRoot% || (exit /b)

            set BuildRoot=T:

            set PackageRoot=%BuildRoot%\\artifacts

            md %PackageRoot%

            :: Setup temporary directories
            md %BuildRoot%\\tmp
            set TEMP=%BuildRoot%\\tmp
            set TMP=%BuildRoot%\\tmp
            set TMPDIR=%BuildRoot%\\tmp

            set NINJA_STATUS=[%%f/%%t][%%p][%%es]

            set TestArg=-Test swift,dispatch,foundation,xctest,

            set "SkipPackagingArg= "

            rd /s /q %WORKSPACE%\\build

            powershell.exe -ExecutionPolicy RemoteSigned -File %WORKSPACE%\\swift\\utils\\build.ps1 ^
            -SourceCache %SourceRoot% ^
            -BinaryCache %BuildRoot% ^
            -ImageRoot %BuildRoot% ^
            -BuildType %CMAKE_BUILD_TYPE% ^
            %SkipPackagingArg% ^
            %TestArg% ^
            -Stage %PackageRoot%

@shahmishal
Copy link
Member Author

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>d /s /q C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\build
'd' is not recognized as an internal or external command,
operable program or batch file.

Odd, I'm not sure where this is coming from. We should investigate this later.

I have a fix for this.

@compnerd compnerd added the Windows Platform: Windows label Nov 2, 2023
@compnerd
Copy link
Member

compnerd commented Nov 2, 2023

It appears that a number of the failures are referencing ../lib/swift/windows and not being able to find sources under there. I think that this might be due to the split of the build tree and the source tree.

I wonder if there was something that we had done before that results in the build tree being linked or something?

@compnerd
Copy link
Member

compnerd commented Nov 2, 2023

Hmm, I wonder if it is the PYTHON_HOME that is causing the issue ... we use python 3.9, but PYTHON_HOME is set to 3.7. Would it be possible to test with PYTHON_HOME unset?

@compnerd
Copy link
Member

compnerd commented Nov 2, 2023

Bad Run:

lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:210: note: Using cmake: C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/bin/cmake.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-frontend: t:\1\bin\swift-frontend.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swiftc: t:\1\bin\swiftc.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-opt: t:\1\bin\sil-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-func-extractor: t:\1\bin\sil-func-extractor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-llvm-gen: t:\1\bin\sil-llvm-gen.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-nm: t:\1\bin\sil-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-passpipeline-dumper: t:\1\bin\sil-passpipeline-dumper.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using lldb-moduleimport-test: t:\1\bin\lldb-moduleimport-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-ide-test: t:\1\bin\swift-ide-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-dependency-tool: t:\1\bin\swift-dependency-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-reflection-dump: t:\1\bin\swift-reflection-dump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-remoteast-test: t:\1\bin\swift-remoteast-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-indent: t:\1\bin\swift-indent.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-symbolgraph-extract: t:\1\bin\swift-symbolgraph-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-api-extract: t:\1\bin\swift-api-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using clang: t:\1\bin\clang.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using clang++: t:\1\bin\clang++.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-link: t:\1\bin\llvm-link.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-llvm-opt: t:\1\bin\swift-llvm-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-profdata: t:\1\bin\llvm-profdata.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-cov: t:\1\bin\llvm-cov.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-strings: t:\1\bin\llvm-strings.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using FileCheck: t:\1\bin\filecheck.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-dwarfdump: t:\1\bin\llvm-dwarfdump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-readelf: t:\1\bin\llvm-readelf.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-dis: t:\1\bin\llvm-dis.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-nm: t:\1\bin\llvm-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-readtapi: t:\1\bin\llvm-readtapi.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-size: t:\1\bin\llvm-size.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sourcekitd-test: t:\1\bin\sourcekitd-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using complete-test: t:\1\bin\complete-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-api-digester: t:\1\bin\swift-api-digester.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-cache-tool: t:\1\bin\swift-cache-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-refactor: t:\1\bin\swift-refactor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-demangle-yamldump: t:\1\bin\swift-demangle-yamldump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-demangle: t:\1\bin\swift-demangle.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_O' program, try setting BENCHMARK_O in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_Driver' program, try setting BENCHMARK_DRIVER in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using wasm-ld: t:\1\bin\wasm-ld.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-plugin-server: t:\1\bin\swift-plugin-server.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-parse-test: t:\1\bin\swift-parse-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:385: note: Using resource dir: ..\lib\swift
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:432: note: Compiling with -swift-version 4
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:528: note: Using Clang module cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\clang-module-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:529: note: Using test results dir: T:/1/./swift-test-results/x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:534: note: Using code completion cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\completion-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:1496: note: Testing Windows x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:2253: note: Using platform module dir: ..\lib\swift\windows
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-reflection-test-windows-x86_64: t:\1\bin\swift-reflection-test-windows-x86_64.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:2861: note: Available features: CMAKE_GENERATOR=Ninja, CODEGENERATOR=AArch64, CODEGENERATOR=ARM, CODEGENERATOR=WebAssembly, CODEGENERATOR=X86, CPU=x86_64, MSVC_VER=17.0, OS=windows-msvc, PTRSIZE=64, SWIFT_VERSION=4, VENDOR=unknown, asserts, concurrency, concurrency_runtime, cplusplus_driver, differentiable_programming, distributed, executable_test, no_asan, no_lto, nonexecutable_test, objc_codegen, observation, optimized_stdlib, optimized_stdlib_x86_64, reflection, sourcekit, string_processing, swift_only_stable_abi, swift_stdlib_no_asserts, swift_swift_parser, swift_test_mode_optimize_none, swift_test_mode_optimize_none_x86_64, swift_tools_extra, thread_safe_runtime, threading_win32

Good Run:

lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:210: note: Using cmake: C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/bin/cmake.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-frontend: t:\1\bin\swift-frontend.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift: t:\1\bin\swift.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swiftc: t:\1\bin\swiftc.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-opt: t:\1\bin\sil-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-func-extractor: t:\1\bin\sil-func-extractor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-llvm-gen: t:\1\bin\sil-llvm-gen.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-nm: t:\1\bin\sil-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-passpipeline-dumper: t:\1\bin\sil-passpipeline-dumper.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using lldb-moduleimport-test: t:\1\bin\lldb-moduleimport-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-ide-test: t:\1\bin\swift-ide-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-dependency-tool: t:\1\bin\swift-dependency-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-reflection-dump: t:\1\bin\swift-reflection-dump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-remoteast-test: t:\1\bin\swift-remoteast-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-indent: t:\1\bin\swift-indent.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-symbolgraph-extract: t:\1\bin\swift-symbolgraph-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-api-extract: t:\1\bin\swift-api-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using clang: t:\1\bin\clang.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using clang++: t:\1\bin\clang++.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-link: t:\1\bin\llvm-link.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-llvm-opt: t:\1\bin\swift-llvm-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-profdata: t:\1\bin\llvm-profdata.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-cov: t:\1\bin\llvm-cov.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-strings: t:\1\bin\llvm-strings.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using FileCheck: t:\1\bin\filecheck.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-dwarfdump: t:\1\bin\llvm-dwarfdump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-readelf: t:\1\bin\llvm-readelf.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-dis: t:\1\bin\llvm-dis.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-nm: t:\1\bin\llvm-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-readtapi: t:\1\bin\llvm-readtapi.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-size: t:\1\bin\llvm-size.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sourcekitd-test: t:\1\bin\sourcekitd-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using complete-test: t:\1\bin\complete-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-api-digester: t:\1\bin\swift-api-digester.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-cache-tool: t:\1\bin\swift-cache-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-refactor: t:\1\bin\swift-refactor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-demangle-yamldump: t:\1\bin\swift-demangle-yamldump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-demangle: t:\1\bin\swift-demangle.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_O' program, try setting BENCHMARK_O in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_Driver' program, try setting BENCHMARK_DRIVER in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using wasm-ld: t:\1\bin\wasm-ld.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-plugin-server: t:\1\bin\swift-plugin-server.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-parse-test: t:\1\bin\swift-parse-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:385: note: Using resource dir: t:\1\lib\swift
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:432: note: Compiling with -swift-version 4
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:528: note: Using Clang module cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\clang-module-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:529: note: Using test results dir: T:/1/./swift-test-results/x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:534: note: Using code completion cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\completion-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:1496: note: Testing Windows x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:2253: note: Using platform module dir: t:\1\lib\swift\windows
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-reflection-test-windows-x86_64: t:\1\bin\swift-reflection-test-windows-x86_64.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:2861: note: Available features: CMAKE_GENERATOR=Ninja, CODEGENERATOR=AArch64, CODEGENERATOR=ARM, CODEGENERATOR=WebAssembly, CODEGENERATOR=X86, CPU=x86_64, MSVC_VER=17.0, OS=windows-msvc, PTRSIZE=64, SWIFT_VERSION=4, VENDOR=unknown, asserts, concurrency, concurrency_runtime, cplusplus_driver, differentiable_programming, distributed, executable_test, no_asan, no_lto, nonexecutable_test, objc_codegen, observation, optimized_stdlib, optimized_stdlib_x86_64, reflection, sourcekit, string_processing, swift_only_stable_abi, swift_stdlib_no_asserts, swift_swift_parser, swift_test_mode_optimize_none, swift_test_mode_optimize_none_x86_64, swift_tools_extra, thread_safe_runtime, threading_win32

This seems to be the diff:

- lit.py: swift\test\lit.cfg:289: note: using swift: t:\1\bin\swift.exe
- lit.py: swift\test\lit.cfg:385: note: Using resource dir: t:\1\lib\swift
+ lit.py: swift\test\lit.cfg:385: note: Using resource dir: ..\lib\swift
- lit.py: swift\test\lit.cfg:2253: note: Using platform module dir: t:\1\lib\swift\windows
+ lit.py: swift\test\lit.cfg:2253: note: Using platform module dir: ..\lib\swift\windows

This does match the majority of the failures. Now the question is, why is the resource dir and platform module dir path different?

@compnerd
Copy link
Member

compnerd commented Nov 2, 2023

The missing

note: using swift: t:\1\bin\swift.exe

along with https://github.com/apple/swift/blob/main/test/lit.cfg#L273-L298

seems to indicate that swift was picked up from Path. Is the image identical to the PR builder? The default value is ..\lib\swift.

@shahmishal
Copy link
Member Author

@compnerd Let me know if I missed setting an env

@compnerd
Copy link
Member

compnerd commented Nov 3, 2023

@shahmishal I don't see off hand any missing environment variables. The only ones that should be concerning are the ones that determine the flags (which repositories to test, whether to package or not, build type), but those seem to be setup.

I would say that removing the PYTHON_HOME setting might be good. One of the nice things about build.ps1 is that we shouldn't need to set up a whole slew of environment variables, and if there is something missing we should try to detect it. If we could remove running the VsDevCmd, it would also simplify the run.

Just to double check my assumption - the image is identical to the PR testing one right?

@compnerd
Copy link
Member

compnerd commented Nov 3, 2023

Hmm ... @tristanlabelle pointed out that I misread the code: swift/test/lit.cfg at main · apple/swift

It seems to imply that there is a SWIFT environment variable being set?

@shahmishal
Copy link
Member Author

@compnerd Can you try running this locally?

 rd /s /q %LocalAppData%\\clang\\ModuleCache

 set CMAKE_BUILD_TYPE=Release
 set PYTHON_HOME="%ProgramFiles(x86)%\\Microsoft Visual Studio\\Shared\\Python37_64"

 set SourceRoot=%WORKSPACE%
 set BuildRoot=%SourceRoot%\\build

 md %BuildRoot%
 subst T: /d
 subst T: %BuildRoot% || (exit /b)

 set BuildRoot=T:

 set PackageRoot=%BuildRoot%\\artifacts

 md %PackageRoot%

 :: Setup temporary directories
 md %BuildRoot%\\tmp
 set TEMP=%BuildRoot%\\tmp
 set TMP=%BuildRoot%\\tmp
 set TMPDIR=%BuildRoot%\\tmp

 set NINJA_STATUS=[%%f/%%t][%%p][%%es]

 set TestArg=-Test swift,dispatch,foundation,xctest,

 set "SkipPackagingArg= "

 rd /s /q %WORKSPACE%\\build

 powershell.exe -ExecutionPolicy RemoteSigned -File %WORKSPACE%\\swift\\utils\\build.ps1 ^
 -SourceCache %SourceRoot% ^
 -BinaryCache %BuildRoot% ^
 -ImageRoot %BuildRoot% ^
 -BuildType %CMAKE_BUILD_TYPE% ^
 %SkipPackagingArg% ^
 %TestArg% ^
 -Stage %PackageRoot%

@compnerd
Copy link
Member

compnerd commented Nov 3, 2023

@shahmishal everything now makes sense - there is an environment variable swift that is being set by Jenkins at Parameters [Jenkins] that is set to main. That is then passed along to the test suite which then uses main as the swift command instead which results in the failures.

@shahmishal
Copy link
Member Author

this is used to pick up the correct SHA, what is the best way to unset variable?

@compnerd
Copy link
Member

compnerd commented Nov 3, 2023

Depends on the shell. With cmd it would be set swift= (or whatever the variable name is). For PowerShell it would be Delete-Item env:\swift -ErrorAction Ignore.

@shahmishal
Copy link
Member Author

Going to try set "swift= "

@shahmishal
Copy link
Member Author

@compnerd
Copy link
Member

compnerd commented Nov 3, 2023

@shahmishal if there is a trailing space, I think it might fail as it will set it to the string containing a space, there should be nothing after the =.

@compnerd
Copy link
Member

compnerd commented Nov 3, 2023

C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>set "swift= "

The trailing space is setting the string to " " rather than deleting the environment variable.

@shahmishal
Copy link
Member Author

@compnerd
Copy link
Member

compnerd commented Nov 3, 2023

Awesome! The 5.10 cherry-pick is at #69608

@shahmishal
Copy link
Member Author

release/5.10 job created: https://ci-external.swift.org/job/swift-5.10-windows-toolchain/

@shahmishal
Copy link
Member Author

I thought this would delete the virtual drive:

            subst T: /d

@compnerd
Copy link
Member

Yes, that removes the "virtual drive", but does not remove the contents that were mapped there.

@shahmishal
Copy link
Member Author

I also have rd /s /q %WORKSPACE%\\build in the job, so it should download new toolchain every time.

            set BuildRoot=%SourceRoot%\\build

            md %BuildRoot%
            subst T: /d
            subst T: %BuildRoot% || (exit /b)

@compnerd
Copy link
Member

Interesting - and the download doesn't say that the toolchain installer is invalid. So, it could be that it is failing to pass the signature check :/ I don't think that I have a way to diagnose this without actually interacting with the host.

@shahmishal
Copy link
Member Author

Can we add verbose mode to the signature check?

@shahmishal
Copy link
Member Author

shahmishal commented Nov 14, 2023

msiexec_failure.log
Attached log from

msiexec.exe /lv T:\toolchains\log.log /qn /a T:\toolchains\a0 TARGETDIR=T:\toolchains\swift-5.9-RELEASE-windows10
MSI (s) (E8:5C) [01:08:01:773]: Found shell folder  by spelunking through the registry.
MSI (s) (E8:5C) [01:08:01:773]: Note: 1: 2103 2: 26 
MSI (s) (E8:5C) [01:08:01:774]: Error: Failed to Create AppData Cache Path with Product ID
MSI (s) (E8:5C) [01:08:01:774]: Note: 1: 2205 2:  3: Error 
MSI (s) (E8:5C) [01:08:01:774]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 2103 
DEBUG: Error 2103:  Could not resolve path for shell folder 26.
MSI (s) (E8:5C) [01:08:01:778]: Note: 1: 2205 2:  3: Error 
MSI (s) (E8:5C) [01:08:01:778]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709 
MSI (s) (E8:5C) [01:08:01:778]: Product: Swift Runtime for Windows x86_64 -- The installer has encountered an unexpected error installing this package. This may indicate a problem with this package. The error code is 2103. The arguments are: 26, , 
�

@shahmishal
Copy link
Member Author

bat '''
            "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\Tools\\vsdevcmd.bat -arch=amd64 -host_arch=amd64"

            rd /s /q %LocalAppData%\\clang\\ModuleCache


            set CMAKE_BUILD_TYPE=Release
            set PYTHON_HOME="%ProgramFiles(x86)%\\Microsoft Visual Studio\\Shared\\Python37_64"

            set SourceRoot=%WORKSPACE%
            set BuildRoot=%SourceRoot%\\build

            md %BuildRoot%
            subst T: /d
            subst T: %BuildRoot% || (exit /b 1)

            set BuildRoot=T:

            set PackageRoot=%BuildRoot%\\artifacts

            md %PackageRoot%

            :: Setup temporary directories
            md %BuildRoot%\\tmp
            set TEMP=%BuildRoot%\\tmp
            set TMP=%BuildRoot%\\tmp
            set TMPDIR=%BuildRoot%\\tmp

            set NINJA_STATUS=[%%f/%%t][%%p][%%es]

            set TestArg=-Test swift,dispatch,foundation,xctest

            set "SkipPackagingArg= "
            set "swift="

            rd /s /q %WORKSPACE%\\build

            powershell.exe -ExecutionPolicy RemoteSigned -File %WORKSPACE%\\swift\\utils\\build.ps1 ^
            -SourceCache %SourceRoot% ^
            -BinaryCache %BuildRoot% ^
            -ImageRoot %BuildRoot% ^
            -BuildType %CMAKE_BUILD_TYPE% ^
            %SkipPackagingArg% ^
            %TestArg% ^
            -Stage %PackageRoot% || (exit /b 1)
            '''

Thats the current script, let me know if I missed something
Re-running it https://ci-external.swift.org/job/swift-main-windows-toolchain/31/console

@compnerd
Copy link
Member

I was thinking of adding the /lvx! ... to the arguments, seems that you did the same thing! This is interesting - the error is definitely Error 2103: Could not resolve path for shell folder 26.. Now, the question is why is that happening 🤔

@compnerd
Copy link
Member

I wonder if the builders are running as different users. LOCAL SYSTEM (service account) might encounter some issues - I wonder if we can quickly test with:

msiexec.exe /lvx! T:\toolchains\log.log /qn /a T:\toolchains\a0 TARGETDIR=T:\toolchains\swift-5.9-RELEASE-windows10 ALLUSERS=1

@compnerd
Copy link
Member

#69852 adds the logging for future use as well.

@compnerd
Copy link
Member

@shahmishal seems that the last re-run succeeded

@shahmishal
Copy link
Member Author

@compnerd sent link in private for review for nightly main toolchain.

@compnerd
Copy link
Member

@shahmishal - reviewed, seems ready to go to me!

@shahmishal
Copy link
Member Author

main branch nightly Windows toolchain available on swift.org

@compnerd
Copy link
Member

@shahmishal - it seems that the main branch nightlies do not auto-update? There was a snapshot from 11/27, but the other platforms are on 12/7.

@compnerd
Copy link
Member

compnerd commented Jan 4, 2024

@shahmishal any progress on the auto-updated nightlies? It seems that the Windows target is out of date by a month.

@compnerd
Copy link
Member

ping again on this @shahmishal - the Windows snapshot on main is now ~2m out of date and we still do not have a 5.10 release snapshot. This really risks the 5.10 release as we will need extra time to validate the Windows releases.

@shahmishal
Copy link
Member Author

@compnerd I am looking into getting this configured, due to other tasks I was not able to get to this. Hopefully, I will be able to focus on this soon.

@shahmishal
Copy link
Member Author

@shahmishal
Copy link
Member Author

main failed:

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>powershell.exe -ExecutionPolicy RemoteSigned -File C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\utils\build.ps1             -SourceCache C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain             -BinaryCache T:             -ImageRoot T:             -BuildType Release                           -Test swift,dispatch,foundation,xctest             -Stage T:\artifacts   || (exit /b 1 ) 
C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\utils\build.ps1 : A parameter cannot be found 
that matches parameter name 'BuildType'.
    + CategoryInfo          : InvalidArgument: (:) [build.ps1], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : NamedParameterNotFound,build.ps1
 

cc: @compnerd

@compnerd
Copy link
Member

Ah, right, we dropped the -BuildType parameter a while ago to allow us to support sccache. You should just be able to remove the -BuildType Release from the invocation (it will always generate release mode builds).

@shahmishal
Copy link
Member Author

@shahmishal
Copy link
Member Author

Ah, right, we dropped the -BuildType parameter a while ago to allow us to support sccache. You should just be able to remove the -BuildType Release from the invocation (it will always generate release mode builds).

Can we also cherry-pick this to release/5.10?

@compnerd
Copy link
Member

Sure, let me look into that!

@shahmishal
Copy link
Member Author

Nightly 5.10 released on swift.org - swiftlang/swift-org-website#488

@compnerd
Copy link
Member

Trying to pull in the changes for -BuildType into 5.10 at #71117

@finagolfin
Copy link
Member

Swift.org now has builds from all the release branches, so this can be closed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants