Meson build fails on windows

[o-mdr]

Describe the bug
Meson build fails on Windows platform

To Reproduce
Steps to reproduce the behaviour:

1. On Windows platform with MSVC tools run a developer command line (this comes with MS Visual Studio. NB: NOT vscode, but full VS)
2. Download source code of zstd (main or release 1.4.5)
3. In developer console execute first time setup

cd build\meson
meson setup builddir
---
C:\Users\om\Downloads\zstd-dev\build\meson>meson setup builddir
The Meson build system
Version: 0.55.3
Source dir: C:\Users\om\Downloads\zstd-dev\build\meson
Build dir: C:\Users\om\Downloads\zstd-dev\build\meson\builddir
Build type: native build
Project name: zstd
Project version: DUMMY
C compiler for the host machine: cl (msvc 19.27.29111)
C linker for the host machine: link link 14.27.29111.0
C++ compiler for the host machine: cl (msvc 19.27.29111)
C++ linker for the host machine: link link 14.27.29111.0
Host machine cpu family: x86
Host machine cpu: x86
Program GetZstdLibraryVersion.py found: YES
Message: Project version is now: 1.4.6
Library m found: NO
Run-time dependency threads found: YES
Did not find pkg-config by name 'pkg-config'
Found Pkg-config: NO
Found CMake: C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.EXE (3.17.20032601)
Run-time dependency zlib found: NO (tried pkgconfig and cmake)
Run-time dependency liblzma found: NO (tried pkgconfig and cmake)
Run-time dependency liblz4 found: NO (tried pkgconfig and cmake)
Message: Enable legacy support back to version 0.5
Message: Enable multi-threading support
Windows resource compiler: Microsoft (R) Windows (R) Resource Compiler Version 10.0.10011.16384
Program ../InstallSymlink.py found: YES (C:\Program Files\Meson\meson.exe runpython C:\Users\om\Downloads\zstd-dev\build\meson\programs\../InstallSymlink.py)
Build targets in project: 5

Found ninja-1.10.0 at "C:\Program Files\Meson\ninja.EXE"

4. Compile code

meson compile -C builddir

---
C:\Users\om\Downloads\zstd-dev\build\meson>meson compile -C builddir
Found runner: ['C:\\Program Files\\Meson\\ninja.EXE']
ninja: Entering directory `builddir'
[1/52] Generating Windows resource for file 'lib_.._.._.._build_VS2010_libzstd-dll_libzstd-dll.rc' with a custom command
FAILED: lib/lib_.._.._.._build_VS2010_libzstd-dll_libzstd-dll.rc_libzstd-dll.res
"C:/Program Files (x86)/Windows Kits/10/bin/10.0.18362.0/x86/rc.EXE" "/nologo" "/folib/lib_.._.._.._build_VS2010_libzstd-dll_libzstd-dll.rc_libzstd-dll.res" "../lib/../../../build/VS2010/libzstd-dll/libzstd-dll.rc"
../lib/../../../build/VS2010/libzstd-dll/libzstd-dll.rc(4) : fatal error RC1015: cannot open include file 'zstd.h'.
[10/52] Compiling C object lib/zstd-1.dll.p/.._.._.._lib_common_fse_decompress.c.obj
ninja: build stopped: subcommand failed.

Expected behavior
Compilation should just work

Screenshots and charts

Desktop (please complete the following information):

• OS: Windows x64
• Version build 2004
• Compiler msvc 19.27.29111
• Flags: none, default build
• Other relevant hardware specs: this works on Linux and Ubuntu (because that line is not hit)
• Build system: Meson

Additional context
This seems to be because of the following:
Step 1. Meson build file calls a resource file and compiles it
Step 2. Resource file includes zstd.h on line 4, which fails.

Originally found when working on a patch

[Cyan4973]

cc @lzutao

Possible related issue: #2304

[tesuji]

It's my mistake that our meson config has never worked on Windows.
I'm trying to debug the issue right now. However I don't have Windows machine,
I have to rely on CI so iteration speed might be slow.

[o-mdr]

@lzutao not a problem, it's easy to mess up code for multiplatform use. I have a Windows machine and happy test/give you a hand with this, just tag me here. I think the script includes /lib folder where zstd.h file is located, so I am a bit unsure why the compiler doesn't see it. Maybe it is included at the wrong stage.

While you are on it, could you please add some simple comments to that area of the build script? I think it's related to changing a version file and compiling a dll for Windows, but there is already some code to update a version above, will be nice to know why we do this. Thank you.

[tesuji]

I don't think the code base tested on Visual C/C++ compiler,
I cannot get the code to statically compile on msvc: some are syntax errors.
I haven't tried cmake/VS* solution project.

[tesuji]

I think until the Visual C/C++ compiler properly supported in the codebase,
I will gate meson with gcc-liked compiler.

Edit: Filed #2363

[o-mdr]

@lzutao I've added just one change on my branch, I think it should be enough - the include_directories statement, so

libzstd_sources += [
        windows_mod.compile_resources(
          join_paths(zstd_rootdir, 'build/VS2010/libzstd-dll/libzstd-dll.rc'),
          include_directories: include_directories(join_paths(zstd_rootdir, 'lib'))  <<<<<<< HERE
      )]

I would have expected windows_mod to pass the include directories to the resource compiler, but it doesn't seem to be happening (error output seems to be out of order, but ok to figure out):

C:\git\zstd\build\meson>meson compile -C builddir -v

...
[31/52] "C:/Program Files (x86)/Windows Kits/10/bin/10.0.18362.0/x86/rc.EXE" "/nologo" "/foprograms/programs_.._.._.._build_VS2010_zstd_zstd.rc_zstd.res" "../programs/../../../build/VS2010/zstd/zstd.rc"
FAILED: programs/programs_.._.._.._build_VS2010_zstd_zstd.rc_zstd.res
"C:/Program Files (x86)/Windows Kits/10/bin/10.0.18362.0/x86/rc.EXE" "/nologo" "/foprograms/programs_.._.._.._build_VS2010_zstd_zstd.rc_zstd.res" "../programs/../../../build/VS2010/zstd/zstd.rc"
../programs/../../../build/VS2010/zstd/zstd.rc(4) : fatal error RC1015: cannot open include file 'zstd.h'.
...

I wonder if it is a problem in the windows_mod windows module, that it doesn't actually pass the include_directory to resource compiler. I will check out that module to see how the include dirs are implemented. I've not found any github source code that actually use that module with include dirs.

[tesuji]

There are one more place needed to change like that, see 7cbc04a

Also, zstd codebase doesn't pass compilation on MSVC 2019. I'd recommend using gcc
or clang in the meantime.

[yoniko]

While the suggested diffs in this issue do fix some of the issues, Meson compilation with MSVC toolchain is still fragile and prone to failures for other reasons (I've had multiple failures with the VS 2022 toolchain when I tried).

Furthermore, while we appreciate and glad to see contributions to the Meson build system, it's not a build systems that's officially supported by the project, especially on Windows.

If anyone feels like tackling this issue, please do, I suggest starting with the patches mentioned in the comments, and continuing with setting up a CI.
For now, due to low community engagement on the topic and low priority of the issue, I'm closing it.

[eli-schwartz]

Hmm, this should actually work though? I wasn't aware of this ticket, but I did submit commit 5b2c6c7 as part of #3039 and I have (external) CI that successfully builds zstd on Windows using Meson.

[yoniko]

@eli-schwartz - Makes sense that you missed it, as it's an older issues.
On my system, running VS 2022 toolchain + Meson didn't build. I've encountered three issues:

1. The one specified in this issue which was resolved with the suggested fixes.
2. The default command line tool uses x86 but the linking target is x64. This is okay when working specifically with the x64 toolchain.
3. Issue with flags when building pzstd: cl : Command line error D8021 : invalid numeric argument '/Wno-shadow', at which point I stopped my attempt.

What toolchain are you using? I'll be surprised if it works with MSVC.

I'm glad to hear that you have a CI that tests it, would you consider porting it to be a part of the official repo CI so we can get a quick signal if something breaks?