-
Notifications
You must be signed in to change notification settings - Fork 15.7k
src build on windows not working #12947
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
It looks like the protobuf version you're using is pretty old and is no longer supported. Can you upgrade to the newest version (23.2) and see if you still see the issue? |
I ran into the same issue trying to upgrade to 23.x and here's my diagnosis: MSVC cl.exe has a limitation (https://developercommunity.visualstudio.com/t/clexe-compiler-driver-cannot-handle-long-file-path/975889) on filename lengths of 260 characters (MAX_PATH). There appears to be no workaround using UNC paths. Within Protobuf there are a number of source files organized like this: with rules like (abbreviated):
When bazel encounters rules like this it constructs include paths to provide the
Counting the hash, using That leaves a maximum of 40 characters for the other parts. It's pretty easy to run out of that. The default output base for my username is >30 characters. One option to fix this is to reorganize the src/google/protobuf/compiler files to remove the nesting. One workaround which I am using now is to set a bazel startup option to gain that 30 characters back: See related bazel issues: bazelbuild/bazel#15961 |
protocolbuffers/protobuf#12947 PiperOrigin-RevId: 540401592 Change-Id: I8998cf6927ad3a59ce296fd6a815eb63d104d1d1
This saves 10 characters in build paths. protocolbuffers/protobuf#12947 PiperOrigin-RevId: 540416043 Change-Id: I5c4abeb751e7ee476d9f183f381e4f8dbe0a0a93
I tried to rename my dependency from
So it turns out that it's a bit challenging to save characters simply by renaming the repository. |
Upgrade protobuf upb to corresponding v23.x branch Upgrade grpc to v1.55.0 NOTE: -DTENSORSTORE_USE_SYSTEM_PROTOBUF=ON doesn't appear to work properly to build gRPC after this change. I suspect that we need to generate calls to `include_directories(${Protobuf_INCLUDE_DIRS})` for protobuf dependencies, however bazel_to_cmake doesn't really have a mechanism for that. NOTE: While building on windows I ran into: fatal error C1083: Cannot open include file: 'google/protobuf/compiler/objectivec/text_format_decode_data.h': No such file or directory If this happens, then underlying issue is that MSVC cannot handle long path lengths. See related bugs: * bazelbuild/bazel#18683 * protocolbuffers/protobuf#12947 PiperOrigin-RevId: 540626233 Change-Id: I0887d0126bab695956c9d11793ac7c6764537d79
protocolbuffers/protobuf#12947 PiperOrigin-RevId: 540800426 Change-Id: I67793ade781c0f77b93ae2b343b8a1c6914e278a
@meteorcloudy could you recommend the best solution for us? |
Using a shorter output base / output user root is the only direct workaround I can think of as recommended at https://bazel.build/configure/windows#long-path-issues @oquenchil Is it possible to create the |
CC @jskeet, would it be an option to not use |
@hlopko: I'm afraid I'm really not the person to ask about this - my understanding of both C++ and Bazel is pretty low. I have no objection to it being changed, but it's not something I'd have confidence either doing or reviewing. |
I guess we can at least try and see if it breaks any tests? ;) |
I don't think so.
That doesn't leave out anything that can be shortened using a hash like we do for shared libraries on Windows. I can't think of anything except what's already been mentioned: not using |
This is currently also a package path right? Can we replace it with a hash? Essentially
becomes
|
I think that this would require a bazel change; basically bazel would need to be able to hand out such a directory with a shorter prefix. I don't know what that would entail. I filed a feature request for that: bazelbuild/bazel#18683 It was recategorized to c++ rules, which may not be a great category. |
Looking around, it seems that the reason to have all the 73 instances if include_prefix in the repo is easier compatibility with the Google internal clone of this repository. One solution here would be to fix header include paths in all C++ sources so we don't need include_prefix at all. |
i try it do work, thanks |
This requires: * Tentatively abandoning the use of Bzlmod. Protobuf doesn't support bzlmod yet [1] and the latest available protobuf in the Bazel Central Registry is 21.7 which provides incompatible JSON APIs [2]. Moving forward, we can re-introduce bzlmod when protobuf and Abseil officially support it. * Using the `--output_base=C:\O ` Bazel statup option on Windows [3] when building for Kokoro tests. [1] protocolbuffers/protobuf#14313 [2] https://github.com/bazelbuild/bazel-central-registry/tree/7004deef786dc4152eedd3dfedf91b5b6b4590d4/modules/protobuf [3] protocolbuffers/protobuf#12947 PiperOrigin-RevId: 586313367 Change-Id: Idc3f6d5a6e34b7da8fe51ea42ca354027a94a369
This requires: * Tentatively abandoning the use of Bzlmod. Protobuf doesn't support bzlmod yet [1] and the latest available protobuf in the Bazel Central Registry is 21.7 which provides incompatible JSON APIs [2]. Moving forward, we can re-introduce bzlmod when protobuf and Abseil officially support it. * Using the `--output_base=C:\O ` Bazel statup option on Windows [3] when building for Kokoro tests. [1] protocolbuffers/protobuf#14313 [2] https://github.com/bazelbuild/bazel-central-registry/tree/7004deef786dc4152eedd3dfedf91b5b6b4590d4/modules/protobuf [3] protocolbuffers/protobuf#12947 PiperOrigin-RevId: 586313367
As of Sept, upb has moved into protobuf; update to support that. This reworks how protos are generated to feel a bit closer to the aspect-like code generation of Bazel. Now each proto generator is registered, and each proto_library() rule will invoke all registered aspects; this will allow consistent and reliable invocation of protoc across all sub-repositories. The change permits removing duplicate .proto files from local_proto_mirror, and simplifies the boostrapping process. Also note, on windows the updated protobuf is more likely to trigger file not found issues due to path length; if this happens, add the bazel startup option: startup --output_user_root=C:/_out See also: bazelbuild/bazel#18683 protocolbuffers/protobuf#12947 grpc/grpc#33986 PiperOrigin-RevId: 588606143 Change-Id: Iafa5f16b0a9bbc1058a876c0cc739d82743f96a4
I'm running into this issue on |
The setup-bazel action sets the output base on Windows to D:\_bazel, which should be short enough to avoid protocolbuffers/protobuf#12947.
The setup-bazel action sets the output base on Windows to D:\_bazel, which should be short enough to avoid protocolbuffers/protobuf#12947.
We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment. This issue is labeled |
The severe file length limits under by bazel+msvc are a known painpoint, not just for external users but also for the protobuf team's development itself (we have been warping our own file names and directory structure to keep our max length of paths to be approx 60 chars, with remainder of the 260 character limit is burned by other overhead). Since we don't have any good way to mitigate this issue, and because bazel+msvc usage is low compared to other ways to build on Windows, we are planning to drop support for bazel+msvc in Protobuf 34 (the Q1 2026 release). Bazel 30 already disables support for this combination, with a flag to reenable support for it which gives time to gather feedback on this as well as give folks a longer horizon to migrate their builds to either make+msvc or bazel+clang-cl which don't suffer from this artificial limitation without being stuck on Protobuf 29 or lower: https://protobuf.dev/news/v30/#poison-msvc--bazel
Unfortunately a lot of bazel behaviors are inherently symlink related (besides what was mentioned above, also e.g. strip_prefix on proto library), I don't think it'll be a winning battle to use bazel on windows without symlinks. If you need to build on an env without symlinks, I think you probably want look into using Make builds on that environment; I'm actually unsure if it also will still have a different set of problems in that env, but at least the most obvious behaviors that are inherently symlink-related are in bazel behaviors. |
@esrauchg The |
Apologies, accidentally used the incorrect url: corrected to https://protobuf.dev/news/v30/#poison-msvc--bazel |
Um, I don't know if this is necessarily a proper statement. Regardless, bazel+msvc should be a supported configuration as it has been. Please don't 'drop support'... lets find a way forward, I know we can. |
Keeps the `README` guidance in sync with what we're actually using in `WORKSPACE` for consistency's sake. @crt-31 and I found that the Windows build failure for bazel-contrib#1710 mentioned in the earlier commit is related to the Windows/MSVC file path length limit. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. - protocolbuffers/protobuf#12947 Furthermore, the Protobuf team currently plans to just drop MSVC support: - https://protobuf.dev/news/v30/#poison-msvc--bazel - protocolbuffers/protobuf#20085 I plan to experiment again with "Protobuf Toolchainization", which I'd tried in October when beginning the Bzlmod experiment. Here are some interesting background resources before I dig in on that: - bazelbuild/rules_proto#213 - bazelbuild/rules_proto#179 - https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0 - https://github.com/aspect-build/toolchains_protoc/ - protocolbuffers/protobuf#20182 - protocolbuffers/protobuf#19679 - protocolbuffers/protobuf#19558
Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Part of bazel-contrib#1482 and bazel-contrib#1652. Stops `protoc` recompilation, and fixes the build breakage in bazel-contrib#1710 due to `protobuf` include paths exceeding the Visual Studio path length limit. The updates to `scala_proto/scala_proto_toolchain.bzl` were inspired by: - protocolbuffers/protobuf: bazel: Remove hardcoded dependency on //:protoc from language runtimes #19679 protocolbuffers/protobuf#19679 The `proto_lang_toolchain` call was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Adds `scripts/update_protoc_integrity.py` to automatically update `scala/private/protoc/protoc_integrity.bzl`. This should make builds of `rules_scala` much faster all around. Given the fact that this feature depends on recent `protobuf` versions, and the Windows `protobuf` build breaks without it, we have a catch-22. It likely can't be separated from the rest of bazel-contrib#1710, though I would prefer that. It also seems likely that we'd eventually need to do this to continue supporting Windows, per: - protocolbuffers/protobuf#12947 - https://protobuf.dev/news/v30/#poison-msvc--bazel - protocolbuffers/protobuf#20085 More background on proto toolchainization: - Proto Toolchainisation Design Doc https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit - bazelbuild/bazel: Protobuf repo recompilation sensitivity bazelbuild/bazel#7095 - bazelbuild/rules_proto: Implement proto toolchainisation bazelbuild/rules_proto#179 - rules_proto 6.0.0 release notes mentioning Protobuf Toolchainisation https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0
Please provide any feedback or questions about our ongoing bazel+msvc support on #20085 Note that my short comment didn't reflect all of the effort and thinking that went into it before taking a decision to move in that direction: we do not take the decision lightly, but unfortunately have not been able to find any sufficient technical solution to the problem. |
What version of protobuf and what language are you using?
Version: main/v3.6.0/v3.5.0 etc. (NOTE: please try updating to the latest version of protoc/runtime possible beforehand to attempt to resolve your problem)
Language: C++/Java/Python/C#/Ruby/PHP/Objective-C/Javascript
What operating system (Linux, Windows, ...) and version?
Windows
What runtime / compiler are you using (e.g., python version or gcc version)
VC 14.16.27023
What did you do?
Steps to reproduce the behavior:
What did you expect to see
build completed
What did you see instead?
C:/users/xyz/_bazel_xyz/y7fcwcs2/external/com_google_protobuf/src/google/protobuf/compiler/csharp/BUILD.bazel:22:11: Compiling src/google/protobuf/compiler/csharp/csharp_repeated_enum_field.cc [for tool] failed: (Exit 2): cl.exe failed: error executing command (from target @com_google_protobuf//src/google/protobuf/compiler/csharp:csharp) C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\cl.exe /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE ... (remaining 65 arguments skipped)
cl : Command line warning D9002 : ignoring unknown option '-std=c++14'
external/com_google_protobuf/src/google/protobuf/compiler/csharp/csharp_repeated_enum_field.cc(31): fatal error C1083: Cannot open include file: 'google/protobuf/compiler/csharp/csharp_repeated_enum_field.h': No such file or directory
ERROR: C:/users/xyz/_bazel_xyz/y7fcwcs2/external/com_google_protobuf/src/google/protobuf/compiler/BUILD.bazel:74:11: Compiling src/google/protobuf/compiler/command_line_interface.cc [for tool] failed: (Exit 2): cl.exe failed: error executing command (from target @com_google_protobuf//src/google/protobuf/compiler:command_line_interface) C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\cl.exe /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE ... (remaining 68 arguments skipped)
cl : Command line warning D9002 : ignoring unknown option '-std=c++14'
external/com_google_protobuf/src/google/protobuf/compiler/command_line_interface.cc(35): fatal error C1083: Cannot open include file: 'google/protobuf/compiler/command_line_interface.h': No such file or directory
ERROR: C:/users/xyz/_bazel_xyz/y7fcwcs2/external/com_google_protobuf/src/google/protobuf/compiler/objectivec/BUILD.bazel:53:11: Compiling src/google/protobuf/compiler/objectivec/primitive_field.cc [for tool] failed: (Exit 2): cl.exe failed: error executing command (from target @com_google_protobuf//src/google/protobuf/compiler/objectivec:objectivec) C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\cl.exe /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE ... (remaining 67 arguments skipped)
cl : Command line warning D9002 : ignoring unknown option '-std=c++14'
external/com_google_protobuf/src/google/protobuf/compiler/objectivec/primitive_field.cc(31): fatal error C1083: Cannot open include file: 'google/protobuf/compiler/objectivec/primitive_field.h': No such file or directory
INFO: Elapsed time: 4.749s, Critical Path: 0.59s
INFO: 29 processes: 29 internal.
FAILED: Build did NOT complete successfully
Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
Anything else we should know about your project / environment
The text was updated successfully, but these errors were encountered: