[Bazel] Migrate to Bzlmod

[aaronmondal]

The workspace build is now gated behind --config=deprecated_workspace.

See utils/bazel/examples for the new setup instructions.

While not necessarily the most intuitive implementation, this seems to
be the only way retain the flexibility of all potential usecases as of
Bazel 8. The module build supports:

• The in-tree build from within the utils/bazel directory.
• The llvm-project cloned into an arbitrary directory (e.g. for local
  development, submodule setups and and Nix users).
• The llvm-project fetched from github via an archive.

Since module overrides are invisible to third-party dependents, every
project that has the llvm project "somewhere" in its dependency tree
needs to create an explicit override in their root module. It's possible
to avoid this inconvenience by creating custom bazel registries, but
only makes sense to do for the next stable tag that contains this commit.

If you need to link against a nonhermetic system variant of an external
dependency you can override the repositories from the
llvm_project_overlay extension like this:

'''MODULE.bazel'''
new_local_repository = use_repo_rule(
    "@bazel_tools//tools/build_defs/repo:local.bzl",
    "new_local_repository",
)

new_local_repository(
    name = "zlib-ng",
    path = "xxxx",
    build_file_content = "xxxx",
)

override_repo(llvm_project_overlay, "zlib-ng")

[aaronmondal]

cc @keith

Still under construction, but at least this works with bazelisk build @llvm-project//clang

[keith]

think you'll get back to this sometime?

[aaronmondal]

I believe this is still blocked by these:

• bazelbuild/bazel@6f254ce, i.e. we currently don't have access to new_local_repository with non-nightly Bazel
• layering_check doesn't work without sandboxing bazelbuild/bazel#21592, i.e. the recent versions of Bazel that we'd need fail with "undeclared inclusions xxx.cppmap".

I just tried building with 8.0.0-pre.20240607.2 and bazelisk test --config=ci @llvm-project//... and it still failed because of that.

[maxbartel]

Seems like the two issues are resolved now. Any chance you want to try the transition again?

[aaronmondal]

Seems like the two issues are resolved now. Any chance you want to try the transition again?

Will retry this weekend 👍

I'll need to figure out a more elegant solution for the patching and I think I figured out a better way for the external dependencies via label-flags so that we no longer need to maintain the custom nonhermeticity logic.

[maxbartel]

@keith @rupprecht Do you think you could maybe review the PR soonish? It is quite the nice QoL improvement

[aaronmondal]

FYI I've pulled out the zlib/zstd changes in a cleaner way in #126729

[rupprecht]

I am not yet very familiar w/ bzlmod, so I'm not a great reviewer. But if it works, it works. Thanks for taking care of this!

[rupprecht]

I kinda like having this pattern, as it makes the build files less cluttered (fewer selects). Could we re-introduce a wrapper around it, similar to the maybe_pfm target in this file? i.e. something like:

cc_library(
    name = "maybe_zlib",
    defines = select({
        ":zlib_enabled": ["LLVM_ENABLE_ZLIB=1"],
        "//conditions:default": [],
    }),
    deps = select({
        ":zlib_enabled": ["@zlib-ng"],
        "//conditions:default": [],
    }),
)

[aaronmondal]

Moved to #126729

I did a lot of back-and-forth with the "maybe pattern". The cleaner looking callsites ("depsites" I guess 😅) are appealing but I ultimately came to the conclusion that it seems like a bad idea because it abuses implementation details that "just happen" to work in the cc_library case. That is, this isn't a generally applicable pattern that could be used for arbitrary rules because deps might not always be implemented transitively.

It also introduces additonal complexity that makes the build graph less efficient (an edge to maybe_zlib even if it's unused) and harder to reason about (can no longer easily see which config option influences the dep - need to go to the maybe target to check).

However, we can still make this less verbose. #126729 moves the LLVM_ENABLE_ZLIB define to the config rather than a defines attribute. This not only saves a lot off command-line length but also reduces the overall number of selects. Now the 4 occurances of the zlib dep in the overlay add just 3 additional selects over the maybe approach (as opposed to the previous 6) and just 1 additional select in the zstd case (as opposed to the previous 2).

[rupprecht]

Not sure what this comment means, would you mind clarifying?

[aaronmondal]

Technically we should treat the http_archive calls as "migration helpers" with the goal of removing all of them and instead importing them via bazel_dep in the MODULE.bazel file. For the deps that are already in the Bazel Central Registry (BCR) I did that. The http_archives are ones that currently don't have a corresponding BCR module.

The nanobind was an exception where a module exists in the BCR, but that variant doesn't work for our usecase at the moment.

I've clarified the comment.

[aaronmondal]

Note: This is now rebased onto (and dependent on) #126729 to separate flag config changes from the bzlmod changes.