Using and editing vcbuild.bat is not as intuitive as Makefile

[avivkeller]

After recently switching from a Linux environment to a Windows machine, I’ve found working with vcbuild.bat in this repository to be significantly more difficult than using the Makefile.

The vcbuild.bat script is harder to read and modify, making even simple changes tedious and prone to errors. In contrast, the Makefile is more maintainable, supports parallel builds, and includes conditional logic.

Modernizing vcbuild.bat, or making the Makefile cross-platform, could greatly improve the development workflow on Windows.

---

I've also heard good things from Cross-Platform tools like CMake, but that's a huge undertaking within itself.

[joyeecheung]

It seems to me what would be the path forward is probably, instead of doing anything about vcbuild.bat, try to make our Makefile work on Windows first, and then we can start ditching vcbuild.bat?

[avivkeller]

Sounds good, however, Makefile's aren't easy (at least in my experience) on Windows. All the bash syntax will fail, even the # comments.

---

I'm going to see if I can get a basic CMakeLists.txt setup, but don't wait on me to improve.

[joyeecheung]

There were some past discussions about switching to CMake, although in that context people were discussing about not just replacing the Makefile/vcbuild.bat (which are basically glorified scripts that drive the build system), but replacing the entire build system nodejs/TSC#648 nodejs/TSC#901

[anonrig]

Bazel is a strong candidate with really good caching and parallelization solutions that can help us ditch ccache and ninja.

[avivkeller]

That also sounds nice! Whatever is chosen, I'd be more than happy to dedicate my time to this :-).

Right now, I've made a basic CMakeLists.txt file that can start generating the node exe (but nothing else yet)

To be honest, I'm partial to (at least for now) starting with CMake because it's more widely adopted, and therefore may be simpler for new contributors to grasp, coming from our Make environment

[bnoordhuis]

I have https://github.com/bnoordhuis/v8-cmake that takes a brute force approach to building V8 with cmake. It's definitely not the best way to do it (static library build works but you can't build it as a shared library, for example) but at least it works. Perhaps it can be a helpful starting point.

I remember from my attempt to port node to cmake that ICU and openssl are huge obstacles.

[avivkeller]

(The below is my opinion, so if anything sounds like a statement, it's not.)

After doing some research, I’ve realized that using a build system like Makefile, CMakeLists.txt, or the vcbuild.bat file is more complicated than necessary for our needs.

Currently, our build system is based on GYP. When we run make, it essentially triggers make out/Makefile (or a similar command depending on the platform/compiler). This out/Makefile is generated by GYP, meaning the only real logic in our Makefile is invoking GYP and GYP generated files.

The rest of the Makefile is essentially a collection of script commands, often Python wrappers. For instance, the jstest target just calls tools/test.py with additional arguments.

In contrast, Bun (https://github.com/oven-sh/bun/blob/main/package.json) handles this more efficiently. It keeps the build logic in CMakeLists.txt, while defining all script commands in package.json as part of the Node.js ecosystem.

I believe we should consider a similar approach. Since much of our scripting is already in Python, why maintain a shell-heavy Makefile that ultimately wraps Python? Instead, we could manage the entire process with Python.

For the build step itself, we’d still use Python to invoke GYP and the required platform-specific compilers. But rather than chaining shell scripts, we could opt for a more maintainable, cross-platform solution.

What do you think about a Python Makefile?

I image it like a file like:

@task(
        dependencies=[DEPENDENCIES],
        outputs=[OUTPUTS],
        description=[DESCRIPTION]
)
def TARGET_NAME(CLI_ARGUMENTS):
    pass

[anonrig]

We can simply move to just file and our transition will be a lot easier

[avivkeller]

Oh! I've never heard of that, but that also looks very good for an easy migration! My concern with Justfile is that it doesn't support dynamic functions, which we make use of (i.e. available-node, etc)

[joyeecheung]

The Makefile is just a glorified script, but with dependency checking out of the box, so that we can implement things like "if the test addons have been built and the test files, the headers and the common config etc. have not changed, don't rebuild before we run the addon tests" (addons are built in parallel by a JS script, it may be possible to port that to GYP but we may lose the parallelism, also GYP is harder to configure to drive the addon building logic), or "if the C++ files don't change, don't run the C++ linter, if they do, only run the linter on the changed files", "if the docs don't change, don't rebuild the docs - which are also needed by doc tests". I think that is also missing from the vcbuild.bat as it almost always rebuild a lot of stuff unnecessarily. If we want to switch to a different scripting option, that dependency checking part should also be ported over.

(Personally most of the time I just use configure + ninja locally, and only very occasionally run the full test suite locally. On Windows I've come to realize that it's better to avoid vcbuild.bat and just use visual studio for proper incremental builds. If some cross-platform scripting with proper incremental build & dependency checking support is available I would be happy to use that. But if dependency checking is missing I would still hope that we can just port the tasks to visual studio custom targets)