diff --git a/data/changelog/posts/dune/2024-08-26-closing-on-public-beta.md b/data/changelog/posts/dune-developer-preview/2024-08-26-closing-on-public-beta.md similarity index 97% rename from data/changelog/posts/dune/2024-08-26-closing-on-public-beta.md rename to data/changelog/posts/dune-developer-preview/2024-08-26-closing-on-public-beta.md index f911450b6c..bd92638fc5 100644 --- a/data/changelog/posts/dune/2024-08-26-closing-on-public-beta.md +++ b/data/changelog/posts/dune-developer-preview/2024-08-26-closing-on-public-beta.md @@ -1,5 +1,5 @@ --- -title: Getting Ready for the Public Beta +title: "Dune Developer Preview: Getting Ready for the Public Beta" tags: [dune, platform, developer-preview] --- diff --git a/data/changelog/posts/dune/2024-09-25-call-for-feedback.md b/data/changelog/posts/dune-developer-preview/2024-09-25-call-for-feedback.md similarity index 98% rename from data/changelog/posts/dune/2024-09-25-call-for-feedback.md rename to data/changelog/posts/dune-developer-preview/2024-09-25-call-for-feedback.md index 3caa9ef96f..e297280b6d 100644 --- a/data/changelog/posts/dune/2024-09-25-call-for-feedback.md +++ b/data/changelog/posts/dune-developer-preview/2024-09-25-call-for-feedback.md @@ -1,5 +1,5 @@ --- -title: Call for Feedback +title: "Dune Developer Preview: Call for Feedback" tags: [dune, platform, developer-preview] --- diff --git a/data/changelog/posts/dune/2024-10-29-shell-completions-in-dune-developer-preview.md b/data/changelog/posts/dune-developer-preview/2024-10-29-shell-completions-in-dune-developer-preview.md similarity index 94% rename from data/changelog/posts/dune/2024-10-29-shell-completions-in-dune-developer-preview.md rename to data/changelog/posts/dune-developer-preview/2024-10-29-shell-completions-in-dune-developer-preview.md index d77f8f5e70..301fd15352 100644 --- a/data/changelog/posts/dune/2024-10-29-shell-completions-in-dune-developer-preview.md +++ b/data/changelog/posts/dune-developer-preview/2024-10-29-shell-completions-in-dune-developer-preview.md @@ -1,5 +1,5 @@ --- -title: Shell Completions in Dune Developer Preview +title: "Dune Developer Preview: Shell Completions in Dune Developer Preview" tags: [dune, developer-preview] --- diff --git a/data/changelog/posts/dune/2024-11-15-installing-developer-tools-with-dune.md b/data/changelog/posts/dune-developer-preview/2024-11-15-installing-developer-tools-with-dune.md similarity index 99% rename from data/changelog/posts/dune/2024-11-15-installing-developer-tools-with-dune.md rename to data/changelog/posts/dune-developer-preview/2024-11-15-installing-developer-tools-with-dune.md index dbc440e627..3137073d80 100644 --- a/data/changelog/posts/dune/2024-11-15-installing-developer-tools-with-dune.md +++ b/data/changelog/posts/dune-developer-preview/2024-11-15-installing-developer-tools-with-dune.md @@ -1,5 +1,5 @@ --- -title: Installing Developer Tools with Dune +title: "Dune Developer Preview: Installing Developer Tools with Dune" tags: [dune, developer-preview] --- diff --git a/data/changelog/posts/dune-developer-preview/2025-05-19-portable-lock-directories-for-dune-package-management.md b/data/changelog/posts/dune-developer-preview/2025-05-19-portable-lock-directories-for-dune-package-management.md new file mode 100644 index 0000000000..0767a4104c --- /dev/null +++ b/data/changelog/posts/dune-developer-preview/2025-05-19-portable-lock-directories-for-dune-package-management.md @@ -0,0 +1,136 @@ +--- +title: "Dune Developer Preview: Portable Lock Directories for Dune Package Management" +tags: [dune, developer-preview] +--- + +_Discuss this post on [Discuss](https://discuss.ocaml.org/t/portable-lock-directories-for-dune-package-management/16669)!_ + +We've recently made a change to how lock directories work in the [Dune Developer +Preview](https://preview.dune.build/). + +Previously when Dune would solve dependencies for a project and generate a lock +directory, the lock directory would be specialized for the computer where it was +generated, with no guarantee it would work on a different computer. This posed a +problem for checking lock directories into version control for projects with +multiple developers, since one developer might lock the project on their Mac, +say, only for another developer on Linux to be unable to build it due to its +MacOS-specific lock directory. + +This post is to announce that Dune now supports generating _portable_ lock +directories; a lock directory generated on one computer will now contain a +dependency solution for a range of different computers, making it safe to check +lock directories into version control. + +## Technical Details + +In Opam the dependencies of a package can be different depending on properties +of the computer where the package is being installed. A package might have a +different set of dependencies when being installed on MacOS verses Linux versus +Windows, or the dependencies might vary depending on the CPU architecture. It's +even possible (though quite rare in practice) for the dependencies of a package +to vary between operating system distributions, or even operating system +versions. + +This expressive power makes Opam very flexible as it allows packages to be +specialized for the environment where they will be installed. The drawback of +this approach is that there might not be a single solution to a dependency +problem that works everywhere. Each combination of +OS/architecture/distro/version could, in theory, require a different dependency +solution. There are way too many combinations of those properties to run Dune's +dependency solver once for each combination in a reasonable amount of time. +Instead we elected to compromise and have Dune only generate a solution for +common platforms by default, while allowing users to specify a custom list of +platforms to solve for in their `dune-workspace` file. + +Lockfiles now look a little different to account for the fact that they now +contain multiple different platform-specific dependency solutions. For example, +formerly, the lockfile for the `ocaml-compiler` package on an x86 machine running +Windows, you might have had a `depends` field like: + +```scheme +(depends arch-x86_64 system-mingw mingw-w64-shims flexdll) +``` + +Most of these dependencies are specific to Windows; it's unlikely that you'll be +able to install any of these dependencies on Linux or MacOS. + +With the portable lock directories feature enabled, this field now might look like: + +```scheme +(depends + (choice + ((((arch x86_64) + (os linux)) + ((arch arm64) + (os linux)) + ((arch x86_64) + (os macos) + (os-distribution homebrew) + (os-family homebrew)) + ((arch arm64) + (os macos) + (os-distribution homebrew) + (os-family homebrew))) + ()) + ((((arch x86_64) + (os win32) + (os-distribution cygwin) + (os-family windows))) + (arch-x86_64 system-mingw mingw-w64-shims flexdll)) + ((((arch arm64) + (os win32) + (os-distribution cygwin) + (os-family windows))) + (system-mingw mingw-w64-shims flexdll)))) +``` + +This new syntax is similar to a match-statement, listing the dependencies for +each platform for which Dune's solver ran. You can change the platforms Dune +will solve for by adding something like this to `dune-workspace`: + +```scheme +(lock_dir + (solve_for_platforms + ((arch arm64) + (os openbsd)) + ((arch x86_32) + (os win32)))) +``` + +After running `dune pkg lock` again, the lockfile for `ocaml-compiler` will be +updated with these dependencies: + +```scheme +(depends + (choice + ((((arch arm64) (os openbsd))) ()) + ((((arch x86_32) + (os win32))) + (system-mingw ocaml-option-bytecode-only flexdll)))) +``` + +A few other fields of lockfiles now also use the new syntax. Dune lockfiles +contain the commands needed to build and install each package, as well as the +names of any system packages needed by the Opam package, and each of these fields +can also have platform-specific values. + +Lockfile names now include the version number of the package. The +`ocaml-compiler` package used to have a lockfile named `ocaml-compiler.pkg` but +now has a name like `ocaml-compiler.5.3.0.pkg` instead. This is because it's +possible that different platforms may require different versions of the same +package in the dependency solution, so the lock directory needs to be able to +contain multiple lockfiles for the same package without them colliding on +filename. + +## How do I get it? + +This feature is live in the latest version of the [Dune Developer +Preview](https://preview.dune.build/). Follow the instructions on that page to +install a version of Dune with this feature. With portable lock directories +enabled, Dune will temporarily remain backwards compatible with the original +lock directory format, though support will likely be dropped at some point. +Generate a new lock directory by running `dune pkg lock`. You'll know your lock +directory is portable if each file inside it has a version number in its +filename. + +Happy reproducible building!