From 4daeb85fd67740cc1cd0d1403e3933097b951b8c Mon Sep 17 00:00:00 2001 From: Stephen Sherratt Date: Mon, 19 May 2025 17:04:48 +1000 Subject: [PATCH 1/4] Add post about portable lock directories in dune Signed-off-by: Stephen Sherratt --- ...directories-for-dune-package-management.md | 136 ++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md diff --git a/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md b/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md new file mode 100644 index 0000000000..0f4d4c3339 --- /dev/null +++ b/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md @@ -0,0 +1,136 @@ +--- +title: Portable Lock Directories for Dune Package Management +tags: [dune, developer-preview] +--- + +_Discuss this post on [Discuss](https://discuss.ocaml.org/t/portable-lock-directories-for-dune-package-management/16669)!_ + +We've recently made a change to how lock directories work in the [Dune Developer +Preview](https://preview.dune.build/). + +Previously when Dune would solve dependencies for a project and generate a lock +directory, the lock directory would be specialized for the computer where it was +generated, with no guarantee it would work on a different computer. This posed a +problem for checking lock directories into version control for projects with +multiple developers, since one developer might lock the project on their Mac, +say, only for another developer on Linux to be unable to build it due to its +MacOS-specific lock directory. + +This post is to announce that Dune now supports generating _portable_ lock +directories; a lock directory generated on one computer will now contain a +dependency solution for a range of different computers, making it safe to check +lock directories into version control. + +## Technical Details + +In Opam the dependencies of a package can be different depending on properties +of the computer where the package is being installed. A package might have a +different set of dependencies when being installed on MacOS verses Linux verses +Windows, or the dependencies might vary depending on the CPU architecture. It's +even possible (though quite rare in practice) for the dependencies of a package +to vary between operating system distributions, or even operating system +versions. + +This expressive power makes Opam very flexible as it allows packages to be +specialized for the environment where they will be installed. The drawback of +this approach is that there might not be a single solution to a dependency +problem that works everywhere. Each combination of +OS/architecture/distro/version could, in theory, require a different dependency +solution. There are way too many combinations of those properties to run Dune's +dependency solver once for each combination in a reasonable amount of time. +Instead we elected to compromise and have Dune only generate a solution for +common platforms by default, while allowing users to specify a custom list of +platforms to solve for in their `dune-workspace` file. + +Lockfiles now look a little different to account for the fact that they now +contain multiple different platform-specific dependency solutions. For example, +formerly, the lockfile for the `ocaml-compiler` package on an x86 machine running +Windows, you might have had a `depends` field like: + +```scheme +(depends arch-x86_64 system-mingw mingw-w64-shims flexdll) +``` + +Most of these dependencies are specific to Windows; it's unlikely that you'll be +able to install any of these dependencies on Linux or MacOS. + +With the portable lock directories feature enabled, this field now might look like: + +```scheme +(depends + (choice + ((((arch x86_64) + (os linux)) + ((arch arm64) + (os linux)) + ((arch x86_64) + (os macos) + (os-distribution homebrew) + (os-family homebrew)) + ((arch arm64) + (os macos) + (os-distribution homebrew) + (os-family homebrew))) + ()) + ((((arch x86_64) + (os win32) + (os-distribution cygwin) + (os-family windows))) + (arch-x86_64 system-mingw mingw-w64-shims flexdll)) + ((((arch arm64) + (os win32) + (os-distribution cygwin) + (os-family windows))) + (system-mingw mingw-w64-shims flexdll)))) +``` + +This new syntax is similar to a match-statement, listing the dependencies for +each platform for which Dune's solver ran. You can change the platforms Dune +will solve for by adding something like this to `dune-workspace`: + +```scheme +(lock_dir + (solve_for_platforms + ((arch arm64) + (os openbsd)) + ((arch x86_32) + (os win32)))) +``` + +After running `dune pkg lock` again, the lockfile for `ocaml-compiler` will be +updated with these dependencies: + +```scheme +(depends + (choice + ((((arch arm64) (os openbsd))) ()) + ((((arch x86_32) + (os win32))) + (system-mingw ocaml-option-bytecode-only flexdll)))) +``` + +A few other fields of lockfiles now also use the new syntax. Dune lockfiles +contain the commands needed to build and install each package, as well as the +names of any system packages needed by the Opam package, and each of these fields +can also have platform-specific values. + +Lockfile names now include the version number of the package. The +`ocaml-compiler` package used to have a lockfile named `ocaml-compiler.pkg` but +now has a name like `ocaml-compiler.5.3.0.pkg` instead. This is because it's +possible that different platforms may require different versions of the same +package in the dependency solution, so the lock directory needs to be able to +contain multiple lockfiles for the same package without them colliding on +filename. + +## How do I get it? + +This feature is live in the latest version of the [Dune Developer +Preview](https://preview.dune.build/). Follow the instructions on that page to +install a version of Dune with this feature. With portable lock directories +enabled, Dune will temporarily remain backwards compatible with the original +lock directory format, though support will likely be dropped at some point. +Generate a new lock directory by running `dune pkg lock`. You'll know your lock +directory is portable if each file inside it has a version number in its +filename. + +Happy reproducible building! From b81bf5617444fb5f12100ad7fc8044356f88156a Mon Sep 17 00:00:00 2001 From: sabine <6594573+sabine@users.noreply.github.com> Date: Thu, 29 May 2025 09:16:31 +0200 Subject: [PATCH 2/4] Update data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md --- ...-19-portable-lock-directories-for-dune-package-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md b/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md index 0f4d4c3339..c2aef732eb 100644 --- a/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md +++ b/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md @@ -25,7 +25,7 @@ lock directories into version control. In Opam the dependencies of a package can be different depending on properties of the computer where the package is being installed. A package might have a -different set of dependencies when being installed on MacOS verses Linux verses +different set of dependencies when being installed on MacOS verses Linux versus Windows, or the dependencies might vary depending on the CPU architecture. It's even possible (though quite rare in practice) for the dependencies of a package to vary between operating system distributions, or even operating system From 359282237bba85bd6d29cce235af8fb1467b7f30 Mon Sep 17 00:00:00 2001 From: sabine <6594573+sabine@users.noreply.github.com> Date: Thu, 29 May 2025 09:16:38 +0200 Subject: [PATCH 3/4] Update data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md --- ...-19-portable-lock-directories-for-dune-package-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md b/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md index c2aef732eb..3e2e94def6 100644 --- a/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md +++ b/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md @@ -1,5 +1,5 @@ --- -title: Portable Lock Directories for Dune Package Management +title: Dune Developer Preview: Portable Lock Directories for Dune Package Management tags: [dune, developer-preview] --- From 463a227dabcb0d6f1f4a318e6f052b2ba246ed5e Mon Sep 17 00:00:00 2001 From: sabine <6594573+sabine@users.noreply.github.com> Date: Thu, 29 May 2025 09:22:19 +0200 Subject: [PATCH 4/4] making it clear that this is about dune developer preview --- .../2024-08-26-closing-on-public-beta.md | 2 +- .../2024-09-25-call-for-feedback.md | 2 +- .../2024-10-29-shell-completions-in-dune-developer-preview.md | 2 +- .../2024-11-15-installing-developer-tools-with-dune.md | 2 +- ...-19-portable-lock-directories-for-dune-package-management.md | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) rename data/changelog/posts/{dune => dune-developer-preview}/2024-08-26-closing-on-public-beta.md (97%) rename data/changelog/posts/{dune => dune-developer-preview}/2024-09-25-call-for-feedback.md (98%) rename data/changelog/posts/{dune => dune-developer-preview}/2024-10-29-shell-completions-in-dune-developer-preview.md (94%) rename data/changelog/posts/{dune => dune-developer-preview}/2024-11-15-installing-developer-tools-with-dune.md (99%) rename data/changelog/posts/{dune => dune-developer-preview}/2025-05-19-portable-lock-directories-for-dune-package-management.md (98%) diff --git a/data/changelog/posts/dune/2024-08-26-closing-on-public-beta.md b/data/changelog/posts/dune-developer-preview/2024-08-26-closing-on-public-beta.md similarity index 97% rename from data/changelog/posts/dune/2024-08-26-closing-on-public-beta.md rename to data/changelog/posts/dune-developer-preview/2024-08-26-closing-on-public-beta.md index f911450b6c..bd92638fc5 100644 --- a/data/changelog/posts/dune/2024-08-26-closing-on-public-beta.md +++ b/data/changelog/posts/dune-developer-preview/2024-08-26-closing-on-public-beta.md @@ -1,5 +1,5 @@ --- -title: Getting Ready for the Public Beta +title: "Dune Developer Preview: Getting Ready for the Public Beta" tags: [dune, platform, developer-preview] --- diff --git a/data/changelog/posts/dune/2024-09-25-call-for-feedback.md b/data/changelog/posts/dune-developer-preview/2024-09-25-call-for-feedback.md similarity index 98% rename from data/changelog/posts/dune/2024-09-25-call-for-feedback.md rename to data/changelog/posts/dune-developer-preview/2024-09-25-call-for-feedback.md index 3caa9ef96f..e297280b6d 100644 --- a/data/changelog/posts/dune/2024-09-25-call-for-feedback.md +++ b/data/changelog/posts/dune-developer-preview/2024-09-25-call-for-feedback.md @@ -1,5 +1,5 @@ --- -title: Call for Feedback +title: "Dune Developer Preview: Call for Feedback" tags: [dune, platform, developer-preview] --- diff --git a/data/changelog/posts/dune/2024-10-29-shell-completions-in-dune-developer-preview.md b/data/changelog/posts/dune-developer-preview/2024-10-29-shell-completions-in-dune-developer-preview.md similarity index 94% rename from data/changelog/posts/dune/2024-10-29-shell-completions-in-dune-developer-preview.md rename to data/changelog/posts/dune-developer-preview/2024-10-29-shell-completions-in-dune-developer-preview.md index d77f8f5e70..301fd15352 100644 --- a/data/changelog/posts/dune/2024-10-29-shell-completions-in-dune-developer-preview.md +++ b/data/changelog/posts/dune-developer-preview/2024-10-29-shell-completions-in-dune-developer-preview.md @@ -1,5 +1,5 @@ --- -title: Shell Completions in Dune Developer Preview +title: "Dune Developer Preview: Shell Completions in Dune Developer Preview" tags: [dune, developer-preview] --- diff --git a/data/changelog/posts/dune/2024-11-15-installing-developer-tools-with-dune.md b/data/changelog/posts/dune-developer-preview/2024-11-15-installing-developer-tools-with-dune.md similarity index 99% rename from data/changelog/posts/dune/2024-11-15-installing-developer-tools-with-dune.md rename to data/changelog/posts/dune-developer-preview/2024-11-15-installing-developer-tools-with-dune.md index dbc440e627..3137073d80 100644 --- a/data/changelog/posts/dune/2024-11-15-installing-developer-tools-with-dune.md +++ b/data/changelog/posts/dune-developer-preview/2024-11-15-installing-developer-tools-with-dune.md @@ -1,5 +1,5 @@ --- -title: Installing Developer Tools with Dune +title: "Dune Developer Preview: Installing Developer Tools with Dune" tags: [dune, developer-preview] --- diff --git a/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md b/data/changelog/posts/dune-developer-preview/2025-05-19-portable-lock-directories-for-dune-package-management.md similarity index 98% rename from data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md rename to data/changelog/posts/dune-developer-preview/2025-05-19-portable-lock-directories-for-dune-package-management.md index 3e2e94def6..0767a4104c 100644 --- a/data/changelog/posts/dune/2025-05-19-portable-lock-directories-for-dune-package-management.md +++ b/data/changelog/posts/dune-developer-preview/2025-05-19-portable-lock-directories-for-dune-package-management.md @@ -1,5 +1,5 @@ --- -title: Dune Developer Preview: Portable Lock Directories for Dune Package Management +title: "Dune Developer Preview: Portable Lock Directories for Dune Package Management" tags: [dune, developer-preview] ---