Skip to content

[Feature] Fuse linker #6834

Open
Open
@goloveychuk

Description

@goloveychuk
Contributor
  • I'd be willing to implement this feature (contributing guide)
    This feature is important to have in this repository; a contrib plugin wouldn't do

Describe the user story

We all want fast yarn install, and node_modules which don't eat our disk.
Pnp is pretty good solution, but it has compatibility issues: bundlers, resolvers, etc. Migration effort if pretty high.

Describe the solution you'd like

Fuse linker.
Idea consist from three elements:

  1. pnpm layout.
    I forked pnpm linker, removed hardlinks logic from it.
    I've split packages into two dirs, .store-fuse and .store-unplugged.
    Unplugged has packages which need postinstalls, they are unpacked to fs.
    Fuse has other packages, and mounted as readonly.
    If fuse linker is not supported in OS, all packages fallback to unplugged.

  2. fuse mount of .store-fuse.
    Supported os: linux (libfuse), macos 15.4 (fskit). Probably windows with wsl 2.
    For unprivileged mounts linux needs fusermount util installed.
    Macos need app installed.
    I've implemented custom fuse fs, which ended to be a bit complicated. After finished I've found out that I can generate erofs binary image instead, and reuse erofs-fuse or even erofs native mount on linux (it's supported by kernel, but needs root).

  3. fetching archives on demand from remote cache (optional)
    Fuse fs supports async fs actions, which means that we can add fetching data when its requested.

This all will make yarn install time = resolution + cache fulfil + postinstalls.

Sources: https://github.com/goloveychuk/fuse

Describe the drawbacks of your solution

Needs binaries downloaded, need support by OS, could have bugs.

Describe alternatives you've considered

pnp

@arcanis wdyt?

Activity

arcanis

arcanis commented on Jun 26, 2025

@arcanis
Member

I deemed it not viable many years ago due to:

Not sure how much of that changed since last time. That's why I went with the Node.js-based FS layer: harder to maintain, but at least we were in control of how it executed on users' machines (usually).

goloveychuk

goloveychuk commented on Jun 26, 2025

@goloveychuk
ContributorAuthor

licensing drama

licensing is now fixed, in macos we can now implement fskit extension without relying on macfuse. Thats what I've done in my repo. Libfuse (linked) has LGPL.

portability headaches + native dependencies

this remains. It could be fixed a bit by building go binaries (for linux + wsl2), which are completely statically linked.
But still, linux should have root or fusermount installed, and macos should have extensions installed.
Still, it could fallback to unpack if platform is not supported.
I'm not saying it should replace pnp. But it could be an alternative which has some pros and const.
Also, it could be external berry plugin, but if it would be part of berry builtin plugins, it could simplify an adoption.

Btw, it would help if berry has a plugin interface for Cache. Currently it's hardcoded to zip all over the place.
I can implement this if it fits into design.

goloveychuk

goloveychuk commented on Jun 26, 2025

@goloveychuk
ContributorAuthor

Btw, about whole pnp + nodejs runtime. I saw your pr to nodejs which adds native zip support, and it's not merged (and probably won't ever).
I was thinking to target this by doing two steps.

  1. RFC for packagemaps
    I think we can write a RFC for nodejs (mb with @guybedford help) which will define a manifest file which will tell nodejs where to find a package from other package. Very similar to importmaps, but from what I understand importmap won't work with nodejs resolution logic we have now (e.g. package exports).
    Idea is to have smth like
        "file:///abspath/package1": {
                   "package2": "file:///abspath/package2"
        },
      ...
       "*": { ...hoistedPackages }

This will fix many problems with node_modules, no need for pnpm, no hoisting needed, can reuse global cache.
Implementation in nodejs resolution algo should be pretty easy, just change the "node_modules/package" lookups with json lookup. Other resolution logic will remain the same (exports field).
2) second step is to introduce other protocol, maybe zip, maybe some other, optimized for performance (erofs, squashfs, custom format)
This will require implementing vfs subsystem in nodejs itself, they probably won't accept it, but we can try.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @arcanis@goloveychuk

        Issue actions

          [Feature] Fuse linker · Issue #6834 · yarnpkg/berry