RFE: Articulate benefits of moving code to internal/...

[isaacs]

I realize that this almost certainly has been discussed already, but could someone point me to an explanation somewhere for the benefits of moving code to be inaccessible by userland JavaScript programs?

The guiding principles of this project explicitly state that "Change just for the sake of change must be avoided" and that there's a goal of "deferring, as much as possible, additional capabilities and features to add-on modules or applications".

What value of the project is served by these sorts of architectural changes? It seems like it disrupts extant userland programs while also reducing the ability to defer additional capabilities to add-on modules or applications, so I'd expect that there's some very significant benefit to make that worthwhile.

(continued from discussion at isaacs/node-graceful-fs@07701dc#commitcomment-18677940)

[ChALkeR]

/cc @nodejs/ctc, @thealphanerd, @thefourtheye, and, perhaps, @phated

[vkurchatkin]

Internal modules are supposed to prevent core stagnation because of userland modules (such as graceful-fs) abusing private APIs and other implementation details.

[Qard]

I think the benefit started largely as a dumping ground for stuff that was needed by core, but core did not want to officially support, like freelist. It seems to have turned into a blackhole though, which is consuming anything even remotely in the realm of things we don't want to support.

[ChALkeR]

@isaacs The thing is — we should either state something as supported or not supported.

I don't think that we could possibly support reevaluating core node modules code, as well as monkey-patching core modules code — because either of that could break in a semver-patch version for various reasons, and there is no way to prevent that.

For monkey-patching — internal implementation details could change, breaking the monkey-patched code.

For reevaluating — the native binings API, for example, are not covered by stability level obligations, afaik, and the internal implementation could change. Also, other things are being hidden there that are not covered by stability level obligations, like internal modules, which are implementation details for core modules.

Internal modules allow us to re-use some code chunks without actually cutting them in stone and without copy-pasting those into every core module that uses them — those are shared in the internal/ dir.

[ChALkeR]

Moreover, I don't think that reevaluating (as well as monkey-patching) was actually supported at any point in the past, it just happened to work, and that was the cause of the whole graceful-fs issue — it used something that happened to work, but was never stated as supported, i.e. it relied on implementation details.

And now, once the graceful-fs issue is resolved completely, we won't have to worry about that anymore, and this issue won't come up again.

Ah, also /cc @jasnell and @bnoordhuis, because personal metions > team mentions.

[jasnell]

"dumping ground" and "black hole" it is not. To be certain, very few things have actually ended up as internal modules, and the things that are there make sense being there. The distinction between internal and external is specifically around which bits of functionality need to be shared across multiple core modules but are not part of the core API. Public APIs are not supposed to change, private internal things can so long as it does not change a public API. Take internal/bootstrap_node.js for instance. In general, the things that should end up in an internal module are the things end users should not be using directly.

One great example of this is the normalizeEncoding() method of internal/util.js. This is a utility function that is used by several other APIs in various locations in core. It is not a public facing API, nor should it be, but we do need a good way of sharing that code among the various core modules that use is. So moving that functionality into internal/util.js made sense and works extremely well.

The printDeprecationMessage() API is another great example. It was never intended to be a public API. It is, however, used by multiple core modules so moving it into internal/util.js so it can be easily shared makes sense.

The graceful-fs problem came up because graceful-fs chose to abuse the Node.js API. It made assumptions about the code internals that go beyond the API contract. Where Node.js chooses to keep it's internal/private code should not be a consideration for module developers so long as the external/public API contract is maintained or the changes are properly signaled (e.g. semver-major with proper deprecation)

[Trott]

graceful-fs chose to abuse the Node.js API

I think this is a better description:

graceful-js used undocumented elements of the Node.js API that probably should not have been exposed in the first place

Node.js has a choice here. I'm not saying one choice or the other is the right one. But we have the ecosystem we have. Implying that the fault lies with graceful-js doesn't inform the conversation. It is an unimportant detail.

[Fishrock123]

Internal modules are great for allowing smoother core development with less potential impact to the ecosystem. I don't think we could reasonably do without them.

Doing without them is like asking $module to make all of it's dependencies part of it's public API. Totally no-go in my opinion.

Sorry this broke some of your stuff, but I think it would be better for us to continue moving forward to getting past that pain.

The guiding principles of this project explicitly state that "Change just for the sake of change must be avoided" and that there's a goal of "deferring, as much as possible, additional capabilities and features to add-on modules or applications".

I don't see how the internal modules are covered in this, but this does sound like something of an argument against the WIP unicode module.

What value of the project is served by these sorts of architectural changes?

Maintainability. If you would like it explicitly stated on the website, I'm sure that would be possible...

while also reducing the ability to defer additional capabilities to add-on modules or applications, so I'd expect that there's some very significant benefit to make that worthwhile.

I don't understand how this would be the case. Care to elaborate @isaacs?

[ChALkeR]

@Trott

graceful-js used undocumented elements of the Node.js API that probably should not have been exposed in the first place

… Implying that the fault lies with graceful-js doesn't inform the conversation. …

«Fault» does not mean anything here, it is not important. According to the above, we had an issue:

API that probably should not have been exposed in the first place

It's was not an API, it was just the fact that re-evaluation was technically possible, but unsupported. Nothing was done to prevent it, but it was a reasonable assumption that it is clear enough that we just can't support it (i.e. it's stability between patch versions).

It's fixed now on Node.js side — re-evaluation throws an error, and only one module was found to be affected directly — graceful-fs, and only its old versions, as graceful-fs@4 was released more than a year ago.

Only a few modules are affected indirectly atm, and the fix path for those is straightforward — it requires only a graceful-fs version bump.

So this in fact would prevent further breakages.

[phated]

@Fishrock123 you broke the ecosystem. That's not okay. Gulp 3.x cannot and will not be updating graceful-fs past 3.x due to semver breaking changes. And now you are completely dropping support for a huge ecosystem player. That seems like an extremely broken stance to take.

[Fishrock123]

@phated Can we patch <4.x to use the new shim or not?

[Fishrock123]

you broke the ecosystem.

And we unbroke it, so now we're discussing how to move forward.