GHC-internal modules in base

[bgamari]

(The proposal eventually approved in this thread is #146 (comment) — Bodigrim, Sep 2023)

1. Background

Currently the base package exposes many internal implementation details of the implementation base functionality. By "internal implementation details" we mean functions and data types that are part of GHC's realisation of some exposed function, but which were never intended to be directly used by clients of the base library. For instance, the GHC.Base.mapFB function is a necessary exposed part of the fusion framework for map but which GHC's authors never intended users to call.

This lack of clarity is bad for several reasons:

• Users have no way to know which functions are part of the "intended, stable API" and which are part of the "internal, implementation details". Consequently, they may accidentally rely on the latter; they simply have no way to tell.

• GHC's developers are hampered in modifying the implementation because too much is exposed. This imposes a high backward-compatibility burden, one that is an accident of history.

This status quo leaves much to be desired: users tend to rely on any interface available to them and therefore GHC developers are susceptible to breaking users when changing implementation details within base. On the other hand, there is a clear need to be able to iterate on the implementation of GHC and its base library: fixing compiler bugs may require the introduction of new internal yet exposed primitives (c.f. the changes made in the implementation of unsafeCoerce in GHC 9.0) and improving runtime performance may require changes in the types of exposed internal implementation (c.f. GHC #22946).

These difficulties are discussed in CLC #105.

2. Proposal

We propose to classify the modules of base into three groups:

• Hidden: these are simply the existing non-exposed modules (other-module in Cabal terms). No change here.

• External: these modules comprise the public API of base.

  • The GHC team makes strenuous efforts to maintain the stability of this API
  • Changes to the API require a major version bump.
  • Changes require agreement of the CLC
  • Modules appear in the exposed-modules Cabal section

• Internal: these modules are part of the internal implementation of base functions.

  • The GHC team makes no effort to maintain the stability of this API
  • Changes to the API require only a minor version bump.
  • Changes do not require agreement of the CLC
  • Appear in the exposed-modules Cabal section

As of today, all modules are either Hidden or External; the CLC policy is that the API of all exposed modules is subject to CLC review.

The main payload of this proposal is

• To establish the concept of an Internal module (exposed, but internal to GHC)
• To outline a process for incrementally moving modules from External to Internal.

2.1 Codifying the Internal vs External split

Our proposal is simply to declare whether a module is Internal or External, using some out-of-band mecanism like a publicly visible list.

However, future reorganizations (notably HF tech propoosal #47) might split base into two packages:

• ghc-base, all of whose exposed modules are Internal.
• base, all of whose exposed modules are External.

That would codify the distinction between Internal and External, which would be a Good Thing. But the burden of this proposal is simply to make that distinction in the first place, and start a dialogue about which modules belong in each category.

Incidentally, the Stability Haddock field of a module is not the same as Internal vs External distinction. A module could be External (i.e. designed for external callers), and yet experimental and not yet stable. That seems to be the intended purpose of the Stability field, although it is not well describe anywhere (please tell us there is a good specification).

We propose to document internal modules via a yet-to-be-named Haddock field.

2.2 Module by module summary

To make the discussion concrete, we have characterized each of the exposed modules in the GHC.* namespace along three axes:

• Proposed status: one of External or Internal
• Real-world usage: how many times each module is imported by end-user code. This data was gathered from Hackage.
• Stability risk: our subjective judgement about whether the module exposes internal details that may be subject to change in future compiler releases

These findings, along with the stability indicated by the modules' Stability Haddock field, are summarized in this spreadsheet. We then used these assessments to define an action plan (seen in the "Action" column) which will bring us closer to the goal of clearly delineating the stable interface of base. We do not intend to pursue this plan as one atomic change; rather, we intend for this plan to be an aspiration which we will iteratively approach over the course of the coming years, largely driven by the needs of the GHC developers.

The proposed actions fall into a few broad buckets:

• Internalize, which denotes the GHC developers' intent to in the future open a CLC proposal to move the module from External to Internal.
• Hide, which denotes the GHC developers' intent to in the future open a CLC proposal to remove the module from External to Hidden.
• Stabilize, where the module is declared to be stable and reaffirmed as External

In the sections below we will discuss some of the reasoning behind these proposed actions and draw attention to some open questions.

3. The question of GHC.Exts

Historically GHC.Exts has been the primary entry-point for users wanting access to all of the primitives that GHC exposes (e.g. primitive types, operations, and other magic). This widely-used module poses a conundrum since, while many of these details are quite stable (e.g. Int#), a few others truly are exposing implementation details which cannot be safely used in a GHC-version-agnostic way (e.g. mkApUpd0#, unpackClosure#, threadStatus#). There are at least two ways by which this might be addressed:

• Export only the subset of primops that we can stabilize (e.g. things like Int#, Weak#, newArray#, etc.) in GHC.Exts, leaving the rest to only be exposed via GHC.Prim (which should not be used by end-users), or
• Declare the entirety of GHC.Exts to be unstable and export the stable subset from another namespace (e.g. Word# and its operations could be exposed by GHC.Unboxed.Word)

4. Non-normative interfaces

Several interfaces exposed by base intentionally reflect internal details of GHC's implementation and, by their nature, should change to reflect changes in the underlying implementation. Here we call such interfaces "non-normative" as they are defined not by a specification of desired Haskell interfaces but rather by the system that they reflect.

One such module is GHC.Stats, which allows the user to reflect on various statistics about the operation of the runtime system. If the runtime system were to change (e.g. by adding a new phase of garbage collection), users would expect the module to change as well. For this reason, we mark such non-normative interfaces as "internal".

[tomjaguarpaw]

We propose to divide the modules of base into three groups

s/divide/classify/, perhaps? "divide" makes it sound like you're intending to carry out some operation on the source repo, but I think you mean just "publish a classifcation". In the subsequent section you use "declare", and explain that the "division" is done out-of-band. I think it might be wise to foreshadow this by changing "divide" to "classify", to avoid misdirecting the reader.

[phadej]

There are two axis of distinction between external and internal proposed:

1. Changes do or do not require agreement of the CLC
2. Changes to the API require a major or a minor version bump.

I stricly disagree with the second point. If change is breaking, than it requires major version bump. I don't care whether the module is marked as "implementation specific internal". If you expose it, than it's part of public API.
If you want to fix that issue, introduce a new package (ghc-base) and version it properly. Then if base doesn't re-export affected stuff from ghc-base, it can stay at the same major version.

And as far as I understand, it's the first point which causes most friction. GHC devs need to change something in GHC specific stuff, but have to go through CLC. And that I agree on, the clarification will make everyone's life eaiser.

But I repeat. Do not bundle these two unrelated considerations, they are not related. As an (occasional) user of low-level base stuff, I do care about them being versioned as PVP specifies. Otherwise I'd be forced to use tighter upper bounds on base. If you want to version GHC specifics separately, put them into separate package. Until then, version them properly.

Finally, if it's really internal modules (i.e. possible future ghc-base) maintenance approach is

The GHC team makes no effort to maintain the stability of this API

then whole thing is not worth doing. You probably want to word that differently.

[bgamari]

If change is breaking, than it requires major version bump.

As a matter of principle, I have no objection to continuing to bump the version of base when internal declarations change in a breaking way. This is one reason why I support #145.

However, until this happens (which could happen in time for GHC 9.8, if we move quickly) we will need to be pragmatic (as we have been in the past). Concretely, this means accepting the (small) possibility that breaking changes in internal modules may occur in minor version bumps. Of course, we will do due diligence to minimize the damage, but if there is a change needed in an internal declaration for a soundness fix (or something of similar severity) then we will make the change with a minor bump. Doing otherwise would merely cause undue harm to users for no perceivable benefit.

[phadej]

@bgamari

(as we have been in the past).

Can you provide a short summary of such breaking changes deep in base which should had technically been major bumps but hadn't? I don't remember any in recent years.

EDIT: I know that ghc library doesn't try very hard to be PVP compliant, as a lot of stuff is exported, and there are many fixes and changes in ghc itself between minor versions. But that's another story: it's not base.

[bgamari]

Can you provide a short summary of such breaking changes deep in base which should had technically been major bumps but hadn't? I don't remember any in recent years.

Such cases are (and should be) rare. Producing a comprehensive summary of these changes would be require a fair amount of effort not because there are numerous cases but because they tend to be small and subtle. On a cursory look through the last few releases I was unable to find a single one.

However, the principle stands: if we need to change, e.g., the type of GHC.Desugar.toAnnotationWrapper in a minor GHC (and therefore base) release to fix a soundness issue then we should do so without putting the rest of the ecosystem through the pain of a major bump in base version.

To reiterate, ideally the likes of GHC.Desugar wouldn't live in base at all, but sadly we don't yet live in a world where that is true.

[phadej]

To reiterate, ideally the likes of GHC.Desugar wouldn't live in base at all

Does that module needs to be public? If it's used in desugaring, can't GHC access symbols in non-public modules? At least TH can AFAIK.

EDIT: I understand that you try to be prepared for very unexpected things. But if you find it hard to find a convincing example, maybe it's a sign that asking CLC whether doing slight PVP sin isn't really not that bad?

EDIT2: Or if it's in non-CLC module, than just doing it and warning users in base changelog, and testing the Stackage that it actually doesn't break anyone.

[bgamari]

EDIT2: Or if it's in non-CLC module, than just doing it and warning users in base changelog, and testing the Stackage that it actually doesn't break anyone.

Yes, this is precisely the pragmatic approach that I was suggesting in the original comment.

[bgamari]

s/divide/classify/

Quite right. I've made this change.

[phadej]

EDIT2: Or if it's in non-CLC module, than just doing it and warning users in base changelog, and testing the Stackage that it actually doesn't break anyone.

Yes, this is precisely the pragmatic approach that I was suggesting in the original comment.

But that is different from what the proposal says. Especially as proposal says that GHC.Stats is internal, meaning that it might change in any minor GHC release. I don't agree on that, such soundness issue might need to wait for major GHC release until GHC.Stats is moved to ghc-base.

The bar to make a breaking change (in base, even "internal" modules) in minor GHC version should be set high, and that should be explicitly said in the proposal, not in the discussion.

[TeofilC]

I think one thing that might be good to draw out is how we will communicate the status of base modules going forward.

How would a first time contributor to base find out if the module they want to modify needs a CLC proposal or not?

Some possibilities:

• commit a version of Ben's spreadsheet into the CLC repo, link to it from the README and keep it up to date
• specify the modules status inside the module
• add a checkbox to the GHC MR template that asks if the user has checked if the change needs a CLC proposal

---

Another question is what the procedure will be for modules moving from one group to another, or for the introduction of new modules. For instance would adding a new internal module need CLC review? What about moving a module from internal to external?

[bgamari]

But that is different from what the proposal says. Especially as proposal says that GHC.Stats is internal, meaning that it might change in any minor GHC release. I don't agree on that, such soundness issue might need to wait for major GHC release until GHC.Stats is moved to ghc-base.

To be clear, this proposal explicitly /does not/ proposal to split out the internal modules of base into ghc-base. While I believe that doing so would be a reasonable step in the future, the goal of this proposal is merely to introduce a distinction between internal and external modules. We are currently proposing that the status of a module be captured in its documentation.

As long as internal modules remain in base we believe that it is reasonable to reserve the right to evolve internal modules outside of the PVP. However, we also agree that the PVP holds value and therefore will naturally take care when exercising this right.

I think one thing that might be good to draw out is how we will communicate the status of base modules going forward.

Yes, clear communication will be essential. I have added a small note suggesting that we introduce a new Haddock field to mark internal modules. Adding a mention to the MR template also seems like a reasonable idea.

Another question is what the procedure will be for modules moving from one group to another, or for the introduction of new modules. For instance would adding a new internal module need CLC review? What about moving a module from internal to external?

This proposal seeks to establish the /concept/ of an internal module and a roadmap for what modules we would l like to move to internal status in the future. While it would be great if the CLC would summarily accept a swath of these changes, we are willing to propose concrete changes in future, smaller CLC proposals if necessary.

[hasufell]

@bgamari are you suggesting this as a stop gap or a permanent solution?

[bgamari]

@bgamari are you suggesting this as a stop gap or a permanent solution?

I am suggesting this as a potentially-permanent solution. That being said, I do believe that #145 would be a considerable improvement and would love to see it adopted in the future..

[bgamari]

Do note that I have dropped the section entitled "where to place internal declarations" as it contained language from the editing process which was not intended to be in this proposal (namely the renaming of existing modules).

[NorfairKing]

I like the idea of code being marked as stable/unstable, however:
no docs > wrong docs > docs of indeterminate correctness

So this can only work if the "this module is stable" documentation is kept accurate somehow.
I'd be strongly in favour of this proposal if there was a mechanism to do so, and undecided if not.

The reason I'd be undecided without such a mechanism, is that we already have a stability field in haddock and it is widely ignored anyway.

(The other question is for some sort of linting about whether unstable modules are used in a project, but that's not in scope of this proposal afaict.)

[hasufell]

@bgamari are you suggesting this as a stop gap or a permanent solution?

I am suggesting this as a potentially-permanent solution. That being said, I do believe that #145 would be a considerable improvement and would love to see it adopted in the future..

I see.

In that case I'm leaning towards -1 on this proposal, since I believe violating PVP in the standard library sets a bad example (and as indicated in the PVP ticket, I'm also against formalizing it into PVP).

The only pragmatic way forward is the base split to me.

I'm starting to find it hard to follow all the inter-connected tickets about this, though.

[Bodigrim]

For instance, the GHC.Base.mapFB function is a necessary exposed part of the fusion framework for map but which GHC's authors never intended users to call.

@bgamari could you please elaborate on this? There is no necessity to expose functions used by the list fusion framework, and indeed no other function of it (e. g., zipWithFB, mapAccumLF, unwordsFB) is exposed. It seems mapFB is just an oversight and unfit to support your point.

[bgamari]

@bgamari could you please elaborate on this? There is no necessity to expose functions used by the list fusion framework, and indeed no other function of it (e. g., zipWithFB, mapAccumLF, unwordsFB) is exposed. It seems mapFB is just an oversight and unfit to support your point.

Yes, this is a fair point; mapFB and friends indeed need not be exposed as they are only used in rules defined in GHC.Base; this could be resolved with a proper export list. However, even without leaving GHC.Base there are other examples; for instance, maxInt exists to only serve other modules in base.

Looking beyond GHC.Base, there are the many exports of Data.Typeable.Internals (e.g. mkTrType), the array implementation in GHC.Arr, and the exports of GHC.Err.

[Bodigrim]

However, even without leaving GHC.Base there are other examples; for instance, maxInt exists to only serve other modules in base.

(Assuming for a moment that maxInt is potentially unstable)
The only place maxInt is used is GHC.Enum, it could have been defined there and not exposed outside. Even if it was used in multiple modules, it could have been defined in an other-module and never exposed to users.

More generally, my question is this. I understand that many fragile entities have been exposed from base. Is there a genuine reason for them to be publicly exposed (and more specifically - exposed from base and not from elsewhere), or is it a historical accident / negligence?

[bgamari]

More generally, my question is this. I understand that many fragile entities have been exposed from base. Is there a genuine reason for them to be publicly exposed (and more specifically - exposed from base and not from elsewhere), or is it a historical accident / negligence?

These cases are indeed largely due to historical accident. In many cases modules have been exposed via exposed-modules which should have rather been hidden in other-modules. These internal modules are often necessary either to serve to break module imports or to satisfy the needs of generated code and were not intended to be used by end-users.

[Bodigrim]

Is it difficult to keep modules which have been exposed due to historical accident stable (or at least avoid breaking changes)? And contain new work to other-modules?

[bgamari]

@Bodigrim it very much depends upon the case; this is essentially what I try to capture in the "stability risk" column of the spreadsheet. Anything assessed to be 0 or 1 will likely be quite easy to keep stable (and consequently I generally propose that these be "stabilized" except in cases where there is no evidence of external usage).

Modules assessed to be a higher stability risk would be harder. In some cases we propose that these be stabilized despite this (in particular, in cases where we find high degrees of dependence in the ecosystem). However, there are certainly a number of modules which we would prefer to hide.

Regardless, yes, we will need to be more careful to contain new work to other-modules in the future.

[Bodigrim]

What does Stability risk grade 3 stand for?

[bgamari]

These assessments are a qualitative, fairly subjective grading. Grading 3 essentially corresponds to things that not only are likely to change but that we would also active discourage users from relying on.

[chshersh]

I like the proposal overall. Here's my constructive feedback.

Incidentally, the Stability Haddock field of a module is not the same as Internal vs External distinction. A module could be External (i.e. designed for external callers), and yet experimental and not yet stable. That seems to be the intended purpose of the Stability field, although it is not well describe anywhere (please tell us there is a good specification).

We propose to document internal modules via a yet-to-be-named Haddock field.

I don't think we need a new field. I suggest reusing Stability and introduce new values if necessary. This is only a matter of documentation. Currently, the relevant section in Haddock docs doesn't specify the meaning of stable, experimental, provisional and internal.

I suggest opening a PR to Haddock with the description of these four fields as the immediate next step.

The proposed actions fall into a few broad buckets:

• Internalize, which denotes the GHC developers' intent to in the future open a CLC proposal to move the module from External to Internal.
• Hide, which denotes the GHC developers' intent to in the future open a CLC proposal to remove the module from External to Hidden.

Currently, the Base stability spreadsheet wants to either hide or internalize a total of 38 modules. Creating 38 CLC proposals (one per each module) sounds like huge amount of work.

I suggest to at least split this process into batches. In fact, I believe we can agree on most of the modules in this CLC proposa directly.

The question of GHC.Exts

GHC.Exts sounds like PITA. My recommendation would be to:

• Stop reexporting new stuff from GHC.Exts immediately today
• Deprecate reexports of unstable functions via a separate CLC proposal (was there GHC work in progress on the deprecating reexports feature?)
• Incrementally refine this module when the need arises

To make the discussion concrete, we have characterized each of the exposed modules in the GHC.* namespace along three axes:

My general view:

• If a spreadsheet indicates 0 usages for a module, I'm okay with hiding this module immediately (as the result of this proposal)

I would like to provide my comments to all other individual modules:

• GHC.Arr -> hide: The array package depends on base and imports GHC.Arr so hiding GHC.Arr won't work
• GHC.Base -> hide: The spreadsheet indicates 353 usages, so hiding this module is too big of breakage to the point it's not feasible
• GHC.Exception.Type -> hide: Has 2 usages, so let's fix them before hiding
• GHC.Fingerprint.Type -> hide: 18 usages is a bit much to hide this module
• GHC.InfoProv -> stabilize: This module appeared in base-4.18, and just recently there was a proposal to change one of the types. I think it's too early to stabilize this module.
• GHC.Pack -> hide: It has 10 usages but it's already deprecated. I suggest to wait a while and allow users to migrate away before hiding.
• GHC.Stack.CloneStack -> stabilize: 0 usages. This really sounds like a GHC internal thing so I'd suggest to keep it internal.
• GHC.Show -> internalize, export Show from new Data.Show module: Sounds like a good plan, and will require a separate CLC proposal
• GHC.TopHandler -> hide: 6 usages is not a lot but I would suggest to help users migrate away before hiding

[tomjaguarpaw]

Currently my approach is to judge every proposal on a case-by-case basis taking into account the precedent of previous decisions, so I don't expect anything to change from my point of view, vis-a-vis the decision-making process. That is to say, if this proposal is passed it sets the precedent that CLC members ought to consider those modules as less stable, and I'm happy to take that precedent as a strong indicator of how I ought to make decisions.

[chshersh]

Sure, but in order for this arrangement to be working in practice, CLC members should indicate whether they're ok with lowering their standards for the modules in question or whether they intend to treat it as strict as the rest.

I'm okay with lowering my standards if it helps Haskell and GHC users to be more productive.

In fact, I already didn't have high requirements for changes in the GHC-internal (as I see) modules. Formalizing this convention and having a list of such modules explicit and uniform for everyone sounds like a step forward.

[Bodigrim]

Dear CLC members, let's vote on the proposal to add the following disclaimer

The API of this module is unstable and not meant to be consumed by general public. If you absolutely must depend on it, make sure to use a tight upper bound, e. g., base < 4.X, not just base < 5, because the interface can change rapidly without much warning.

to these modules:

• GHC.Arr
• GHC.Conc
• GHC.Conc.IO
• GHC.Encoding.UTF8
• GHC.Exception
• GHC.Exception.Type
• GHC.ExecutionStack
• GHC.Fingerprint.Type
• GHC.InfoProv
• GHC.IO
• GHC.IO.Buffer
• GHC.IO.Device
• GHC.IO.Encoding
• GHC.IO.Exception
• GHC.IO.Handle.Text
• GHC.RTS.Flags
• GHC.Stack.Types
• GHC.Stats
• GHC.TopHandler
• GHC.TypeLits.Internal
• GHC.TypeNats.Internal
• GHC.Desugar
• GHC.Event.TimeOut
• GHC.ExecutionStack.Internal
• GHC.Float.RealFracMethods
• GHC.GHCi
• GHC.GHCi.Helpers
• GHC.IO.Handle.Internals
• GHC.IO.Handle.Types
• GHC.IO.SubSystem
• GHC.IOPort
• System.Posix.Internals

(I'd usually ask proposers to put up an MR, but in this case it seems better to hold a vote first. If there is a delay, I'll prepare an MR myself)

@tomjaguarpaw @chshersh @hasufell @mixphix @angerman @parsonsmatt

---

+1 from me.

[mixphix]

+1

[tomjaguarpaw]

+1

[hasufell]

+1

[chshersh]

+1

[parsonsmatt]

+1

[Bodigrim]

Thanks all, 6 votes in favour are enough to approve.

@bgamari please go ahead with an MR (and leave a link here for the reference).

[Bodigrim]

@bgamari this is a gentle reminder to prepare an MR implementing the proposal.

[bgamari]

@Bodigrim thanks for the ping. I have not forgotten; to the contrary, we have been hard at work getting the groundwork in place to implement this and the rest of our proposal.

[chreekat]

Could someone perhaps link to (or copy outright) the content of #146 (comment) into the issue description, since it is what actually ended up being voted on for this issue? :) This thread goes for a wild ride that perhaps most future readers don't need to go through. In the end the proposed change was literally just documentation, if I'm not mistaken, while many other things of far greater import were argued for and against, as everyone gradually came to understand each other over time.

[simonpj]

@bgamari this is a gentle reminder to prepare an MR implementing the proposal.

@Bodigrim thanks for the ping. I have not forgotten; to the contrary, we have been hard at work getting the groundwork in place to implement this and the rest of our proposal.

Could someone perhaps link to (or copy outright) the content of #146 (comment) into the issue description, since it is what actually ended up being voted on for this issue?

I think we are waiting for @bgamari to formulate a concrete proposal (indeed along the lines of the comment you cite).

Ben, it may well be better to start a new issue, to avoid the "wild ride".

[Bodigrim]

I think we are waiting for @bgamari to formulate a concrete proposal (indeed along the lines of the comment you cite).

Ben, it may well be better to start a new issue, to avoid the "wild ride".

I'm somewhat confused. There was a very specific proposal voted on in this thread; it's an MR which is missing, but no reason to start a new issue for it.

[bgamari]

Yes, I am also rather confused. My understanding is that the proposal as-written in the summary is accepted. The operative changes are described in #146 (comment) and #146 (comment).

However, I now realize that I did forget to notify the CLC of the implementation of the proposal, which was carried out in !10422; apologies for that. This MR introduced Data.Enum and added the appropriate disclaimers. Over time I expect that we will open further proposals to deprecate various exports of the modules covered by the proposal.

[Bodigrim]

Ben, now I'm confused even more. The approved changes we voted on are in the second comment you linked only, not the first one. I thought I asked both you and Simon, and repeated again the precise scope in the call for voting. If you want to introduce Data.Enum or anything else, I'd suggest raising a separate proposal.

(I've been very slow recently and accumulated a huge backlog of work here, and I'm sorry for this. I hope to clear up things during the upcoming months)

[bgamari]

Ben, now I'm confused even more. The approved changes we voted on are in the second comment you linked only, not the first one. I thought I asked both you and Simon, and repeated again the precise scope in the call for voting. If you want to introduce Data.Enum or anything else, I'd suggest raising a separate proposal.

Apologies, this was my miscommunication; I floated the idea in a comment but failed to express it in the proposal. I have raised a separate proposal for Data.Enum (#208).

(I've been very slow recently and accumulated a huge backlog of work here, and I'm sorry for this. I hope to clear up things during the upcoming months)

Quite alright. You have a lot on your plate and yet manage it admirably.

[chreekat]

Thanks for the clarification, @Bodigrim and @bgamari !