Policy: Discuss extras repository for additional languages

[joshgoebel]

I see we have a lot of requests for new languages in the PRs... if the issue is time/maintenance over time, etc... might we perhaps consider an "extra" respository or something with community/unsupported syntaxes? That way the criteria to be approved could be lessened a little and obscure languages that might not really make sense in Highlight.js proper could still have a home?

Or is the idea that eventually we'll get to them?

[joshgoebel]

Are we still moving languages into repositories? Is that ALL the languges, or only "additional" ones?

[marcoscaceres]

Only new ones because then we can make submitters the repo maintainers.

[joshgoebel]

So what is the policy on new (or very old) PRs already in the queue? Is there some cut off date or pretty much the answer now is always "separate repo"?

[marcoscaceres]

Yep, separate repo.

[jf990]

The separate language repos concept addresses a maintenance problem that has caused consternation here since @isagalaev slowed down the regular maintenance. I think the separate repo idea is going to address that but we need to discuss the many problems it introduces. We should decide how to handle these things because they change the way this library has been managed since its inception.

1. Discovery. Available languages and how to use them are documented on highlightjs.org. By putting new languages in separate repos we are effectively invalidating this entire page. No longer could we have a single source of truth as to what languages are available and how to get them. Even worse, old languages that were part of the original highlight js will be handled and documented differently than the new ones. this will make the library difficult to use and documentation will be all over the place as different language authors choose their own way of doing things without the oversight of the maintainers. We should figure out and document some minimum standards for language contributions. (To do this right) we should even move all the existing languages into this new format such that all languages are handled the same making using the library consistent. We should also come up with a meta data file that each language repo is responsible to maintain so that we can automate some aspects of the discovery, testing, detection, and packaging.

2. Auto-detection. I'm not sure we are going to easily have language auto-detection work as well as it does now. with the separate repos there's no easy way to run the test against all the other languages and determine if relevance is working.

3. Testing. Current testing methodology is well defined and there is a process to handle it. This method is no longer valid and individual repos are on their own. So far with the new languages definitions we have not established any requirement for testing, and we see that some do and some don't.

4. Packaging and deployment. We probably can no longer support a page like Getting highlight.js, or not for the new languages. This may or may not be a real problem for some as packagers and modern workflows may reduce this requirement, but some developers are still loading a custom package in a script tag and would be required to completely change their method if they wanted to include a individual language. It would also be not so nice to offer that page and not include the newer languages, so we have some thinking to do there.

5. Quality. Since we are now deferring individual language maintenance to the individual authors (or those interested in contributing) and not the central highlightjs maintainers, going forward the quality and maintenance of languages will be indeterminate at best. Our current process at least has the maintainers as the gatekeepers making sure we don't include poorly implemented, undocumented, or buggy language contributions. But the new way will remove that oversight. For a community driven project I guess this is the expectation but in the past this repo has been very well maintained, and I think that is part of the reason it gained such traction and popularity.

Most of this is solvable with new documentation on the language contributors page and some new development such as a language registry and supporting process scripts like test and build to support that. But it's all a lot of work.

I still haven't been able to figure out how to update the documentation. Have we documented how stuff in docs gets built and deployed to highlightjs.org and highlightjs.readthedocs.io?

BTW, really nice work @yyyc514 going through all those issues 🚀

[joshgoebel]

. I'm not sure we are going to easily have language auto-detection work as well as it does now. with the separate repos there's no easy way to run the test against all the other languages and determine if relevance is working.

Well, the "test it" part is just a tooling problem but I think logically this will prove impossible with the infinite range of possibilities as languages grow and grow, yes.

Testing

I think to be "semi-official" or included in some type of global list that there should be some minimal amount of specs that a syntax is required to pass.

But obviously if someone just wants to rip a language down from somewhere and use it then we can't stop them. We do have power to decide what gets hosted at highlightjs org though, so that's something.

It would also be not so nice to offer that page and not include the newer languages, so we have some thinking to do there.

Well, I wasn't going to say it publicly, but I guess now I will. I can't speak for everyone but I can't imagine this ban on new languages in core is ABSOLUTE. It's to prevent a proliferation of 100 tiny languages stagnating that no one has time or inclination to maintain (or to baby sit PRs, tests, QA, etc). If the next Swift comes around and 50% of the world is writing code in it, I imagine we'd consider adding it to core and someone would make time to maintain it.

Although if we figure out this whole "separate repo" thing perhaps eventually none of the languages will be in core... but seems that's a bit far away at the moment.

Our current process at least has the maintainers as the gatekeepers making sure we don't include poorly implemented, undocumented, or buggy language contributions. But the new way will remove that oversight.

Yes, this is a for sure concern and why I think there should be some gateway between "one of the core contributors has read this, or agreed it passes "reasonable" specs and "who knows I just found this laying around somewhere". It would be very bad if someone installs a shiny new "Pancakes 1.0" syntax that locks up their website and blames it instead on Highlight.js.

I still haven't been able to figure out how to update the documentation. Have we documented how stuff in docs gets built and deployed to highlightjs.org and highlightjs.readthedocs.io?

No idea. Someone who knows how needs to find time to write up something ROUGH... and then we need to find people who have the time an inclination to help keep docs updated. They could iterate on the rough docs and push them forward.

BTW, really nice work @yyyc514 going through all those issues 🚀

Thanks.

[jf990]

@yyyc514 I had offered to pitch in on the docs in some other issue here, and in particular to help rewrite the contribute a new language guide. I had worked on redoing the languages I help maintain as separate repos to work out the way to do it.

I setup a github template project, we can review this effort and see if it's on the right track. It sets up a template project with a unit test to get started with a new language.

https://github.com/jf990/highlightjs-language-template

[joshgoebel]

@jf990 See my thoughts on this thread regarding auto-detection:

#1213

I actually think there are some pretty great ideas (or the beginnings of ideas there). In this context instead of saying "Yes, all 10,000 languages from all maintainers have NO conflicts!" We'd run the tests and then say:

Hey, you, maintainer of "Pancake 1". Your language is 95% "sticky", it thinks almost any language is Pancake... our recommended threshold is Y... you need to tune your relevancy scores to prevent false positives against other languages.

Need a better word than sticky, lol.

And then we'd have a metric we could use for including in a master list, including on the "main website" for build packs, etc...

[jaredlll08]

So I have been looking at a few of the issues over the past few days, and feel I can maybe provide some outsider input.

Before I start, I want to make it clear, I am not a proper JavaScript Developer, I primarily work in other back-end languages, so my approach may be naive or going against best practices, but I do believe it will solve most issues.

The approach is very similar to what GitHub does with their Linguist library, and that is using Git Submodules.

I have made a proof of concept for everything I am going to say below, which can be found here, like I said above, I'm not that well versed in JS build tools (and it is currently 5AM), so I did a dirty hack to get it to work, in reality it will either need to be all or nothing, with all languages following the same format (being in a folder with the language name for example), but having a standard format for languages is good in my opinion.

Things I changed in the proof of concept:

• Languages are now loaded from a folder, so src/languages/$LANGUAGE_NAME/*.js
• Reading snippets uses the name of the folder the language is in(so $LANGUAGE_NAME) instead of the js file (could be changed, I did it because my file was called index.js, you could enforce a proper name when making the repo)

Right now, based on issues I have read on this repo, if someone wants a new language, they make an issue / PR on this repo, then a @highlightjs member creates a repo for their language, and adds the original author of the issue / PR to that repo for their language to live in.

As @jf990 said, while it helps with maintenance, it comes with a few drawbacks, I believe Git Submodules can fix most of those, like so:

1. Discovery

This is the whole reason why I am even commenting here, I made support for a language, and now trying to get products to use it is an up hill battle, most developers feel that I should try and get my language to be supported in HighlightJS itself, which right now, isn't an option.

So the solution, since manual work is already required when someone wants to add language support (making a repo for them and add them as a collaborator), then it shouldn't be an issue to make a commit to this repo, adding that newly created repo as a submodule, like I do in this commit of the PoC.

This will only add the submodule to the src/languages folder, so I do not have a solution for files in the test folder, I can look more into it if this solution is something that will actually be considered and would work for this project.

So with the 3rd party languages now in the languages folder, it is treated exactly the same as a "first party" language, so it is built when running node tools/build and shows up on the build/demo/index.html, and I believe, tests are still ran on it, I don't see a reason why they wouldn't be.

2. Auto-detection

See above about third party languages being treated as first party, so this would no longer be an issue.

3. Testing

Like I said above, I don't see why tests wouldn't run on these third party languages, besides the issue of getting other files into the test directory.

4. Packaging and deployment

Once again this is all handled because the files are physically there when the commands are ran, I'm not sure how the Getting highlight.js page is generated, but if it is generated based on the src/languages folder, it should be trivial to move to the new src/languages/$LANGUAGE_NAME format.

The one thing that would need doing, or at least would be a quality of life feature, would be having the build script update all the submodules to their latest commit, but this can be done manually in the case of broken commits on submodules, or just forcing a submodule to use a specific commit.

5. Quality

I have no solution for this, the only thing that comes to mind is implementing a set of requirements and guidelines for new languages, for example, Github Linguist (example), they require the language to be used in "hundreds of repositories", for them, being Github, that makes sense, since the PR would affect those repositories, so for HighlightJS, it gets a bit tricky, you could use the same metric as Github, and that way you could help ensure that:

1. A language is used (I don't know how highlightJS members feel about this, the docs do say any and all languages are allowed).
2. There are open source developers using the language (if the maintainer of the highlightJS language support dissapears, it is possible a new maintainer who has experience with the language could step up).

Or highlightJS could work on a different metric, I will admit that this doesn't seem like an easy issue to solve.

As for the documentation, I have used ReadTheDocs in the past and have some experience with it, so I am happy to help figure out how it works, and from there document any of the changes I listed above (if implemented) to help ensure that everyone knows what the new protocol is.

I hope this all makes sense, I am happy to go into more detail on git submodules if need be, or even brainstorm a different solution (possibly having the build script traverse the highlightJS organization and pull the languages from there, that way there are no submodules).

[joshgoebel]

From another thread (I'm replying to @egor-rogov) (#1829):

Egor: This language is already in the core. Why should we move it away?

Well, already in or not is a very weird (meaningless?) metric (IMHO) to decide whether that's where they BELONG. I thought the whole idea of not letting more languages in [to core] had to do with developer time/maintenance/responsibility/who is in the best position to maintain the language long-term, etc... so surely the right way to think about existing languages ALSO is how they fare on those exact same metrics...

This "already in core" vs "sorry, you just missed the cut-off!" is a VERY weird and arbitrary line.

[joshgoebel]

Not a fan of git submodules, though I haven't worked with them in years. It's possible they have improved. Back then all I heard was whining about how annoying they were.

I do see the advantage of "just works" (other than for tests, which you didn't go into in great detail)... but I don't think the paths is the hard part...

Having a languages.toml or languages.json file that anyone could contribute too and a smart build tool (doesn't have to be that smart) could accomplish the same thing.

Our build pipeline is crazy old and needs replacing anyways - so keeping it "as-is" isn't a priority.