chore(ci): add individual publishing of packages MONGOSH-1871 #2289

gagik · 2024-12-10T11:17:29Z

This adds the ability to individually publish packages in mongosh. Also, notably, all packages's version inside package.json will now match the latest released version and update accordingly upon release.

Structure

Package Bumping

With this change, there are 2 types of package bumping depending on what package it is:

mongosh release-related packages bump
mongosh and @mongosh/cli-repl are bumped only at mongosh release and have their versions synced.
This gets triggered only during a mongosh release.
npm auxiliary packages bump
All other packages are bumped independently of mongosh using bump-packages from monorepo-tools. Each package can have divergent versions as a result.
This can be triggered anytime using GitHub actions which will create PR for bump package versions, merging of which will cause CI to publish the packages.

Individual Package Release

You can release an individual package by running a bump packages GitHub Action and merging the created PR.

mongosh Release

A mongosh release will cause both an npm package bump and a mongosh release bump before starting the release.

lerna.json

packages/snippet-manager/package.json

packages/build/src/npm-packages/publish.ts

packages/build/src/release.ts

lerna.json

gagik · 2024-12-20T17:10:25Z

@addaleax Didn't get to spend too much time on this before my holiday with some VSCode and coverage test back and forths. Would be happy to jump back into it in a week but if it'd be better to finish this earlier, feel free to pick it up 🙂 I have some WIP files that I may polish up and push at a later time also though not sure how useful they are; will see.

gagik · 2024-12-31T11:28:08Z

.evergreen/evergreen.yml.in

@@ -917,6 +917,9 @@ functions:
          {
          export NODE_JS_VERSION=${node_js_version}
          source .evergreen/setup-env.sh
+          npm run evergreen-release bump
+          git add .
+          git commit --no-allow-empty -m "chore(release): bump to prepare for mongosh release"


The idea here is that this will bump both "independent" and mongosh packages before the release draft and push it to main, so our branch is consistent with what we're releasing.
It might also make sense to only persist this after the release is published.

I am assuming this should work? Though not sure if the evergreen git setup has ability to push to main like this?

gagik · 2024-12-31T11:33:02Z

packages/build/src/npm-packages/constants.ts

+export const MONGOSH_RELEASE_PACKAGES = ['mongosh', '@mongosh/cli-repl'];
+
+/** Packages which always get ignored when doing a release or bump */
+export const IGNORE_BUMP_PACKAGES = [


ignoring these since these are private, unless it'd make sense to publish anyways?

gagik · 2024-12-31T11:34:48Z

packages/arg-parser/package.json

@@ -1,6 +1,6 @@
 {
  "name": "@mongosh/arg-parser",
-  "version": "0.0.0-dev.0",
+  "version": "2.3.7",


these were all set to 2.3.7 manually to get the current repository consistent with the latest release

.evergreen/install-npm-deps.sh

.github/workflows/bump-packages.yml

.github/workflows/publish-auxiliary-packages.yml

packages/build/src/config/config.ts

packages/build/src/npm-packages/publish.ts

packages/build/src/npm-packages/bump.ts

packages/build/src/npm-packages/publish.ts

addaleax · 2025-01-06T15:28:42Z

packages/shell-api/package.json

@@ -1,6 +1,6 @@
 {


I don't know where we need to to leave this comment, so I'll leave it here – in shell-api.ts, we fetch the current mongosh version via require('../package.json').version. We should replace that with a new method on EvaluationListener that we then call from there, so that the mongosh CLI can provide the correct version to the shell-api package (and probably also add an e2e test that mongosh --eval 'version()' and mongosh --version print the same output).

Co-authored-by: Anna Henningsen <[email protected]>

addaleax · 2025-01-10T08:17:36Z

packages/e2e-tests/test/e2e.spec.ts

-      shell.assertContainsOutput(expected);
+      shell.assertNoErrors();
+      shell.assertContainsOutput(expectedPackageVersion);
+      shell.assertContainsOutput(expectedVersionFromRepl);


Not a huge deal, but I think it would be a tad cleaner to use really compare version() against the output of mongosh --version instead of adding more explicit source dependencies on the cli-repl package contents (and even internals) here.

addaleax · 2025-01-10T08:18:24Z

packages/shell-api/src/shell-api.ts

-    const version = require('../package.json').version;
+    const version = this._instanceState.evaluationListener.version;
+    if (!version) {
+      throw new MongoshInternalError('mongosh version not known');


Scripts don't expect version() to throw, I don't think we can do this here

Hm fair, I expect this to hypothetically never happen, just handled it this way to respect its possibly undefined type, would it be better to ignore it (I guess then returning undefined) or some placeholder i.e. 0.0.0?

Hm yeah. If we do want to rely on this property always present, let's make it an actual required property and ensure that a value gets provided in Compass and the java-shell implementation, otherwise a placeholder like 0.0.0 is probably a good idea

But thinking about it more, especially in conjunction with the concern about the other non-CLI products, maybe we should just find an easy way to use the version of the cli-repl package here, that's what most people will be thinking of as the "mongosh version" anyway (even though then multiple shell-api versions can refer to the same CLI version)

I am realizing that expecting a preset version on EvaluationListener does mean browser-repl, etc. need to define it also; or should it just be a required field in MongoshNodeRepl?

Would it be a realistic thing to depend on a programmatically set environment variable for MONGOSH_VERSION and/or a global variable for browser-based contexts? Not sure how it'd work in a Java / IntelliJ context.

does mean browser-repl, etc. need to define it also

Yeah, that's kind of what I'm getting at, right now version() returns a meaningful value in e.g. Compass, and we probably want to keep that behavior around

Would it be a realistic thing to depend on a programmatically set environment variable for MONGOSH_VERSION and/or a global variable for browser-based contexts? Not sure how it'd work in a Java / IntelliJ context.

Well, the EvaluationListener interface is our interface for getting this type of information into the executable, so we should be using that.

But yeah, ultimately I think people would want to see a version that correlates with the mongosh CLI version that corresponds to the current version of the shell-api package. This could also be something like the compile step in shell-api writing down the cli-repl package version somewhere inside of the package

Should it be in the compile step or as part of the mongosh release process since that's when that value would change anyways?
Along with the package.json-s bump we could update a constant inside shell-api and bump it along with that to main. Then there won't be need for any changes to EvaluationListener, right?

That would also work, yes. I figured compile would be easier since it requires maintaining less state, but it does have the downside of adding a new generated file to the repository, so no strong feelings about which solution we go with (but we should do either of them 🙂)

addaleax · 2025-01-10T08:19:10Z

packages/node-runtime-worker-thread/src/worker-thread-evaluation-listener.ts

@@ -56,6 +59,7 @@ export class WorkerThreadEvaluationListener {
            (Promise.resolve() as Promise<never>)
          );
        },
+        version: workerRuntime.evaluationListener?.version,


How do we provide this value in non-CLI products (vscode/Compass/IntelliJ)? I do think that's something we're going to have to figure out, and unfortunately I don't have an obvious answer for it myself

if I understand correctly, making it a generated constant inside shell-api essentially handles that right?

…idual-publishing

nirinchev

Looks reasonable, I have mostly very minor comments. One thing I was wondering is - this looks like it'll tag all the auxiliary packages released, but won't create a github release for those. Do we want to also create releases, or do we think it'll get too noisy if we did it for every package?

.evergreen/evergreen.yml.in

.github/workflows/bump-auxiliary-packages.yml

.github/workflows/publish-auxiliary-packages.yml

nirinchev · 2025-01-10T16:16:10Z

.github/workflows/publish-auxiliary-packages.yml

+    - name: Create Github App Token
+      uses: mongodb-js/devtools-shared/actions/setup-bot-token@main
+      id: app-token
+      with:
+        app-id: ${{ vars.DEVTOOLS_BOT_APP_ID }}
+        private-key: ${{ secrets.DEVTOOLS_BOT_PRIVATE_KEY }}


Why do we need this? Is it because publish-auxiliary pushes some tags that we want to configure the git user for?

I wasn't sure if the git workflow can just push anything at all. To be honest the relationship between the bot token and just an action is a bit unclear to me, might be nice to have a quick explainer on that sometime

.github/workflows/publish-auxiliary-packages.yml

Co-authored-by: Nikola Irinchev <[email protected]>

gagik · 2025-01-13T08:57:17Z

@nirinchev I think it’d be better to keep releases exclusive to mongosh to keep a clean timeline of releases for users to look into; which is also how we do it in Compass. The tags + PRs should be nice enough for our internal reference I’d think. Though open to adding some other form cc: @addaleax

…idual-publishing

gagik force-pushed the gagik/individual-publishing branch from 879eb0f to 40d6e71 Compare December 10, 2024 11:17

WIP: individual publishing of packages

a2f246b

gagik force-pushed the gagik/individual-publishing branch from f48cc5c to a2f246b Compare December 18, 2024 12:05

gagik commented Dec 18, 2024

View reviewed changes

lerna.json Show resolved Hide resolved

use 2.3.7

417e1f4

gagik commented Dec 18, 2024

View reviewed changes

lerna.json Show resolved Hide resolved

update package-lock.json and packages

447e8d1

addaleax reviewed Dec 18, 2024

View reviewed changes

packages/snippet-manager/package.json Outdated Show resolved Hide resolved

packages/build/src/npm-packages/publish.ts Show resolved Hide resolved

packages/build/src/release.ts Show resolved Hide resolved

lerna.json Show resolved Hide resolved

gagik added 3 commits December 19, 2024 11:08

restore changes

4c97468

ensure compatibility with old bump

8c9c214

wip

03ba373

gagik force-pushed the gagik/individual-publishing branch from 8c9c214 to 41bc10a Compare December 20, 2024 14:55

switch to bump script

ad630dc

gagik force-pushed the gagik/individual-publishing branch from 41bc10a to ad630dc Compare December 30, 2024 11:48

gagik added 8 commits December 30, 2024 19:15

Merge branch 'main' into gagik/individual-publishing

28bfce7

fix package bumping

45469f9

check if bump is necessary for installs

be919e0

remove old tests

43c551d

add publish-packages

71aaa79

remove package.json changes

ceb13e7

add ignore list

42d19ea

set non-release package.json to 2.3.7

bad9d59

gagik changed the title ~~WIP: individual publishing of packages~~ chore(ci): add individual publishing of packages MONGOSH-1871 Dec 31, 2024

gagik added 2 commits December 31, 2024 12:24

add bumping commit to release draft

193aa09

set all to 2.3.7

a19f408

gagik commented Dec 31, 2024

View reviewed changes

.evergreen/install-npm-deps.sh Show resolved Hide resolved

gagik marked this pull request as ready for review December 31, 2024 11:38

gagik commented Dec 31, 2024

View reviewed changes

.github/workflows/bump-packages.yml Outdated Show resolved Hide resolved

gagik force-pushed the gagik/individual-publishing branch from 3c557da to 68e3e62 Compare January 6, 2025 14:04

addaleax reviewed Jan 6, 2025

View reviewed changes

gagik and others added 6 commits January 6, 2025 16:40

Update .github/workflows/publish-auxiliary-packages.yml

ae5b5d4

Co-authored-by: Anna Henningsen <[email protected]>

re-add bump

fc9a406

Apply most changes from review

72f04bd

use npx and use tags in all cases

69baf66

use monorepo tools as dependency

72cde48

remove bump, use version

4da9db1

addaleax reviewed Jan 10, 2025

View reviewed changes

add and fix publish tests

2b912d9

gagik force-pushed the gagik/individual-publishing branch 2 times, most recently from a9d2422 to ed9ffa1 Compare January 10, 2025 13:08

use a shell-api constant and switch to 2.3.8

e2ceaa5

gagik force-pushed the gagik/individual-publishing branch from ed9ffa1 to e2ceaa5 Compare January 10, 2025 13:13

gagik added 4 commits January 10, 2025 14:17

Merge branch 'main' of github.com:mongodb-js/mongosh into gagik/indiv…

c736e32

…idual-publishing

set all packages to 2.3.8

e7d0a8c

revert to dev version

fa253de

use 2.3.8

9c4c358

gagik force-pushed the gagik/individual-publishing branch from de993f9 to 9c4c358 Compare January 10, 2025 13:36

fix tests and match with mongosh --version

bb111a5

nirinchev approved these changes Jan 10, 2025

View reviewed changes

gagik and others added 2 commits January 10, 2025 17:38

add basic bump tests, update evergreen config, and review fixes

4d4de5a

Update .github/workflows/publish-auxiliary-packages.yml

755e87a

Co-authored-by: Nikola Irinchev <[email protected]>

gagik added 3 commits January 13, 2025 12:30

use token from bot

f0e985b

Merge branch 'main' of github.com:mongodb-js/mongosh into gagik/indiv…

5513ad7

…idual-publishing

minor change to test

28b8c37

addaleax approved these changes Jan 14, 2025

View reviewed changes

gagik merged commit 435d4ce into main Jan 14, 2025
12 of 13 checks passed

gagik deleted the gagik/individual-publishing branch January 14, 2025 12:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(ci): add individual publishing of packages MONGOSH-1871 #2289

chore(ci): add individual publishing of packages MONGOSH-1871 #2289

gagik commented Dec 10, 2024 •

edited

Loading

gagik commented Dec 20, 2024

gagik Dec 31, 2024

gagik Dec 31, 2024

gagik Dec 31, 2024

addaleax Jan 6, 2025

addaleax Jan 10, 2025

addaleax Jan 10, 2025

gagik Jan 10, 2025

addaleax Jan 10, 2025

gagik Jan 10, 2025 •

edited

Loading

addaleax Jan 10, 2025

gagik Jan 10, 2025 •

edited

Loading

addaleax Jan 14, 2025

addaleax Jan 10, 2025

gagik Jan 13, 2025

nirinchev left a comment

nirinchev Jan 10, 2025

gagik Jan 10, 2025

gagik commented Jan 13, 2025

chore(ci): add individual publishing of packages MONGOSH-1871 #2289

chore(ci): add individual publishing of packages MONGOSH-1871 #2289

Conversation

gagik commented Dec 10, 2024 • edited Loading

Structure

Package Bumping

Individual Package Release

mongosh Release

gagik commented Dec 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gagik Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gagik Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nirinchev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gagik commented Jan 13, 2025

gagik commented Dec 10, 2024 •

edited

Loading

gagik Jan 10, 2025 •

edited

Loading

gagik Jan 10, 2025 •

edited

Loading