feat(compass-indexes): add index build progress COMPASS-9495 #7200

nbbeeken · 2025-08-15T04:15:25Z

Description

Add a percentage to the index build row in the table if it is in progress.

https://jira.mongodb.org/browse/COMPASS-9495

Checklist

New tests and/or benchmarks are included
Documentation is changed or added
If this change updates the UI, screenshots/videos are added and a design review is requested
I have signed the MongoDB Contributor License Agreement (https://www.mongodb.com/legal/contributor-agreement)

Motivation and Context

Bugfix
New feature - parity with legacy DE
Dependency update
Misc

Open Questions

For reviewers:

Did I make good use of hooks by writing useIndexProgress?
- No 😅
The getInProgressIndexInfo used to just list stub indexes because even in progress ones would be listed as "ready" while they continued to build on the server. We now will list "real" indexes via this function. Is that the right approach?

Dependents

Types of changes

Backport Needed
Patch (non-breaking change which fixes an issue)
Minor (non-breaking change which adds functionality)
Major (fix or feature that would cause existing functionality to change)

Screen.Recording.2025-08-15.at.2.29.25.PM.mov

index_finish.mov

Copilot

Pull Request Overview

This PR adds index build progress tracking to the MongoDB Compass indexes table, showing a percentage indicator when indexes are being built. The implementation introduces a new hook for polling index progress via currentOp command and updates the UI to display build progress and a spinner for in-progress indexes.

Adds useIndexProgress hook for tracking index build progress via polling
Updates Redux state to track progress percentage for in-progress indexes
Modifies UI components to display progress indicators and spinners for building indexes

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`test/setup-store.ts`	Adds mock `currentOp` method to test data service
`src/stores/store.ts`	Extends data service type to include `currentOp` for progress tracking
`src/modules/regular-indexes.ts`	Adds progress tracking logic, new actions, and `getIndexesProgress` thunk
`src/modules/regular-indexes.spec.ts`	Adds tests for index progress tracking functionality
`src/hooks/use-index-progress.ts`	New custom hook for managing index build progress polling
`src/components/regular-indexes-table/regular-indexes-table.tsx`	Integrates progress hook and updates index filtering logic
`src/components/regular-indexes-table/regular-indexes-table.spec.tsx`	Updates tests to mock the progress hook
`src/components/regular-indexes-table/regular-index-actions.tsx`	Adds progress display UI with spinner and percentage

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

packages/compass-indexes/src/hooks/use-index-progress.ts

packages/compass-indexes/src/components/regular-indexes-table/regular-index-actions.tsx

packages/compass-indexes/src/modules/regular-indexes.ts

packages/compass-indexes/src/hooks/use-index-progress.ts

packages/compass-indexes/src/components/regular-indexes-table/regular-index-actions.tsx

packages/compass-indexes/src/components/regular-indexes-table/in-progress-index-actions.tsx

packages/compass-indexes/src/components/regular-indexes-table/regular-index-actions.tsx

packages/compass-indexes/src/components/regular-indexes-table/in-progress-index-actions.tsx

packages/data-service/src/data-service.ts

packages/compass-indexes/src/modules/regular-indexes.ts

gribnoysup · 2025-08-22T07:47:08Z

packages/compass-indexes/src/components/regular-indexes-table/in-progress-index-actions.tsx

@@ -43,12 +56,27 @@ const IndexActions: React.FunctionComponent<IndexActionsProps> = ({
    [onDeleteFailedIndexClick, index]
  );

+  const progress = index.buildProgress * 100;
+  const isBuilding = progress > 0 && progress < 100;


I'm not sure I follow this logic, we either have these in "in-progress" bucket or in "regular", how are we expecting this UI to show up for both?

Perhaps I've over engineered a bit, and I can confirm this again. IIUC the inprogress state is a "fake" one to make the UI show the user's index in the table right away after they create it. It changes to "ready" as soon as we see the index in a listIndexes response. I think the progress logic is left over here from when I would show users 0% when we had a true 0 value. I think I can remove this.

Yeah, your understanding is totally right here, although I guess now that we actually include in progress in "real" ones, the "in progress" name doesn't make much sense anymore

gribnoysup

I haven't had a chance to track this down, but I'm definitely seeing some weird behaviors locally: if I start the index creation (so it first gets into the "in progress" bucket in the state) I never see the progress shown for it, it just shows "in progress" badge until the index finished building fully, if I look at the indexes for the same collection in a separate tab (so the index is fetched from the beginning, not created as a virtual "in progress" one) it correctly shows the building progress, but the badge shows ready. Screenshots below show how this looks, notice that title_1 is in progress but without the progress in one tab, and ready but with progress in another

gribnoysup · 2025-08-26T12:01:31Z

packages/compass-indexes/src/components/regular-indexes-table/regular-index-actions.tsx

+  if (buildProgress > 0 && buildProgress < 1) {
+    return (
+      <div className={buildProgressStyles} data-testid="index-building-spinner">
+        <Body>Building... {(buildProgress * 100) | 0}%</Body>


Nit: we're not using much of bitwise operators in the codebase, can we use truncate here instead for readability?

Suggested change

<Body>Building... {(buildProgress * 100) | 0}%</Body>

<Body>Building… {Math.trunc(buildProgress * 100)}%</Body>

gribnoysup · 2025-08-26T12:02:13Z

packages/compass-indexes/src/components/regular-indexes-table/regular-indexes-table.tsx

    .filter((index) => !rollingIndexNames.has(index.name))
+    .filter((index) => !inProgressIndexNames.has(index.name))


Super nit, to avoid two passes

Suggested change

.filter((index) => !rollingIndexNames.has(index.name))

.filter((index) => !inProgressIndexNames.has(index.name))

.filter((index) => !rollingIndexNames.has(index.name) && !inProgressIndexNames.has(index.name))

More than a nit to me, good catch.

Someday we'll use the new Iterator to stack complex set of filters and maps perhaps...

gribnoysup · 2025-08-26T12:05:39Z

packages/compass-indexes/src/components/regular-indexes-table/in-progress-index-actions.tsx

@@ -43,12 +56,27 @@ const IndexActions: React.FunctionComponent<IndexActionsProps> = ({
    [onDeleteFailedIndexClick, index]
  );

+  const progress = index.buildProgress * 100;
+  const isBuilding = progress > 0 && progress < 100;


Yeah, your understanding is totally right here, although I guess now that we actually include in progress in "real" ones, the "in progress" name doesn't make much sense anymore

gribnoysup · 2025-08-26T12:13:59Z

packages/compass-indexes/src/components/regular-indexes-table/regular-index-actions.spec.tsx

+      expect(() => screen.getByTestId('index-building-spinner')).to.throw;
+      expect(() => screen.getByText(/Building\.\.\. \d+%/)).to.throw;


Needs to be called to validate

Suggested change

expect(() => screen.getByTestId('index-building-spinner')).to.throw;

expect(() => screen.getByText(/Building\.\.\. \d+%/)).to.throw;

expect(() => screen.getByTestId('index-building-spinner')).to.throw();

expect(() => screen.getByText(/Building\.\.\. \d+%/)).to.throw();

fixed, dunno how I missed this. On the node team I recall we either had a lint rule or we monkey-patched this assertion to require it has an argument because you could start throwing something else and not realize.

Hmmm, do you mind digging up what ruleset that was if you have time for this? I just did a quick search through the codebase and seems like we have a bunch of those uncalled ones, I didn't realize that this is easy to miss... 😬 "Worst" case we can probably just add our own custom rule for this

I found it, we modified the chai assertion: https://github.com/mongodb/js-bson/blob/d32397730426cc8423b89ddc1dd20ef4180f1677/test/register-bson.js#L14-L41

Which I realize won't help bc you won't hit this unless you call it, I think I fixed those mistakes with search and replace back when I introduced this. No "throw;" was pretty easy to eliminate 😅

nbbeeken · 2025-08-27T17:05:12Z

I am now seeing it display the build percentage correctly by undoing the filtering changes. This does mean the index now has a "ready" badge while there's a "building... x%" string right next to it. Not ideal 🤔

We also have a "building" state but for rolling indexes only, I'm not sure whats the best approach here because it seems like it would be a heavy lift to reframe the statuses from how they are currently used since they're bound to how the UI treats them (like how in-progress is really a dummy state until we get a ping from the server)

Any thoughts on the best investments we can make here? Could the status badge also rely on buildProgress if there is any?

gribnoysup · 2025-09-01T16:22:46Z

Badge can depend on progress, this is probably the easiest way to deal with this quickly and we can do this if we want to resolve this ticket sooner rather than later. Just generally I think the problem here is that your change changes the definition of in progress completely for compass but we're not adjusting for that in a more general sense: what compass was calling in progress before is not in progress anymore, so if we're changing that, it would also make sense to either remove this confusion from the codebase or treat the raw-ish output of data service in a way that maps to this concept properly. Your changes are somewhere in between right now, we're getting the indexes as part of what we now incorrectly call "ready", then we update the in progress ones to get the build progress, but at the same time we're still rendering the "ready" ones, not the "in progress" ones.

So either we change the code to make it clear that in progress is not in progress and ready is not always ready, or when receiveing data from data service we continue to map in progress to in progress group without mixing it with the ready one. If the ui wasn't so tied to the state, we'd probably have easier time going the first route (which is also preferable I'd say), but as that's not the case, we can probably try the latter and separate data service returned indexes before they end up in plugin state

addaleax · 2025-09-02T18:07:09Z

packages/compass-indexes/src/components/regular-indexes-table/regular-index-actions.spec.tsx

+      expect(() => screen.getByTestId('index-building-spinner')).to.throw(
+        /Unable to find/
+      );
+      expect(() => screen.getByText(/Building\.\.\. \d+%/)).to.throw(


Free ellipsis to copy if you like: … (it does look a small bit nicer than three separate dots)

addaleax · 2025-09-02T18:12:45Z

packages/data-service/src/data-service.ts

+      'META'
+    )
+      .aggregate([
+        { $currentOp: { allUsers: true, localOps: true } }, // get all ops


allUsers: true requires extra privileges. Do we want to try to fall back to allUsers: false if we're getting a permissions (or Atlas proxy) error?

localOps: true means we don't see the index build if it happened on a mongos that's different from the one hit by this aggregation, which seems quite realistic in clusters with multiple mongos instances. Do we care about that?

Good catches, we should do both these things I think. I switched to localOps false and I retry the agg with allUsers: false.

Should we parallelize both aggs and take the one with more information? I also didn't check the error message because I imagine there's no good way to make that not brittle

nbbeeken · 2025-09-02T20:19:15Z

I have started a new PR on top of this one to refactor the index status handling to be more accurate to reality. There is now a creating state and regular indexes make use of the building state that was previously only used by rolling indexes:

Maybe we can land this one as is, or if we're happy with the changes in that PR we can bundle them into one, I just didn't want to cloud the important bits of this work with refactorings. It seems to work pretty well! ⬇️

#7272

github-actions bot added the feat label Aug 15, 2025

nbbeeken force-pushed the COMPASS-9495 branch from 002fefc to 3dc8abd Compare August 15, 2025 18:31

nbbeeken added the release notes label Aug 15, 2025

nbbeeken force-pushed the COMPASS-9495 branch from 0d117da to 0a570dd Compare August 15, 2025 19:18

nbbeeken marked this pull request as ready for review August 15, 2025 19:18

Copilot AI review requested due to automatic review settings August 15, 2025 19:18

nbbeeken requested a review from a team as a code owner August 15, 2025 19:18

Copilot AI reviewed Aug 15, 2025

View reviewed changes

gribnoysup reviewed Aug 18, 2025

View reviewed changes

packages/compass-indexes/src/hooks/use-index-progress.ts Outdated Show resolved Hide resolved

gribnoysup reviewed Aug 19, 2025

View reviewed changes

packages/compass-indexes/src/components/regular-indexes-table/regular-index-actions.tsx Outdated Show resolved Hide resolved

gribnoysup reviewed Aug 22, 2025

View reviewed changes

nbbeeken requested a review from gribnoysup August 22, 2025 19:09

nbbeeken added 16 commits August 25, 2025 11:24

feat(compass-indexes): add index build progress COMPASS-9495

d16ee17

fix: tests

3408a51

fix: switch back to 10s intervals and round the number at the component

ab39f35

chore: dont need math

7d4209f

chore: copilot

df84f03

fix: move index progress to data-service

891d3d0

fix hook deps

9106a0a

fix: styles and types

e10b79d

test: buildProgress display

764f474

fix: styles

3f5faa5

fix: rename indexActionsContainerStyles

0853225

fix: use …

dbfa09f

fix: spacing

b43a671

docs: comments on filter

d2a93a2

feat: make the agg run the division

a902b7f

fix: remove progress from "in-progress"

e2e309f

nbbeeken force-pushed the COMPASS-9495 branch from 180dfaf to e2e309f Compare August 25, 2025 15:26

gribnoysup reviewed Aug 26, 2025

View reviewed changes

nbbeeken added 5 commits August 26, 2025 15:09

fix: assert thrown error correctly

990a128

fix: use trunc

4cced3e

remove double pass

c92237b

fix: return index filtering to what it was

163e40c

chore: diff

c179d47

nbbeeken requested a review from gribnoysup August 27, 2025 17:05

addaleax reviewed Sep 2, 2025

View reviewed changes

nbbeeken added 2 commits September 2, 2025 16:21

test: fix using ... vs …

10b2237

chore: try without setting "allUsers"

8ab3c95

nbbeeken requested a review from addaleax September 2, 2025 20:37

	<Body>Building... {(buildProgress * 100) \| 0}%</Body>
	<Body>Building… {Math.trunc(buildProgress * 100)}%</Body>

		.filter((index) => !rollingIndexNames.has(index.name))
		.filter((index) => !inProgressIndexNames.has(index.name))

		expect(() => screen.getByTestId('index-building-spinner')).to.throw;
		expect(() => screen.getByText(/Building\.\.\. \d+%/)).to.throw;

feat(compass-indexes): add index build progress COMPASS-9495 #7200

Are you sure you want to change the base?

feat(compass-indexes): add index build progress COMPASS-9495 #7200

Conversation

nbbeeken commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Motivation and Context

Open Questions

Dependents

Types of changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gribnoysup left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nbbeeken commented Aug 27, 2025

Uh oh!

gribnoysup commented Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nbbeeken commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nbbeeken commented Aug 15, 2025 •

edited

Loading

gribnoysup commented Sep 1, 2025 •

edited

Loading

nbbeeken commented Sep 2, 2025 •

edited

Loading