CDRIVER-6080 track client streams in connection counting tests #2095

kevinAlbs · 2025-08-18T20:14:02Z

Summary

This PR adds a test utility to count client-side streams: stream_tracker_t.

Tests in CDRIVER-6080 are unskipped and updated to replace server-side connection tracking with the client-side stream tracking.

As a drive-by fix, the llvm-symbolizer path is updated to prefer /opt/mongodbtoolchain/v4 if available.

Motivation

Running only /client_pool/disconnects_removed_servers/on_push on 100x-repeat reproduced failures: https://spruce.mongodb.com/version/689f2acc28a3590007ece216:

test error in: ../../../src/libmongoc/tests/test-mongoc-client-pool.c 445:disconnects_removed_servers_on_push()
Got unexpected connection count to localhost:27017:
Expected 6, got 4

With fixes from this PR, the test consistently appears to pass: https://spruce.mongodb.com/version/68a1fc25c3e9fc00072bbb70.

The test previously used the server-reported connections.current from serverStatus. Quoting docs:

The value will include all incoming connections including any shell connections or connections from other servers

I suspect connections.current might fluctuate from internal connections or (possibly) other connecting host processes. Testing locally, I saw connections.current increase after adding a replica set member, then later decrease after some time passed. And I am unsure if other processes on a host might be connecting.

llvm-symbolizer path

Updating the llvm-symbolizer path was motivated by an observed failure to report a leak: https://spruce.mongodb.com/version/68a0851cae97e00007709c1a:

llvm-symbolizer: Unknown command line argument '--inlines'.  Try: '/opt/mongodbtoolchain/v3/bin/llvm-symbolizer -help'
llvm-symbolizer: Did you mean '-inlining'?

After updated llvm-symbolizer path: https://spruce.mongodb.com/version/68a1fb780780030007cac342:

==10291==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 5824 byte(s) in 1 object(s) allocated from:
[...]

Update tests checking server-side connections. Intended to fix flaky failures.

To fix IPv4/IPv6 tests that use IP literals

Accidentally removed during rebase

eramongodb · 2025-08-18T21:18:42Z

src/libmongoc/tests/TestSuite.c

-   bson_t *cmd = BCON_NEW("serverStatus", BCON_INT32(1));
-   bson_t reply;
-   bson_error_t error;
-   bool ok = mongoc_client_command_simple(client, "admin", cmd, NULL, &reply, &error);
-   if (!ok) {
-      printf("serverStatus failed: %s\n", error.message);
-      abort();
-   }
-   int32_t conns;
-   // Get `connections.current` from the reply.
-   {
-      bson_iter_t iter;
-      BSON_ASSERT(bson_iter_init_find(&iter, &reply, "connections"));
-      BSON_ASSERT(bson_iter_recurse(&iter, &iter));
-      BSON_ASSERT(bson_iter_find(&iter, "current"));
-      conns = bson_iter_int32(&iter);
-   }


There is a similar function which uses connections.totalCreated. Should this also be updated to use the new stream tracker?

eramongodb · 2025-08-18T21:19:51Z

src/libmongoc/tests/stream-tracker.h

+void
+stream_tracker_track_pool(stream_tracker_t *st, mongoc_client_pool_t *pool);
+
+unsigned


Suggested change

unsigned

int

Avoid unsigned integers for counting. (CppCoreGuidelines)

eramongodb · 2025-08-18T21:20:58Z

src/libmongoc/tests/stream-tracker.c

+}
+
+
+mongoc_stream_t *


Suggested change

mongoc_stream_t *

static mongoc_stream_t *

Internal linkage.

src/libmongoc/tests/stream-tracker.h

eramongodb · 2025-08-19T15:19:52Z

src/libmongoc/tests/stream-tracker.h

+                       expect,                                                \
+                       _got);                                                 \
+         }                                                                    \
+         mlib_sleep_for(10, ms);                                              \


10ms is pretty short given a 5 seconds timeout. Perhaps use 100ms or 1s instead?

eramongodb · 2025-08-19T15:20:55Z

src/libmongoc/tests/stream-tracker.c

+
+typedef struct {
+   mongoc_host_list_t host;
+   unsigned count;


Suggested change

unsigned count;

int count;

Avoid unsigned integers for counting. (CppCoreGuidelines)

eramongodb · 2025-08-19T18:54:22Z

src/libmongoc/tests/test-mongoc-dns.c

+   stream_tracker_assert_count(st, "localhost.test.build.10gen.cc:27017", 0u);
+   stream_tracker_assert_count(st, "localhost.test.build.10gen.cc:27018", 0u);


Suggested change

stream_tracker_assert_count(st, "localhost.test.build.10gen.cc:27017", 0u);

stream_tracker_assert_count(st, "localhost.test.build.10gen.cc:27018", 0u);

stream_tracker_assert_count(st, host0, 0u);

stream_tracker_assert_count(st, host1, 0u);

These hostnames should be made variables due to repeated reuse.

eramongodb · 2025-08-19T18:54:38Z

src/libmongoc/tests/test-mongoc-client-pool.c

+   stream_tracker_assert_count(st, "localhost:27017", 0u);
+   stream_tracker_assert_count(st, "localhost:27018", 0u);


Suggested change

stream_tracker_assert_count(st, "localhost:27017", 0u);

stream_tracker_assert_count(st, "localhost:27018", 0u);

stream_tracker_assert_count(st, host0, 0u);

stream_tracker_assert_count(st, host1, 0u);

These hostnames should be made variables due to repeated reuse.

Co-authored-by: Ezra Chung <[email protected]>

Previously checking a connection count (by running `serverStatus`) could trigger creating a new connection.

kevinAlbs added 3 commits August 18, 2025 16:03

prefer v4 for llvm-symbolizer

df507fe

add stream_tracker_t

4c51279

Update tests checking server-side connections. Intended to fix flaky failures.

remove unused utilities

0173969

kevinAlbs requested a review from eramongodb August 18, 2025 20:22

kevinAlbs marked this pull request as ready for review August 18, 2025 20:22

kevinAlbs requested a review from a team as a code owner August 18, 2025 20:22

kevinAlbs added 4 commits August 18, 2025 21:57

target localhost:27017

9e57b98

use first host+port for self-test

6dab725

To fix IPv4/IPv6 tests that use IP literals

fix leak

3150ae7

restore remove header

0a19fdd

Accidentally removed during rebase

eramongodb requested changes Aug 19, 2025

View reviewed changes

kevinAlbs and others added 10 commits August 19, 2025 16:25

use parens for (void)0

0a6300f

Co-authored-by: Ezra Chung <[email protected]>

add cumulative count

7b9b27e

use cumulative count in exhaust cursor tests

1e0c6cf

refactor stepdown tests to use stream tracker

18b5bcc

check connection count after connection created

57ded78

Previously checking a connection count (by running `serverStatus`) could trigger creating a new connection.

update comments: serverStatus is no longer used

a41775c

use int to track count

aa00590

use internal linkage

479e77b

use longer pause in eventual check

07f522c

use variables for hosts

fa22744

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CDRIVER-6080 track client streams in connection counting tests #2095

CDRIVER-6080 track client streams in connection counting tests #2095

kevinAlbs commented Aug 18, 2025

Uh oh!

eramongodb Aug 18, 2025

Uh oh!

eramongodb Aug 18, 2025

Uh oh!

eramongodb Aug 18, 2025

Uh oh!

Uh oh!

Uh oh!

eramongodb Aug 19, 2025

Uh oh!

eramongodb Aug 19, 2025

Uh oh!

eramongodb Aug 19, 2025

Uh oh!

eramongodb Aug 19, 2025

Uh oh!

Uh oh!

		stream_tracker_assert_count(st, "localhost.test.build.10gen.cc:27017", 0u);
		stream_tracker_assert_count(st, "localhost.test.build.10gen.cc:27018", 0u);

CDRIVER-6080 track client streams in connection counting tests #2095

Are you sure you want to change the base?

CDRIVER-6080 track client streams in connection counting tests #2095

Conversation

kevinAlbs commented Aug 18, 2025

Summary

Motivation

llvm-symbolizer path

Uh oh!

eramongodb Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

eramongodb Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

eramongodb Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eramongodb Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

eramongodb Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

eramongodb Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

eramongodb Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!