[NO-MERGE][dotnet-trace] Add collect-linux verb #5570

mdh1418 · 2025-09-17T21:50:29Z

Blocked on: Microsoft.OneCollect.RecordTrace version with FFI support

Following the addition of emitting native runtime and custom EventSource events as user_events through dotnet/runtime#115265 and the public release of https://github.com/microsoft/one-collect which supports collecting both .NET user_events and Linux perf events into a single .nettrace file, dotnet-trace will support a new verb, collect-linux, that wraps around record-trace.

This PR does the following:

Adds collect-linux verb and serializes a subset of dotnet-trace collect options in addition to a collect-linux specific --perf-events option into record-trace args. (see [Diagnostics][dotnet-trace] Add collect-linux verb docs#47894 for overarching details)
Adds record-trace dynamic library to dotnet-trace
Updates existing profiles (cpu-sampling -> dotnet-common + dotnet-sampled-thread-time) and adds collect-linux specific profiles
Updates list-profiles verb with revamped profiles + multiline description formatting
Refactors EventPipeProvider composition logic (MergeProfileAndProviders + ToProviders -> ComputeProviderConfig) and rename Extensions.cs -> ProviderUtils.cs
Revamp EventPipeProvider composition tests (ProviderParsing.cs -> ProviderCompositionTests.cs)
Various cleanup: Update CLREventKeywords + Update logging + refactor collect logic + expand dotnet-trace common options

Testing

dotnet-trace collect-linux

On Linux

collect-linux

collect-linux --help

$ ./dotnet-trace collect-linux -h
Description:
  Collects diagnostic traces using perf_events, a Linux OS technology. collect-linux requires admin privileges to capture kernel- and user-mode events, and by default, captures events from all processes. This Linux-only command includes the same .NET
  events as dotnet-trace collect, and it uses the kernel’s user_events mechanism to emit .NET events as perf events, enabling unification of user-space .NET events with kernel-space system events.

Usage:
  dotnet-trace collect-linux [options]

Options:
  --providers       A comma delimited list of EventPipe providers to be enabled. This is in the form 'Provider[,Provider]',where Provider is in the form: 'KnownProviderName[:[Flags][:[Level][:[KeyValueArgs]]]]', and KeyValueArgs is in the form:
                    '[key1=value1][;key2=value2]'.  Values in KeyValueArgs that contain ';' or '=' characters need to be surrounded by '"', e.g., FilterAndPayloadSpecs="MyProvider/MyEvent:-Prop1=Prop1;Prop2=Prop2.A.B;".  Depending on your shell, you may
                    need to escape the '"' characters and/or surround the entire provider specification in quotes, e.g., --providers 'KnownProviderName:0x1:1:FilterSpec=\"KnownProviderName/EventName:-Prop1=Prop1;Prop2=Prop2.A.B;\"'. These providers are in
                    addition to any providers implied by the --profile argument. If there is any discrepancy for a particular provider, the configuration here takes precedence over the implicit configuration from the profile.  See documentation for
                    examples.
  --clreventlevel   Verbosity of CLR events to be emitted.
  --clrevents       List of CLR runtime events to emit.
  --perf-events     Comma-separated list of perf events (e.g. syscalls:sys_enter_execve,sched:sched_switch).
  --profile         A named, pre-defined set of provider configurations for common tracing scenarios. You can specify multiple profiles as a comma-separated list. When multiple profiles are specified, the providers and settings are combined (union), and
                    duplicates are ignored.
  -o, --output      The output path for the collected trace data. If not specified it defaults to '<appname>_<yyyyMMdd>_<HHmmss>.nettrace', e.g., 'myapp_20210315_111514.nettrace'. [default: default]
  --duration        When specified, will trace for the given timespan and then automatically stop the trace. Provided in the form of dd:hh:mm:ss.
  -n, --name        The name of the process to collect the trace.
  -p, --process-id  The process id to collect the trace.
  -?, -h, --help    Show help and usage information

`collect-linux` without elevated privileges

$ ./dotnet-trace collect-linux
No profile or providers specified, defaulting to trace profiles 'dotnet-common' + 'cpu-sampling'.
Applying profile 'dotnet-common': Microsoft-Windows-DotNETRuntime:0x000000100003801D:4
Applying profile 'cpu-sampling': --on-cpu

Provider Name                           Keywords            Level               Enabled By
Microsoft-Windows-DotNETRuntime         0x000000100003801D  Informational(4)    --profile

Error: Tracefs is not accessible: Permission denied (os error 13)

`collect-linux` with elevated privileges

$ sudo ./dotnet-trace collect-linux
[sudo] password for mihw:
No profile or providers specified, defaulting to trace profiles 'dotnet-common' + 'cpu-sampling'.
Applying profile 'dotnet-common': Microsoft-Windows-DotNETRuntime:0x000000100003801D:4
Applying profile 'cpu-sampling': --on-cpu

Provider Name                           Keywords            Level               Enabled By
Microsoft-Windows-DotNETRuntime         0x000000100003801D  Informational(4)    --profile

Recording started.  Press CTRL+C to stop.
^C
Recording stopped.
Resolving symbols.
Finished recording trace.
Trace written to trace_20250919_205934.nettrace

On Windows (and I presume other non-Linux OS):

`collect-linux`

.\artifacts\bin\dotnet-trace\Debug\net8.0\dotnet-trace.exe collect-linux
The collect-linux command is only supported on Linux.

dotnet-trace list-profiles

`list-profiles`

dotnet-trace profiles:
        dotnet-common                        - Lightweight .NET runtime diagnostics designed to stay low overhead.
                                               Includes:
                                                   GC
                                                   AssemblyLoader
                                                   Loader
                                                   JIT
                                                   Exceptions
                                                   Threading
                                                   JittedMethodILToNativeMap
                                                   Compilation
                                               Equivalent to --providers "Microsoft-Windows-DotNETRuntime:0x100003801D:4".
        dotnet-sampled-thread-time (collect) - Samples .NET thread stacks (~100 Hz) toestimate how much wall clock time code is using.
        gc-verbose                           - Tracks GC collections and samples object allocations.
        gc-collect                           - Tracks GC collections only at very low overhead.
        database                             - Captures ADO.NET and Entity Framework database commands
        cpu-sampling (collect-linux)         - Kernel CPU sampling events for measuring CPU usage.
        thread-time (collect-linux)          - Kernel thread context switch events for measuring CPU usage and wall clock time

src/Tools/dotnet-trace/CommandLine/Commands/CollectCommand.cs

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

noahfalk · 2025-09-18T21:46:27Z

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

+                return (int)ReturnCode.TracingError;
+            }
+
+            proc.OutputDataReceived += (_, e) =>


Fine for now but once we've got the FFI working I'm hoping that dotnet-trace will control the majority of the output and only pass along warnings/errors.

Hooked in the FFI and have the output type routed to Console.Out and Console.Error. Didn't add handling for progress as I haven't seen records yet.

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

src/Tools/dotnet-trace/CommandLine/Options/CommonOptions.cs

src/Tools/dotnet-trace/ProviderUtils.cs

src/tests/dotnet-trace/ProfileProviderMerging.cs

src/Tools/dotnet-trace/ProviderUtils.cs

Extensions houses provider composition helpers, instead of literal extension methods for strings. Centralize all provider parsing + composition logic and keep Profile as a data container.

mdh1418 · 2025-09-19T21:15:22Z

src/Tools/dotnet-trace/dotnet-trace.csproj

+    </PropertyGroup>
+
+    <ItemGroup>
+      <_RecordTraceResolved Include="$(_RecordTraceLocal)" Condition="'$(RuntimeIdentifier)' != '' AND Exists('$(_RecordTraceLocal)')" />


Oops, artifact from before, will remove

noahfalk

This is looking pretty good. Aside from some small comments inline a few broader things:

Historically we relied on manual testing to keep these tools operating well but now that we have less time available from the testers I think we need to improve our automated testing. Partly this is to ensure we aren't inadvertently changing the original 'collect' verb and partly to ensure going forward the 'collect-linux' behavior doesn't regress either. I think the best way to do this would be:

Open a new PR that we'll check in first containing some basic tests of the existing collect verb.
Commit this PR 2nd and all the tests in the 1st PR should continue to pass. This ensures we didn't change anything unintended.

To do the testing we probably need to create some small interface shims. We already have an IConsole interface defined that could be moved to the shared Common folder. We could also create a small interface around the DiagnosticsClient.EventPipeCollect() API so that a test can return some dummy data in a stream instead. dotnet-counters has some example tests that show how we can run some code and then confirm the console output is what we expect. In this case I imagine we'd be running the Collect() function and giving some chosen input arguments.

I think there is a bit more adjustment to be done on some of the output text, but it should be fine to get this one in first, then tweak afterwards in some 3rd PR.

noahfalk · 2025-09-23T22:55:54Z

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

+            string options = string.Join(' ', recordTraceArgList);
+            byte[] command = Encoding.UTF8.GetBytes(options);
+            int rc;
+            try


Nit: I'd shift the try { to start prior to registering the keypress handler. We want to ensure that any error from then onwards cleans everything up.

noahfalk · 2025-09-23T22:59:14Z

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

+
+        private static int RunRecordTrace(CollectLinuxArgs args)
+        {
+            s_recordStatus = 0;


Lets use a boolean flag or an enum rather than magic constants.

noahfalk · 2025-09-23T23:08:16Z

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

+            return collectLinuxCommand;
+        }
+
+        private static int RunRecordTrace(CollectLinuxArgs args)


The naming and factoring feels a little unexpected. I think of "RunRecordTrace" to mean invoking the FFI, waiting for it to finish, and maybe updating the UI during the process. I didn't expect it to include parsing configuration, formatting the script file, determining the output path, or any other calculations to produce the record-trace arguments. I'd suggest just fold this into CollectLinux and name RecordTrace -> RunRecordTrace

noahfalk · 2025-09-24T00:00:22Z

src/Tools/dotnet-trace/ProviderUtils.cs

+                    {
+                        profileEffect = traceProfile.CollectLinuxArgs;
+                    }
+                    Console.WriteLine($"Applying profile '{traceProfile.Name}': {profileEffect}");


This looks like debug output that we should remove

I added this and the perf event log because collect-linux introduces .NET Provider-free configurations such as --profile cpu-sampling and --perf-events block:block_io_start. I figured it would be a good sanity check for users to see those effects to know that the tracing session is actually capturing something, rather than

No providers were configured. Recording started. Press CTRL+C to stop.

Previously, those would have also just been

Provider Name Keywords Level Enabled By Recording started. Press CTRL+C to stop.

until adding to ProviderUtils.cs PrintProviders

if (providers.Count == 0) { Console.WriteLine("No providers were configured."); return; }

With this and the below perf-events log, users can see

$ dotnet-trace collect-linux --profile cpu-sampling Applying profile 'cpu-sampling': --on-cpu No providers were configured. Recording started. Press CTRL+C to stop.

and

$ dotnet-trace collect-linux --perf-events block:block_io_start,block:block_io_done No providers were configured. Enabling perf event 'block:block_io_start' Enabling perf event 'block:block_io_done' Recording started. Press CTRL+C to stop.

noahfalk · 2025-09-24T00:03:23Z

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

+
+                string perfProvider = split[0];
+                string perfEventName = split[1];
+                Console.WriteLine($"Enabling perf event '{perfEvent}'");


Debug output?

mdh1418 requested review from noahfalk, lateralusX and hoyosjs September 17, 2025 21:50

noahfalk reviewed Sep 19, 2025

View reviewed changes

mdh1418 force-pushed the dotnet_trace_collect_linux branch from c09b095 to 0989d21 Compare September 19, 2025 05:45

mdh1418 added 12 commits September 19, 2025 11:08

[DotnetTrace] Update CLREventKeywords

536cd50

[DotnetTrace] Move MergeProfileAndProviders to Extensions

49a875b

Extensions houses provider composition helpers, instead of literal extension methods for strings. Centralize all provider parsing + composition logic and keep Profile as a data container.

[DotnetTrace] Rename Extensions to ProviderUtils

b4787fc

[DotnetTrace] Add provider unifying helper

c921869

[DotnetTrace] Move shared options to CommonOptions

e1c15c7

[DotnetTrace] Add collect-linux skeleton

b4112b5

[DotnetTrace][CollectLinux] Start record-trace

e5f3975

[DotnetTrace][CollectLinux] Build record-trace args

677e54e

[DotnetTrace] Update profiles

31186a8

[DotnetTrace] Update collect to new provider unifier

44593bb

[DotnetCounters] Remove Extensions reference

b190d73

[DotnetTrace] Update tests

9f1ef6d

mdh1418 force-pushed the dotnet_trace_collect_linux branch 2 times, most recently from 76e65e1 to 61b2569 Compare September 19, 2025 18:32

Address Feedback

c8e53ea

mdh1418 force-pushed the dotnet_trace_collect_linux branch from 61b2569 to c8e53ea Compare September 19, 2025 18:56

[DotnetTrace] Print profile effects and clrevents ignore warning

590a203

mdh1418 marked this pull request as ready for review September 19, 2025 19:43

mdh1418 requested a review from a team as a code owner September 19, 2025 19:43

mdh1418 added the DO NOT MERGE do not merge this PR label Sep 19, 2025

mdh1418 changed the title ~~[dotnet-trace] Add collect-linux verb~~ [NO-MERGE][dotnet-trace] Add collect-linux verb Sep 19, 2025

mdh1418 commented Sep 19, 2025

View reviewed changes

[DotnetTrace] Print PerfEvents and include in default condition

8e2453f

noahfalk reviewed Sep 24, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[NO-MERGE][dotnet-trace] Add collect-linux verb #5570

[NO-MERGE][dotnet-trace] Add collect-linux verb #5570

mdh1418 commented Sep 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

noahfalk Sep 18, 2025

Uh oh!

mdh1418 Sep 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mdh1418 Sep 19, 2025

Uh oh!

noahfalk left a comment •

edited

Loading

Uh oh!

noahfalk Sep 23, 2025

Uh oh!

noahfalk Sep 23, 2025

Uh oh!

noahfalk Sep 23, 2025

Uh oh!

noahfalk Sep 24, 2025

Uh oh!

mdh1418 Sep 29, 2025

Uh oh!

noahfalk Sep 24, 2025

Uh oh!

Uh oh!

[NO-MERGE][dotnet-trace] Add collect-linux verb #5570

Are you sure you want to change the base?

[NO-MERGE][dotnet-trace] Add collect-linux verb #5570

Conversation

mdh1418 commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Blocked on: Microsoft.OneCollect.RecordTrace version with FFI support

Testing

dotnet-trace collect-linux

dotnet-trace list-profiles

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

noahfalk left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mdh1418 commented Sep 17, 2025 •

edited

Loading

noahfalk left a comment •

edited

Loading