NetworkList Sometimes Goes Out of Sync in Distributed Authority Setup

[nikajavakha]

Description

In a distributed authority setup, I noticed that the NetworkList sometimes goes out of sync. For example, I have a private NetworkList called connectedPlayers. I built the game and ran three instances, where Client 1 was the session owner. Only the session owner is allowed to update connectedPlayers.

For each client connection, I added the new PlayerConnection from the session owner's side. Initially, it logged correctly for the first two clients as:
SceneEventType: SynchronizeComplete | ConnectedPlayerClientIds: 1,3,2

However, for the third client, the log showed:
SceneEventType: SynchronizeComplete | ConnectedPlayerClientIds: 1,3,2,3,2

indicating that some entries were duplicated. this not happens often(mostly happens when I run multiple built instances simultaneously, causing my PC's CPU usage to reach 100%.) so it's hard to reproduce but it makes me feel that NetworkList is unreliable to use. Is there any alternative of NetworkList? or if not could this be indicating bug in NGO?

public class NetworkPlayerManager : NetworkBehaviour
{
    private NetworkList<PlayerConnection> connectedPlayers = new();

    public override void OnNetworkSpawn()
    {
            NetworkManager.SceneManager.OnSceneEvent += OnSceneEvent;
    }

    private void OnSceneEvent(SceneEvent sceneEvent)
    {
        Debug.Log($"SceneEventType: {sceneEvent.SceneEventType} | ConnectedPlayerClientIds: {string.Join(",", connectedPlayers.ToList().Select(c => c.ClientId))}")
    }
}

using System;
using Assets.Scripts.Tools;
using Unity.Collections;
using Unity.Netcode;

namespace Assets.Scripts
{
    public struct PlayerConnection : INetworkSerializable, IEquatable<PlayerConnection>
    {
        public ulong ClientId;
        public FixedString64Bytes PlayerId;
        public FixedString64Bytes PlayerName;
        public NetworkObjectReference NetworkPlayerRef;
        
        public PlayerConnection(
            ulong clientId,
            FixedString64Bytes playerId,
            FixedString64Bytes playerName,
            NetworkObjectReference networkPlayerRef)
        {
            ClientId = clientId;
            PlayerId = playerId;
            PlayerName = playerName;
            NetworkPlayerRef = networkPlayerRef;
        }

        public void NetworkSerialize<T>(BufferSerializer<T> serializer) where T : IReaderWriter 
        {
            serializer.SerializeValue(ref ClientId);
            serializer.SerializeValue(ref PlayerId);
            serializer.SerializeValue(ref PlayerName);
            serializer.SerializeValue(ref NetworkPlayerRef);
        }

        public bool Equals(PlayerConnection other)
        {
            return ClientId == other.ClientId &&
                   PlayerId == other.PlayerId &&
                   PlayerName == other.PlayerName &&
                   NetworkPlayerRef.Equals(other.NetworkPlayerRef);
        }
    
    }
}

Environment

• Unity Version: Unity 6 (6000.0.32f1)
• Netcode Version: [2.2.0]

[NoelStephensUnity]

@nikajavakha
Could you provide the script where the SessionOwner is updating the connectedPlayers property?

Optionally, you might try an alternate approach like this:

/// <summary>
/// Assuming this is on an in-scene placed NetworkObject
/// </summary>
public class NetworkPlayerManager : NetworkBehaviour
{
    private NetworkVariable<Dictionary<ulong, PlayerConnection>>  connectedPlayers = new NetworkVariable<Dictionary<ulong, PlayerConnection>>(new Dictionary<ulong, PlayerConnection>());
    private ulong LastKnownSessionOwner;
    public override void OnNetworkSpawn()
    {
        NetworkManager.OnSessionOwnerPromoted += NetworkManager_OnSessionOwnerPromoted;
        LastKnownSessionOwner = NetworkManager.CurrentSessionOwner;
        if (NetworkManager.LocalClient.IsSessionOwner)
        {
            NetworkManager.OnConnectionEvent += NetworkManager_OnConnectionEvent;
        }
        else
        {
            connectedPlayers.OnValueChanged += OnconnectedPlayersChanged;
        }
    }

    /// <summary>
    /// Upon a new session owner being promoted due to the previous one having disconnected,
    /// register for connection events and take over tracking connected players.
    /// </summary>
    private void NetworkManager_OnSessionOwnerPromoted(ulong sessionOwnerPromoted)
    {
        if (NetworkManager.LocalClientId == sessionOwnerPromoted) 
        {
            connectedPlayers.OnValueChanged -= OnconnectedPlayersChanged;
            NetworkManager.OnConnectionEvent += NetworkManager_OnConnectionEvent;
            // Handle removing the previous session owner
            if (connectedPlayers.Value.ContainsKey(LastKnownSessionOwner))
            {
                connectedPlayers.Value.Remove(LastKnownSessionOwner);
                connectedPlayers.CheckDirtyState();
            }
        }

        LastKnownSessionOwner = sessionOwnerPromoted;
    }

    protected override void OnNetworkSessionSynchronized()
    {
        // Each non-sessioin owner client will log all connected players upon finishing synchronization
        LogConnectedPlayers();
        base.OnNetworkSessionSynchronized();
    }

    private void NetworkManager_OnConnectionEvent(NetworkManager networkManager, ConnectionEventData eventData)
    {
        switch(eventData.EventType)
        {
            case ConnectionEvent.PeerConnected:
                {
                    // Add player info
                    connectedPlayers.Value.Add(eventData.ClientId, new PlayerConnection()
                    {
                        ClientId = eventData.ClientId,
                    });
                    connectedPlayers.CheckDirtyState();
                    break;
                }
            case ConnectionEvent.PeerDisconnected:
                {
                    // Remove player
                    connectedPlayers.Value.Remove(eventData.ClientId);
                    connectedPlayers.CheckDirtyState();
                    break;
                }
        }
    }

    private void LogConnectedPlayers()
    {
        Debug.Log($"ConnectedPlayerClientIds: {string.Join(",", connectedPlayers.Value.Values.Select(c => c.ClientId))}");
    }

    private void OnconnectedPlayersChanged(Dictionary<ulong, PlayerConnection> previous, Dictionary<ulong, PlayerConnection> current)
    {
        LogConnectedPlayers();
    }
}

[nikajavakha]

I listen for SceneEvent on the session owner's side. If the event is LoadComplete, I spawn a NetworkPlayer for the SessionOwner. If the event is SynchronizeComplete, I spawn a NetworkPlayer for the connected client. Then, each NetworkPlayer, after spawning, sends its PlayerId and PlayerName to the session owner, who adds it to the connectedPlayers list.

First, I am hosting a game, so the session owner is connected and ready. Then, I open multiple game instances simultaneously to join the session. However, the network list often goes out of sync by that time (I run all instances on the same machine at the same time, and sometimes the CPU briefly reaches 100%).

However, if I open the game instances with a 2–3 second delay, everything works correctly. During the mid-game, the network list also remains properly synchronized—I haven't noticed any issues.

Note: NetworkManager.Singleton.ConnectedClientIds is always correct, even when I open game instances simultaneously.

here is pseudo code:

public class NetworkPlayerManager : NetworkBehaviour
{
    private void OnSceneEvent(SceneEvent sceneEvent)
    {
        Debug.Log($"SceneEventType: {sceneEvent.SceneEventType} | ConnectedPlayerClientIds: {string.Join(",", connectedPlayers.ToList().Select(c => c.ClientId))}")

        if (!NetworkManager.LocalClient.IsSessionOwner)
            return;

        switch (sceneEvent.SceneEventType)
        {
            case SceneEventType.LoadComplete:
                // spawn session owner
                if (sceneEvent.ClientId == NetworkManager.LocalClientId && !HasConnectedPlayer(sceneEvent.ClientId))
                {
                    SpawnNetworkPlayer(sceneEvent.ClientId);
                }

                break;
            case SceneEventType.SynchronizeComplete:
                //spawn peer client
                SpawnNetworkPlayer(sceneEvent.ClientId);
                break;
        }
    }

    private void SpawnNetworkPlayer(ulong clientId)
    {
        var networkPlayer = Instantiate(networkPlayerPrefab);
        networkPlayer.NetworkObject.SpawnWithOwnership(clientId);
    }

    [Rpc(SendTo.Authority)]
    public void RegisterPlayerAuthorityRpc(PlayerConnection playerConnection, RpcParams rpcParams = default)
    {
        connectedPlayers.Add(playerConnection);
    }

     public bool HasConnectedPlayer(ulong clientId)
     {
            foreach (var player in connectedPlayers)
            {
                if (player.ClientId == clientId)
                    return true;
            }

            return false;
       }
}

public class NetworkPlayer : NetworkBehaviour 
{
    public override void OnNetworkSpawn(){
        if (IsOwner){
            var playerId = AuthenticationService.Instance.PlayerId
            var playerName = AuthenticationService.Instance.Profile;
            var playerConnection = new PlayerConnection(OwnerClientId, playerId, playerName, NetworkObject);
            NetworkPlayerManager.Instance.RegisterPlayerAuthorityRpc(playerConnection);
        }
    }
}

[NoelStephensUnity]

@nikajavakha
I would highly recommend using an alternate approach as you most likely will want to spawn the player from the client owner side of things and you typically don't need to use scene events for determining when a client can spawn their player prefab when using a distributed authority network topology.

As another example (which is an adjustment from the above script you provided), you can obtain the PlayerConnection information for each player instance without using a NetworkList or scene events:

public class NetworkPlayerManager : NetworkBehaviour
{
    // This should only be the prefab asset (i.e. GameObject)
    public GameObject networkPlayerPrefab;

    /// <summary>
    /// Session owner spawns when this in-scene placed NetworkObject has finished spawning.
    /// </summary>
    protected override void OnNetworkPostSpawn()
    {
        if (IsSessionOwner)
        {
            SpawnNetworkPlayer();
        }
        base.OnNetworkPostSpawn();
    }

    /// <summary>
    /// All other clients will automatically spawn their player prefab when they have finished synchronizing
    /// </summary>
    protected override void OnNetworkSessionSynchronized()
    {
        if (!IsSessionOwner)
        {
            SpawnNetworkPlayer();
        }
        base.OnNetworkSessionSynchronized();
    }

    /// <summary>
    /// Spawns a unique player prefab or the default player prefab.
    /// This could be invoked again if the player has to be re-spawned.
    /// </summary>
    public void SpawnNetworkPlayer(GameObject playerPrefab = null, Vector3 position = default, Quaternion rotation = default)
    {
        if (playerPrefab == null)
        {
            playerPrefab = NetworkManager.NetworkConfig.PlayerPrefab;
        }
        // instantiates the player prefab based on the relative client instance
        NetworkObject.InstantiateAndSpawn(playerPrefab, NetworkManager, NetworkManager.LocalClientId, isPlayerObject: true, position: position, rotation: rotation);
    }

    /// <summary>
    /// Returns the PlayerConnection for the player/client identifier specified
    /// </summary>
    public PlayerConnection GetPlayerInfo(ulong playerId)
    {
        var playerNetworkObject = NetworkManager.SpawnManager.GetPlayerNetworkObject(playerId);
        if (playerNetworkObject)
        {
            return playerNetworkObject.GetComponent<NetworkPlayer>().PlayerConnection;
        }
        return null;
    }
}

public class NetworkPlayer : NetworkBehaviour
{
    private NetworkVariable<PlayerConnection> PlayerConnectionInfo = new NetworkVariable<PlayerConnection>();
    public PlayerConnection PlayerConnection => PlayerConnectionInfo.Value;
    public override void OnNetworkSpawn()
    {
        if (IsOwner)
        {
            var playerId = AuthenticationService.Instance.PlayerId;
            var playerName = AuthenticationService.Instance.Profile;
            // Each owner will automatically register their PlayerConnection information with this NetworkPlayer instance
            PlayerConnectionInfo.Value = new PlayerConnection(OwnerClientId, playerId, playerName, NetworkObject);
        }
    }
}

The idea here is to keep this kind of information grouped with the NetworkPlayer so you can just use existing NGO methods to gain access to things like PlayerConnection. For this particular scenario I would recommend not trying to link tracking connected players and connection information to scene loading events. Not to say there are times where you might need to wait for a specific scene to finish loading... but since the modified example above has each client spawn their own player prefab instance when they are done synchronizing it reduces the complexity.

You might also be interested in looking at the NetworkManager.OnFetchLocalPlayerPrefabToSpawn callback.

Let me know if the adjusted script example helps?

[nikajavakha]

@NoelStephensUnity
I initially tried that approach, but when each client spawns its own NetworkPlayer, I run into even stranger issues. For example, if I start a session and try to connect 16 players, some clients might end up with 15 NetworkPlayers, while others might have 10, 12, or a different number. This means that if each client spawns its own NetworkPlayer, synchronization is not handled properly.

However, when the session owner spawns the NetworkPlayers, everything remains properly synchronized for everyone. The only issue is that I occasionally get duplicate client IDs in the NetworkList, but this doesn’t happen often. Even when it does, I can detect and disconnect the affected client. Aside from that, spawning NetworkPlayers from the session owner’s side is much more stable.

I also tried using a NetworkVariable within NetworkPlayer itself instead of maintaining a separate connectedPlayers list. However, in that case, setting the NetworkVariable in the OnNetworkSpawn method doesn’t always synchronize properly. On the other hand, keeping a separate list in an already spawned NetworkObject (such as NetworkPlayerManager in my case) is much more stable.

[NoelStephensUnity]

@nikajavakha
Hmmm, when you say you are trying to connect 16 players are you trying to do this all at the exact same time or are they staggered slightly like they would be under a normal session? If you have a replication project that causes this issue I would be very interested in seeing where you are losing CreateObjectMessages.

[nikajavakha]

@NoelStephensUnity
Here is my Unity project:
I have a ServiceHelper class which creates or joins a session in the Start method.
NetworkPlayer is set as the Default Player Prefab in the NetworkManager. If you spawn NetworkPlayer from NetworkPlayerManager, you need to disable "Auto Spawn Player Prefab" in the NetworkManager; otherwise, you need to enable it. In both cases, NetworkPlayer synchronization is not handled properly. In an empty scene, you only need to have ServiceHelper, NetworkPlayerManager, and NetworkManager.

ServiceHelper:

using System;
using System.Collections;
using System.Linq;
using System.Threading.Tasks;
using Assets.Scripts;
using Unity.Netcode;
using Unity.Services.Authentication;
using Unity.Services.Core;
using Unity.Services.Multiplayer;
using UnityEngine;

namespace DefaultNamespace
{
    public class ServiceHelper : MonoBehaviour
    {
        public static ServiceHelper Instance { get; private set; }

        public string sessionName = "test";
        
        private ISession m_CurrentSession;
        private bool m_IsLeavingSession;

        private void Awake()
        {
            if (Instance == null)
            {
                Instance = this;
                DontDestroyOnLoad(gameObject);
            }
            else
            {
                Destroy(gameObject);
            }
        }

        private async void Start()
        {
            Application.wantsToQuit += OnWantToQuit;

            Debug.Log("Initializing Unity Services...");

            await UnityServices.InitializeAsync();
            await SignIn();
            Debug.Log($"Initial load complete, connecting to session 'Test'");
            await ConnectSession();
        }
        
        void OnDestroy()
        {
            Application.wantsToQuit -= OnWantToQuit;
        }

        private async Task SignIn()
        {
            if (!AuthenticationService.Instance.IsSignedIn)
            {
                string profileId = GetRandomString(5);
                Debug.Log($"Signing in with anonymous profile: {profileId}");
                AuthenticationService.Instance.SwitchProfile(profileId);
                await AuthenticationService.Instance.SignInAnonymouslyAsync();
                Debug.Log("Authentication successful");
            }
        }

        private async Task ConnectSession()
        {
            try
            {
                var options = new SessionOptions()
                {
                    Name = sessionName,
                    MaxPlayers = 64,
                }.WithDistributedAuthorityNetwork();

                var response = await MultiplayerService.Instance.QuerySessionsAsync(new QuerySessionsOptions());
                var session = response.Sessions.FirstOrDefault(m => m.Name == sessionName);
                if (session != null)
                {
                    Debug.Log($"Joining existing session with ID: {response.Sessions[0].Id}");
                    m_CurrentSession = await MultiplayerService.Instance.JoinSessionByIdAsync(response.Sessions[0].Id);
                }
                else
                {
                    Debug.Log($"Creating new session: {sessionName}");
                    m_CurrentSession = await MultiplayerService.Instance.CreateSessionAsync(options);
                }

                m_CurrentSession.RemovedFromSession += RemovedFromSession;
                m_CurrentSession.StateChanged += CurrentSessionOnStateChanged;
            }
            catch (Exception e)
            {
                Debug.LogException(e);
            }
        }

        public async void LeaveSession()
        {
            if (m_CurrentSession != null && !m_IsLeavingSession)
            {
                try
                {
                    m_IsLeavingSession = true;
                    Debug.Log("Leaving current session...");
                    await m_CurrentSession.LeaveAsync();
                }
                catch (Exception e)
                {
                    Debug.LogException(e);
                    throw;
                }
                finally
                {
                    m_IsLeavingSession = false;
                    ExitedSession();
                }
            }
        }

        void RemovedFromSession()
        {
            Debug.Log("Removed from session by server");
            ExitedSession();
        }

        void CurrentSessionOnStateChanged(SessionState sessionState)
        {
            Debug.Log($"Session state changed to: {sessionState}");
            if (sessionState != SessionState.Connected)
            {
                Debug.Log("Session disconnected");
                ExitedSession();
            }
        }

        void ExitedSession()
        {
            if (m_CurrentSession != null)
            {
                Debug.Log("Cleaning up session resources");
                m_CurrentSession.RemovedFromSession -= RemovedFromSession;
                m_CurrentSession.StateChanged -= CurrentSessionOnStateChanged;
                m_CurrentSession = null;
            }
        }

        private bool OnWantToQuit()
        {
            Application.wantsToQuit -= OnWantToQuit;

            var canQuit = m_CurrentSession == null;
            if (!canQuit)
            {
                StartCoroutine(LeaveBeforeQuit());
            }

            return canQuit;
        }

        private IEnumerator LeaveBeforeQuit()
        {
            try
            {
                Debug.Log("Initiating session cleanup before application quit");
                LeaveSession();
            }
            catch (Exception e)
            {
                Debug.LogException(e);
            }

            yield return null;
            Application.Quit();
        }

        static string GetRandomString(int length)
        {
            var r = new System.Random();
            var result = new char[length];

            for (int i = 0; i < length; i++)
            {
                result[i] = (char)r.Next('a', 'z' + 1);
            }

            return new string(result);
        }
    }
}

NetworkPlayer:

using System.Collections.Generic;
using Assets.Scripts;
using Unity.Netcode;
using Unity.Services.Authentication;

public class NetworkPlayer : NetworkBehaviour
{
    public static List<NetworkPlayer> Players = new();
    
    public static NetworkPlayer Local { get; private set; }
    
    private NetworkVariable<PlayerConnection> playerConnectionInfo = new();
    
    public string PlayerId => playerConnectionInfo.Value.PlayerId.ToString();

    public override void OnNetworkSpawn()
    {
        if (IsOwner)
        {
            Local = this;
            
            var playerId = AuthenticationService.Instance.PlayerId;
            var playerName = AuthenticationService.Instance.Profile;
            // Each owner will automatically register their PlayerConnection information with this NetworkPlayer instance
            playerConnectionInfo.Value = new PlayerConnection(OwnerClientId, playerId, playerName, NetworkObject);
        }
        
        if (!Players.Contains(this))
        {
            Players.Add(this);
        }
    }
}

NetworkPlayerManager:

using System.Linq;
using Assets.Scripts;
using Unity.Netcode;
using UnityEngine;

namespace DefaultNamespace
{
    public class NetworkPlayerManager : NetworkBehaviour
    {
        private void Update()
        {
            if (Input.GetKeyDown(KeyCode.L))
            {
                LogRpc();
            }
        }

        /// <summary>
        /// Session owner spawns when this in-scene placed NetworkObject has finished spawning.
        /// </summary>
        protected override void OnNetworkPostSpawn()
        {
            if (IsSessionOwner)
            {
                SpawnNetworkPlayer(NetworkManager.LocalClientId);
            }
        }

        /// <summary>
        /// All other clients will automatically spawn their player prefab when they have finished synchronizing
        /// </summary>
        protected override void OnNetworkSessionSynchronized()
        {
            if (!IsSessionOwner)
            {
                SpawnNetworkPlayer(NetworkManager.LocalClientId);
            }
        }

        /// <summary>
        /// Spawns a unique player prefab or the default player prefab.
        /// This could be invoked again if the player has to be re-spawned.
        /// </summary>
        private void SpawnNetworkPlayer(ulong clientId)
        {
            if (NetworkPlayer.Players.All(p => p.OwnerClientId != clientId))
            {
                Debug.Log($"Spawning {clientId}");
                NetworkObject.InstantiateAndSpawn(NetworkManager.NetworkConfig.PlayerPrefab, NetworkManager, NetworkManager.LocalClientId, isPlayerObject: true);
            }
        }
        
        [Rpc(SendTo.Everyone)]
        private void LogRpc()
        {
            Debug.Log($"PlayerCount: {NetworkPlayer.Players.Count}, Networkplayers: {string.Join(",", NetworkPlayer.Players.Select(m => $"{m.OwnerClientId}:{m.PlayerId}"))}");
        }
    }
}

PlayerConnection:

using System;
using Assets.Scripts.Tools;
using Unity.Collections;
using Unity.Netcode;

namespace Assets.Scripts
{
    public struct PlayerConnection : INetworkSerializable, IEquatable<PlayerConnection>
    {
        public ulong ClientId;
        public FixedString64Bytes PlayerId;
        public FixedString64Bytes PlayerName;
        public NetworkObjectReference NetworkPlayerRef;
        
        public PlayerConnection(
            ulong clientId,
            FixedString64Bytes playerId,
            FixedString64Bytes playerName,
            NetworkObjectReference networkPlayerRef)
        {
            ClientId = clientId;
            PlayerId = playerId;
            PlayerName = playerName;
            NetworkPlayerRef = networkPlayerRef;
            Debug.Log($"{this}");
        }

        public void NetworkSerialize<T>(BufferSerializer<T> serializer) where T : IReaderWriter 
        {
            serializer.SerializeValue(ref ClientId);
            serializer.SerializeValue(ref PlayerId);
            serializer.SerializeValue(ref PlayerName);
            serializer.SerializeValue(ref NetworkPlayerRef);
        }

        public bool Equals(PlayerConnection other)
        {
            return ClientId == other.ClientId &&
                   PlayerId == other.PlayerId &&
                   PlayerName == other.PlayerName &&
                   NetworkPlayerRef.Equals(other.NetworkPlayerRef);
        }
        
        public override string ToString()
        {
            return $"PlayerId: {PlayerId}, PlayerName: {PlayerName}, ClientId: {ClientId}, PlayerObjectReference: {NetworkPlayerRef.NetworkObjectId}";
        }
    }
}

Testing Tool

I also have a console app to launch multiple game instances. With the Launch command, you can run multiple instances like this: Launch 6 (will launch 6 instances). You can also specify a delay (e.g., Launch 6 3000), which will launch instances with a 3000ms delay between each. Otherwise, all instances will launch simultaneously.

The first instance will run normally with graphics, while other instances run without graphics to save computer resources. Once you finish testing, you can run the exit command, and it will kill all processes. Each instance creates its own log file in the _logDirectory, which you can check after testing.
You can create an empty .NET 9.0 console project and paste the following code (you should set buildDirectory and exeName in ClientProcessManager):

ClientProcessManager

using System.Diagnostics;
namespace GameInstanceLauncher;

public class ClientProcessManager : IDisposable
{
    private readonly List<Process> _clientProcesses = new();
    private readonly string _buildDirectory;
    private readonly string _logDirectory;
    private readonly string _exeName;
    private bool _disposed;

    public ClientProcessManager(string buildDirectory = @"C:\BuldPath", string exeName = "Game.exe")
    {
        _buildDirectory = buildDirectory;
        _logDirectory = Path.Combine(buildDirectory, "Logs");
        _exeName = exeName;
        Directory.CreateDirectory(_logDirectory);
    }

    public async Task LaunchProcessesAsync(int count, int launchDelay)
    {
        ThrowIfDisposed();
        
        int startIndex = _clientProcesses.Count;

        for (int i = 0; i < count; i++)
        {
            int processIndex = startIndex + i;
            var process = CreateProcess(processIndex);

            if (process != null)
            {
                _clientProcesses.Add(process);
                Console.WriteLine($"Started client {processIndex}");

                if (launchDelay > 0 && i < count - 1)
                {
                    await Task.Delay(launchDelay);
                }
            }
        }
    }

    public void ListRunningProcesses()
    {
        ThrowIfDisposed();
        Console.WriteLine("\nCurrently running processes:");
        
        RemoveExitedProcesses();
        
        for (int i = 0; i < _clientProcesses.Count; i++)
        {
            PrintProcessStatus(i, _clientProcesses[i]);
        }
    }

    public async Task KillProcessAsync(int index)
    {
        ThrowIfDisposed();
        
        if (index < 0 || index >= _clientProcesses.Count)
        {
            throw new ArgumentOutOfRangeException(nameof(index), "Invalid process index");
        }

        var process = _clientProcesses[index];
        await KillProcessAsync(process, index);
        
        if (process.HasExited)
        {
            process.Dispose();
            _clientProcesses.RemoveAt(index);
        }
    }

    public async Task KillRandomProcessAsync()
    {
        ThrowIfDisposed();
        
        // Remove any already exited processes first
        RemoveExitedProcesses();
        
        // If we have 2 or fewer processes (including the first one we want to preserve), return
        if (_clientProcesses.Count <= 1)
        {
            Console.WriteLine("Not enough processes to perform random kill (need at least 2 running processes).");
            return;
        }

        // Generate random index excluding 0 (first process)
        var random = new Random();
        int randomIndex = random.Next(1, _clientProcesses.Count);
        
        Console.WriteLine($"\nRandomly selected process {randomIndex} for termination.");
        await KillProcessAsync(randomIndex);
    }

    public async Task KillAllProcessesAsync()
    {
        ThrowIfDisposed();
        
        Console.WriteLine("\nKilling all processes...");

        var processes = _clientProcesses.ToList(); // Create a copy to avoid modification during enumeration
        var killTasks = processes
            .Where(p => !p.HasExited)
            .Select(p => KillProcessAsync(p));

        await Task.WhenAll(killTasks);

        // Clean up the processes list
        RemoveExitedProcesses();

        Console.WriteLine("All processes terminated.");
    }

    private Process? CreateProcess(int processIndex)
    {
        string exePath = Path.Combine(_buildDirectory, _exeName);
        string logFile = Path.Combine(_logDirectory, $"client_{Environment.MachineName}_{processIndex}.log");

        var startInfo = new ProcessStartInfo
        {
            FileName = exePath,
            Arguments = BuildProcessArguments(processIndex, logFile),
            UseShellExecute = false,
            RedirectStandardOutput = true,
            RedirectStandardError = true,
            WorkingDirectory = _buildDirectory
        };

        try
        {
            var process = Process.Start(startInfo);
            if (process == null)
            {
                throw new Exception($"Failed to start process {processIndex}");
            }
            return process;
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error starting client {processIndex}: {ex.Message}");
            throw;
        }
    }

    private static string BuildProcessArguments(int processIndex, string logFile)
    {
        var argsList = new List<string> { "-testing" };

        if (processIndex > 0)
        {
            argsList.Add("-batchmode");
            argsList.Add("-nographics");
        }

        argsList.AddRange(new[] { "-logFile", logFile });

        return string.Join(" ", argsList);
    }

    private void RemoveExitedProcesses()
    {
        for (int i = _clientProcesses.Count - 1; i >= 0; i--)
        {
            var process = _clientProcesses[i];
            if (process.HasExited)
            {
                process.Dispose();
                _clientProcesses.RemoveAt(i);
            }
        }
    }

    private static void PrintProcessStatus(int index, Process process)
    {
        string status = process.HasExited ? "Exited" : "Running";
        Console.WriteLine($"Process {index}: {status} (PID: {process.Id})");
    }

    private static async Task KillProcessAsync(Process process, int? index = null)
    {
        if (process.HasExited)
        {
            Console.WriteLine(index.HasValue ?
                $"Process {index} has already exited." :
                "Process has already exited.");
            return;
        }

        try
        {
            process.Kill(true);
            await Task.Run(() => process.WaitForExit(1000));

            if (index.HasValue)
            {
                Console.WriteLine($"Process {index} terminated successfully.");
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine(index.HasValue ?
                $"Error killing process {index}: {ex.Message}" :
                $"Error killing process: {ex.Message}");
        }
    }

    private void ThrowIfDisposed()
    {
        if (_disposed)
        {
            throw new ObjectDisposedException(nameof(ClientProcessManager));
        }
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    protected virtual void Dispose(bool disposing)
    {
        if (!_disposed)
        {
            if (disposing)
            {
                // Kill all processes synchronously during disposal
                KillAllProcessesAsync().GetAwaiter().GetResult();
                
                foreach (var process in _clientProcesses)
                {
                    process.Dispose();
                }
                _clientProcesses.Clear();
            }

            _disposed = true;
        }
    }

    ~ClientProcessManager()
    {
        Dispose(false);
    }
}

Program

namespace GameInstanceLauncher;

public static class Commands
{
    public const string Launch = "launch";
    public const string List = "list";
    public const string Kill = "kill";
    public const string KillAll = "killall";
    public const string KillRandom = "killrandom";
    public const string Exit = "exit";
    public const string Help = "help";
}

class Program
{
    private static bool isRunning = true;

    static async Task Main(string[] args)
    {
        using var processManager = new ClientProcessManager();

        try
        {
            await HandleInitialCommand(args, processManager);

            while (isRunning)
            {
                Console.Write("> ");
                string? input = Console.ReadLine();
                if (!string.IsNullOrEmpty(input))
                {
                    await HandleCommand(input.Split(' ', StringSplitOptions.RemoveEmptyEntries), processManager);
                }
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error: {ex.Message}");
        }
    }

    private static async Task HandleInitialCommand(string[] args, ClientProcessManager processManager)
    {
        if (args.Length > 0)
        {
            await HandleCommand(args, processManager);
        }
        else
        {
            DisplayHelp();
        }
    }

    private static async Task HandleCommand(string[] args, ClientProcessManager processManager)
    {
        if (args.Length == 0) return;

        try
        {
            switch (args[0].ToLower())
            {
                case Commands.Launch:
                    if (args.Length > 1 && int.TryParse(args[1], out int count))
                    {
                        int delay = args.Length > 2 && int.TryParse(args[2], out int d) ? d : 0;
                        await processManager.LaunchProcessesAsync(count, delay);
                    }
                    else
                    {
                        Console.WriteLine($"Usage: {Commands.Launch} <count> [delay_ms]");
                    }
                    break;

                case Commands.List:
                    processManager.ListRunningProcesses();
                    break;

                case Commands.Kill:
                    if (args.Length > 1 && int.TryParse(args[1], out int index))
                    {
                        await processManager.KillProcessAsync(index);
                    }
                    else
                    {
                        Console.WriteLine($"Usage: {Commands.Kill} <process_index>");
                    }
                    break;

                case Commands.KillRandom:
                    await processManager.KillRandomProcessAsync();
                    break;

                case Commands.KillAll:
                    await processManager.KillAllProcessesAsync();
                    break;

                case Commands.Exit:
                    await processManager.KillAllProcessesAsync();
                    isRunning = false;
                    break;

                case Commands.Help:
                    DisplayHelp();
                    break;

                default:
                    Console.WriteLine($"Unknown command. Type '{Commands.Help}' for available commands.");
                    break;
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Command error: {ex.Message}");
        }
    }

    private static void DisplayHelp()
    {
        Console.WriteLine("\nAvailable commands:");
        Console.WriteLine($"  {Commands.Launch} <count> [delay_ms] - Launch specified number of clients with optional delay");
        Console.WriteLine($"  {Commands.List}                      - List all running processes");
        Console.WriteLine($"  {Commands.Kill} <index>              - Kill specific process by index");
        Console.WriteLine($"  {Commands.KillRandom}                - Kill a random process (excludes first process)");
        Console.WriteLine($"  {Commands.KillAll}                   - Kill all running processes");
        Console.WriteLine($"  {Commands.Exit}                      - Kill all processes and exit");
        Console.WriteLine($"  {Commands.Help}                      - Display this help message");
    }
}

Testing Workflow

1. Build the game
2. Run the game from the editor and wait until a session is created
3. Run multiple instances from the console app:

• First without delay (e.g., Launch 15)
• Then with delay (e.g., Launch 15 3000)

In both cases, synchronization is not handled correctly. As you can see, NetworkPlayer has a static list of NetworkPlayers, and NetworkPlayerManager has a LogRpc method which logs player count on each client. You can trigger LogRpc by pressing the L key. Then you can check the log files for each client. The console app creates a separate log file for each client. You'll notice different network player counts across all clients, which indicates they are not properly synchronized.

[NoelStephensUnity]

Thank you for the scripts. It will take me a bit to get all of this put together into a project (perhaps this weekend).

Just out of curiosity, do you have the player prefab's NetworkObject marked with distributable permissions?
If you do, then I would set the player prefab to have "None" permissions to assure the service isn't trying to redistribute the player instances (this would be a service bug btw) which could ?maybe? cause this kind of "mixing" of spawned players when you get to a high enough number of instances.

You could also try something like this for your LogRpc:

    [Rpc(SendTo.Everyone)]
    private void LogRpc()
    {
        Debug.Log($"PlayerCount: {NetworkPlayer.Players.Count}, Networkplayers: {string.Join(",", NetworkPlayer.Players.Select(m => $"{m.OwnerClientId}:{m.PlayerId}"))}");
        var connectedClients = NetworkManager.SpawnManager.GetConnectedPlayers();
        var players = new List<NetworkPlayer>();
        foreach(var connectedClient in connectedClients) 
        {
            var playerPrefabInstance = NetworkManager.SpawnManager.GetPlayerNetworkObject(connectedClient);
            if (playerPrefabInstance == null)
            {
                Debug.LogError($"Client-{connectedClient} has no player prefab assigned!");
            }
            else if (playerPrefabInstance)
            {
                if (playerPrefabInstance.OwnerClientId != connectedClient)
                {
                    Debug.LogError($"Client-{connectedClient} has a player prefab assigned but it is owned by Client-{connectedClient}!");
                }
                players.Add(playerPrefabInstance.GetComponent<NetworkPlayer>());
            }
        }
        Debug.Log($"[Secondary] PlayerCount: {connectedClients.Count}, Networkplayers: {string.Join(",", players.Select(m => $"{m.OwnerClientId}:{m.PlayerId}"))}");
    }

Then see if there is a delta between the static list your are building and what NGO is already tracking internally.
I agree that (from what I can see so far) it should return the same values amongst all clients...we do 100 player tests regularly and haven't run into this particular scenario... so I will need to get all of your scripts together in a project and then make another project for your launcher to debug and see what all could be happening.