How to trigger NGO CodeGen via custom buildpipeline?

[Laumania]

This is a long shot and I'm assuming/guessing things based on what I have researched.
I also know I'm pushing the limits here - but hey somebody have to do it :)

Background

The background for this is that my game supports mods and I'm currently working on adding multiplayer to my game using NGO.

So far things are going pretty good - but there is one major issue I still haven't figured out and it's a really big deal as mod creators will be very limited if I can't get this to work.

I'm using UMod to help support mods in my game (https://assetstore.unity.com/packages/tools/integration/umod-2-0-58293).
As far as I understand, it adds feature on top and around Asset Bundles.

The way UMod work is pretty nice.

As the developer of the game (me) I setup various things, like what scripts should be included in the "Mod Tools" and other settings. I then build the "Mod Tools" for my game from inside Unity. This result in a .unitypackage file.

As the mod creators (people from the community) you then open up a fresh Unity and import this .unitypackage file (Mod Tools) and get some Unity windows to create a "mod" and can then build a mod - which can include custom scripts.

The mod creator then "build mod" from a custom Unity menu made by UMod, which generates a .mod file that the game can then pick up. At least simply put - the important part here is the UMod builds the mod here. Remember that.

The problem

So, I have discovered that ClientRPCs, ServerRPCs and NetworkVariables are NOT working if a modder is using it in his custom scripts.

I have done some research that indicates (and here comes the guessing part) that NGO is doing some CodeGen at "game build time" - I guess here: https://github.com/Unity-Technologies/com.unity.netcode.gameobjects/blob/develop/com.unity.netcode.gameobjects/Editor/CodeGen/NetworkBehaviourILPP.cs

I'm guessing that it doesn't work, because this CodeGen is not happening on the custom scripts in the mod when it's build by UMod.

UMod supports custom BuildProcessors to run custom code when building the mods like this:
using System.Collections.Generic;
using UMod.BuildEngine;
using UMod.BuildPipeline;
using UMod.BuildPipeline.Build;
using UMod.Shared.Resources;
using UnityEngine;

// Add a build processor for script assets since we are only interested in scripts that will be compiled by uMod.
// High priority means that it will run after script compilation so build assemblies collection will already be filled out with compiled assemblies.
[UModBuildProcessor(".cs", 1000, true)]
public class PatchAssembliesExample : BuildEngineProcessor
{
    // Methods
    // Called by the build engine just after compilation if the mod includes any script content, passing all script assets as a parameter (Not interested in those).
    public override void ProcessAssetBatch(BuildContext context, IEnumerable<BuildPipelineAsset> assets)
    {
        // get assemblies
        ModAssemblyEntry[] entries = context.BuildAssemblies.Assemblies;

        // Patch all
        foreach(ModAssemblyEntry entry in entries)
        {
            PatchAssembly(context, entry);
        }
    }

    private void PatchAssembly(BuildContext context, ModAssemblyEntry entry)
    {
        // Name info
        string asmName = entry.AssemblyName;

        // Get assembly image - loadable by 'Assembly.Load'
        byte[] asmImage = entry.AssemblyImage;


        // TODO - Run custom patcher code


        // Get the patched result
        byte[] asmImagePatched = ...;

        // Check for patched
        if (asmImagePatched != null)
        {
            // Remove original assembly
            context.BuildAssemblies.RemoveFromBuild(entry);

            // Register new assembly
            if (context.BuildAssemblies.RegisterAssemblyForBuild(asmImagePatched, true) == false)
                Debug.LogWarning("Failed to register patches assembly for build! : " + asmName);
        }
    }
}

So, here comes the million dollar question that will make the sky the limits for modders of my game - how can I trigger this CodeGen for the scripts in the mod?

Note that NGO is installed as a dependency to the Mod Tools, so NGO is installed in the mod creators Unity project.

[Laumania]

Hi @NoelStephensUnity

Its me again, pushing the limits - you have an idea on this one?

Its REALLY important for my game that I get this to work, else modders will be really limited making multiplayer mods.

[NoelStephensUnity]

@Laumania
Hello! 👋
Hmmm... I am sure it is an order of operations type of issue...
The one thing I am not 100% clear on is whether the user creating the mod is adding additional NetworkVariables and RPCs =or= if they are just extending your classes that already have the NetworkVariables and RPCs and when they extend/mod certain classes that contain NetworkVariables (most likely with custom serializable types) and RPCS those NetworkVariables and RPCs no longer function correctly.

If the later scenario is the case, then this is most likely what is happening:

• User installs your product that already has the generated assemblies with the ILPP code injected
• User then creates a mod based on those assemblies and part of the mod includes "modifying classes" contained within your already built DLLs.
• When UMod "patches the assembly" they are basically removing your package's original assembly and then replacing it with the modified version (or along those lines...where the assembly that contains the injected code is no longer used).

So...unless you are trying to preserve already existing mods (most likely is too late for already UMod generated patches), the best way to avoid losing the ILPP generated code is to create a full abstraction layer between the classes the users will modify and the classes that contain the RPCs and NetworkVariables:

• Your Project's Primary Netcode Class (in primary Assembly/Assemblies) --> i.e. PrimaryAssembly.asmdef

  • SomeCoreNetcodeClass
    • Contains RPCs and NetworkVariables

• Your Project's Customizable Classes (in secondary Assembly/Assemblies)--> i.e. CustomAssembly.asmdef

  • CustomizableClass
    • References SomeCoreNetcodeClass (but cannot derive from!)

You would then most likely need to create some form of generic interface that handles the specific behavior(s) you want to allow users to override/modify...so the CustomizableClass would implement something like ISomeCoreNetcodeClassActions that would then register itself with the referenced SomeCoreNetcodeClass component...then when specific "customizable" actions need to be invoked the SomeCoreNetcodeClass would invoke the registered ISomeCoreNetcodeClassActions.

Where SomeCoreNetcodeClass already has a "default" ISomeCoreNetcodeClassActions implementation in the event nothing is "customized".

Basically you need to only allow UMod to be able to modify assembly/assemblies that only have references to classes in your core netcode enabled assemblies but not modify the assemblies that contain those classes...thus the reason for having different assemblies... one with the compiled NGO derived classes...and one with just the interface implementations (i.e. the default) and the classes that register those implementations with the core netcode enabled classes.

This way UMod will only replace the "default interface implementation" assemblies and not the assemblies with the injected code.

I am a bit heads down currently working on some more updates for NGO, but if you don't hear back from me by the end of this week just ping me and I will try to provide you with a very simple skeleton project of what I am talking about.

[scottyboy805]

@Laumania Hello! 👋 Hmmm... I am sure it is an order of operations type of issue... The one thing I am not 100% clear on is whether the user creating the mod is adding additional NetworkVariables and RPCs =or= if they are just extending your classes that already have the NetworkVariables and RPCs and when they extend/mod certain classes that contain NetworkVariables (most likely with custom serializable types) and RPCS those NetworkVariables and RPCs no longer function correctly.

If the later scenario is the case, then this is most likely what is happening:

• User installs your product that already has the generated assemblies with the ILPP code injected
• User then creates a mod based on those assemblies and part of the mod includes "modifying classes" contained within your already built DLLs.
• When UMod "patches the assembly" they are basically removing your package's original assembly and then replacing it with the modified version (or along those lines...where the assembly that contains the injected code is no longer used).

So...unless you are trying to preserve already existing mods (most likely is too late for already UMod generated patches), the best way to avoid losing the ILPP generated code is to create a full abstraction layer between the classes the users will modify and the classes that contain the RPCs and NetworkVariables:

• Your Project's Primary Netcode Class (in primary Assembly/Assemblies) --> i.e. PrimaryAssembly.asmdef

  • SomeCoreNetcodeClass

    • Contains RPCs and NetworkVariables

• Your Project's Customizable Classes (in secondary Assembly/Assemblies)--> i.e. CustomAssembly.asmdef

  • CustomizableClass

    • References SomeCoreNetcodeClass (but cannot derive from!)

You would then most likely need to create some form of generic interface that handles the specific behavior(s) you want to allow users to override/modify...so the CustomizableClass would implement something like ISomeCoreNetcodeClassActions that would then register itself with the referenced SomeCoreNetcodeClass component...then when specific "customizable" actions need to be invoked the SomeCoreNetcodeClass would invoke the registered ISomeCoreNetcodeClassActions.

Where SomeCoreNetcodeClass already has a "default" ISomeCoreNetcodeClassActions implementation in the event nothing is "customized".

Basically you need to only allow UMod to be able to modify assembly/assemblies that only have references to classes in your core netcode enabled assemblies but not modify the assemblies that contain those classes...thus the reason for having different assemblies... one with the compiled NGO derived classes...and one with just the interface implementations (i.e. the default) and the classes that register those implementations with the core netcode enabled classes.

This way UMod will only replace the "default interface implementation" assemblies and not the assemblies with the injected code.

I am a bit heads down currently working on some more updates for NGO, but if you don't hear back from me by the end of this week just ping me and I will try to provide you with a very simple skeleton project of what I am talking about.

Hi @NoelStephensUnity,
My name is Scott, co founder of Trivial Interactve who develop the uMod 2.0 modding asset mentioned here.
Just wanted to offer some insight into what uMod is doing as part of its compile process and maybe you will have some more ideas with a better understanding of what is happening.

The first point to mention is that uMod does not actually patch any assemblies or game code as such. Instead it supports additive modding which means compiling additional script assemblies that work as add-ons to the game. To do this uMod runs the Roslyn compiler manually when building mods to compile the .csproj files for the mod project, and I guess since this step is done manually and not via the Unity build pipeline, the code gen is not running on those assemblies which is to be expected.

Since uMod offers extensive build events and callback, one solution that could work is to run the code gen manually just after uMod has compiled the scripts into an assembly. That was the sample code above I provided to @Laumania which is a uMod build event, but obviously we don't know how to run the code gen manually yet. I had a quick look at the code gen source and it looks to me like it is probably not designed to be run manually from some source other than the Unity build pipeline. Is that the case or are we missing some public API that we can use to trigger this process? If we can get to run the code gen manually then I think it will be problem solved and issue closed, but I suspect it may not be that easy? And just to be clear, that would need to be invoked on an already compiled managed assembly (.dll). I see there is an ICompiledAssembly interface which I guess can take care of that, but again it is all internal and probably not designed to be used by the likes of us.

I appreciate what you say about abstracting the networking aspect of the code base away so that modders don't need to deal with that at all. It does seems like a suitbale solution but my only concerns would be the amount of extra work that would mean for game developers. Ie. needing to desing an API to abstract away implementation details of the networking while making sure the API is feature complete and usable in the real world. Seems to me to be quite a complex task although it would mean that modders don't need to learn/work with netcode directly which is probably a plus point, and developers could switch networking backends without issue. Pehaps a quick example of such an abstracting layer would be helpful if you have time just to see what sort of thing we are ataling about. I guess it is probably not as simple as modders marking thier code with RPC attributes or anything like that, and instead will need to work directly with the API to send RPC's and serialized data, so a different more manualy sort of setup I would assume.

Anyway, hopefully that extra info can be somewhat helpful to provide a nice solution to this unconevntional issue/use case.
Regards, Scott.

[NoelStephensUnity]

@scottyboy805
Hi Scott,

I agree that the solution I offered isn't an "optimal" path to this type of functionality. What I guess I had excluded from all of that is that currently we are spread a bit thin regarding time allocation due future development in tandem with team members working on some long standing feature requests that are a bit more time intensive.

The biggest question I have is whether this issue applies to @Laumania 's project's assemblies or if this is strictly limited to the scope of user's mod assemblies (or both)? Knowing this would help me focus what to look at specifically when I get a spot of time later today or tomorrow.

[scottyboy805]

@NoelStephensUnity Thanks for getting back so quick. Yes I understand that you probably don't have the resources to devote to such issues so that is no problem. Maybe it can be something to look into in the future but like you say there are probably more pressing issues to deal with first.

I guess this would come down to an issue effecting uMod modders only (The game portion of the code is unaffected by this issue). Essentially when a modder tries to include new C# scripts in a mod, uMod will compile those scripts manually which skips the code gen leaving netcode in what I guess would be a raw/unimplemented state. This process is completly separate from the game (A separate Unity project dedicated for creating a specific mod), so yes it only applies to code created by modders, and modders in general (for games implementing uMod 2.0), not just modders of @Laumania's game.

Hope that helps. Let me know if you need any more info.

[NoelStephensUnity]

@scottyboy805
Ahhh, so that makes more sense... user specific mod with scripts that might implement either @Laumania's game's netcode classes and/or make their own classes will not be able to add additional unique netcode behaviors (i.e. RPCs, NetworkVariables, or anything else requiring ILPP) because of how the scripts are compiled.

So, I am thinking the only real way to do that is to be able to invoke the postprocessors after you have compiled the scripts which would just require you to have access to all of the internal classes here:
https://github.com/Unity-Technologies/com.unity.netcode.gameobjects/tree/develop/com.unity.netcode.gameobjects/Editor/CodeGen

I will noodle over this a bit more, but I think exposing some of the CodeGen classes could provide enough to possibly make it possible to handle this in the PatchAssembliesExample.PatchAssembly method...
The post processors basically need something like:

namespace Unity.CompilationPipeline.Common.ILPostProcessing
{
    public interface ICompiledAssembly
    {
        InMemoryAssembly InMemoryAssembly { get; }

        string Name { get; }

        string[] References { get; }

        string[] Defines { get; }
    }
}

Need to look at this side of things a bit more...

[scottyboy805]

@NoelStephensUnity Awesome, yes exposing some of the internal classes would solve the problem if that is not too much hassle. Certainly implementing something like the above interface will be ideal and will be easy for us or even developers themselfs to extend, since uMod already allows access to such information in a build event.
Alternativley if it is too much work to expose some of the API or would break the intended usage etc, then even just a simple helper method like byte[] CodeGenUtil.PatchExternalAssembly(byte[] asmImage) could work for this purpose just as an example, but what ever you think if best.

Thanks for all the help.

[Laumania]

@scottyboy805 @NoelStephensUnity I'm so happy to see that this might be possible - as said - the sky will be the limit if my mod community can use RPCs and NetworkVariables :D

@scottyboy805 If we knew what method to call during the UMod build pipeline, couldn't we maybe use reflection to call it - just a a way to check if it actually works?

But of cause, some more bulletproof solution is needed.

Really excited about this progress! :D

[scottyboy805]

@Laumania Yeah looks like it will make life a lot easier if this can be added.
Reflection is an idea (although a fragile one) but I don't think it can work in this case since it looks like the API will require an implementation of ICompiledAssembly, which can't be done from reflection, so some public API will need to be added I would think to get this all working nicley.

[Laumania]

Hi @NoelStephensUnity

Anything new on this? It's becoming more and more limiting for my modders that they can't do this :)

[guanaco0403]

Hi @NoelStephensUnity

Anything new on this? It's becoming more and more limiting for my modders that they can't do this :)

Yes I confirm am one of them

[Laumania]

Hi @NoelStephensUnity

Happy New Year :)

Any news on this one - because it's really really needed as modders are very limited without this.

[guanaco0403]

Yeah as one of the modders i really need to be able to use at least the NetworkVariables, its really boring that a lot of my ideas cant happen due to that @NoelStephensUnity