Proposal: Handle multiple versioned WinRT packages loaded side-by-side in a process.

[DefaultRyan]

Versioned WinRT components side-by-side

This feature provides conventions for framework package authors to enabled loading multiple versions of a WinRT component side-by-side within a single process.

Background

When WinRT types were shipped as part of the OS, the versioning largely consisted of putting new functionality into a new interface, and not changing the previously shipped interfaces. When it came time to activate a runtimeclass, there was only one choice: the version of the class supplied by the OS.

As Project Reunion grows and evolves, more WinRT-based functionality will become available and consumed from framework packages shipped separately from the OS. In some ways, this can make versioning simpler. Because framework packages use a scheme whereby a different major version of a package is an entirely different package, I don't need to ensure that Widgets3 is ABI-compatible with Widgets2. (Gratuitous breaking changes are still hostile to developers, but updating your app to consume a newer version involves a recompile).

In other ways, however, this complicates the versioning story. In particular, MyApp might be consuming multiple components, each of which depend on a different version of a framework package. In short, something that looks like this:

MyApp.exe
|
+-SomeComponent.dll
| |
| +-Widgets2.dll (version 2 of Widgets)
|
+-OtherComponent.dll
  |
  +-Widgets3.dll (version 3 of Widgets)

Axiom 1: Framework packages will ship, make breaking changes, and ship again with a newer major version.

Axiom 2: It will sometimes be infeasible, if not impossible, for an app to update its dependencies so that all components can agree upon a single major version of a package.

If the types in different versions of a WinRT component have the same fully qualified names (i.e. namespace + type name), this will lead to ambiguities in various parts of the runtime.

All WinRT type activation goes through RoGetActivationFactory, and that API accepts a string holding the name of the activatable class - there is no way to supply the version or filename of an assembly or dll. This will introduce ambiguitiy to activation.

If C#/.NET attempts to load multiple versions of identically-named types from multiple assemblies, I've been told the developer is going to Have A Bad Time.

Description

A package author can resolve these ambiguities and issues by adopting a versioning convention for their type names. This can either be done manually, or simplified with enlightened tooling.

The crux of the approach is to suffix a type's namespace with a string containing the major version number.

This approach resolves ambiguities with activation because the type names involved are now unique. A big advantage is that this can work with existing tooling, today. But we can enlighten our tools to improve the developer experience.

We can enlighten the MIDL compiler to use this versioning scheme via a new command line option, so that it appends the namespace automatically using the value supplied to the ContractVersionAttribute or VersionAttribute.

Similarly, we can enlighten projection tools (cppwinrt.exe, abi.exe) to recognize this versioning scheme and reduce developer friction by eliminating the need to spell out the versioned namespaces.

Examples

Englightened toolchain

With an enlightened toolchain, a package author can use the Version (or ContractVersion) attribute to specify the package version.

namespace Widgets
{
    [version(3.0)]
    runtimeclass Doodad // Puts Doodad into the _v3_ namespace
    {
        // Methods
    }
}

An enlightened projection tool can then use its knowledge of this versioning scheme to improve the developer experience. For example, a C++ projection could use inline namespaces to "hoist" the latest version into the top-level namespace.

// C++/WinRT projection
namespace winrt::Widgets
{
    inline namespace _v3_
    {
        struct Doodad;
    }
    
    namespace _v2_
    {
        struct Doodad;
    }
}

winrt::Widgets::Doodad b; // Uses _v3_::Doodad
winrt::Widgets::_v2_::Doodad b2; // _v2_::Doodad is still reachable
winrt::Widgets::_v3_::Doodad b3; // _v3_::Doodad is also reachable

Unenlightened toolchain

Even with an unenlightened MIDL compiler, an author can manually add the namespace versioners.

namespace Widgets._v3_ // Version 3.x of Widgets
{
    runtimeclass Doodad
    {
        // Methods
    }
}

An unenlightened projection tool wouldn't be able to perform any namespace hoisting, but types would still be visible, and most languages of interest have mechanisms to remove some of the friction manually.

// C++/WinRT projection
namespace winrt::Widgets
{
    namespace _v3_
    {
        struct Doodad;
    }
    
    namespace _v2_
    {
        struct Doodad;
    }
}

// No projection magic, so add a "using namespace"
using namespace winrt::Widgets::_v3_;

winrt::Widgets::Doodad b; // Uses _v3_::Doodad
winrt::Widgets::_v2_::Doodad b2; // _v2_::Doodad is still reachable
winrt::Widgets::_v3_::Doodad b3; // _v3_::Doodad is also reachable

Other benefits

A feature worth noting is that if MIDL3 syntax is used so that the MIDL compiler automatically generates the interface UUIDs, the different namespace will force different UUIDs to be generated, preventing consumers from accidentally performing invalid conversions between the two versions.

winrt::Widgets::_v2_::Doodad f();
winrt::Widgets::_v3_::Doodad obj = f(); // Error

None of the above, however, precludes a package author from opting in and providing consumers the ability to convert between versioned types. This could take different forms, depending on the degree of interoperability desired by the author.

One approach would be for a newer version of a runtimeclass to implement the older versions of its interfaces, so that it responds to QueryInterface requests.

Another approach would be to provide an overloaded constructor or static method that could convert a newer versioned object (or wrapper) around an older object.

Remarks

Alternative approaches

Explicitly supply the desired DLL to activation

One considered approach would be for versioned WinRT types to use the same namespace between versions, and detour/augment RoGetActivationFactory so that callers would supply the desired version of the DLL. Something like:

HRESULT hr = RoGetActivationFactory(L"Widgets3.dll!Widgets.Doodad", iid, ppResult);

An appealing aspect of this approach is the apparent simplicity.However, this approach was rejected, because putting all the versioned WinRT types into the same namespace created many ambiguities:

• C#/.NET present serious hurdles to get this to work. Even if it got "working", it lays plenty of traps and pitfalls to catch non-expert developers, who will doubtless wonder why their compiler says they can't convert from Widget.Doodad to Widget.Doodad.
• If versioned types have the same name and namespace, things get very ugly whenever both types need to coexist in the source code for the same module. This is a situation that's easy to create if an app's components expose any APIs that use the versioned type:

// Component1.h
Widgets::Doodad foo(); // Uses Widgets 2.0


// Component2.h
Widgets::Doodad bar(); // Uses Widgets 3.0

// I've got a bad feeling about this...

What type is the object under the hood of the Doodad object? Is it v2 or v3? How would we even get this to compile without encountering "type redefinition" errors. Even if we did compile successfully, we'd be in a fragile state of affairs, flirting with runtime Undefined Behavior caused by violations of the "One Definition Rule".

[DefaultRyan]

Let's discuss concerns about namespace changes driving code changes. There is going to be tension between keeping the names the same, and having type collisions.

[DefaultRyan]

Forcing consumers to rename types may be a bridge too far, and hamper adoption to an unacceptable degree.

[jeffstall]

Here's our highest priorities for Xaml and IXP:

1. It must work with C#, especially with .NET Native for UWP apps until either Cs/WinRT can be supported for .NET Native, or UWP can support .NET 5.
2. It must not require any C# code changes to use, but it can involve updating my tool chain to refer to different metadata references. C# doesn't have the "using namespace" inlining support or other language features that's being proposed for C++.
3. It doesn't need to support a single DLL that refers to both Muffin4 and Muffin5. It's actually good if we don't allow this, as it prevents completely unsupported interop between different versions.

[DrusTheAxe]

@jeffstall Agreed on 1+2. 3 is unclear

3. It doesn't need to support a single DLL that refers to both Muffin4 and Muffin5

I'm not following

As in, I the component author don't need to make 1 dll providing both Muffin4 and Muffin5 objects?

Or, I the component consumer don't need to make 1 dll consuming both Muffin4 and Muffin5 objects?

If the latter, is it I don't need to write a dll that directly consumes both Muffin4 and Muffin5 objects? e.g. I don't need to write

var m4 = new Foo._v4_.Muffin();
var m5 = new Foo._v5_.Muffin();

Or is it I don't need to write a dll that directly or indirectly consumes both Muffin4 and Muffin5 objects? e.g. I don't need to write

fee.DoThis(); //calls code that calls code that eventually does stuff with Muffin4
fie.DoThat(); //calls code that calls code that eventually does stuff with Muffin5

Please clarify

[jevansaks]

First, I'm not convinced this is a problem that needs to be solved. In particular in the recommended developer scenario where packages are consumed through nuget and version dependencies are expressed through nuget then the scenario outlined occurs when the developer of "MyApp" updates the nuget version of "OtherComponent" to the version where it now depends on "Widgets.3.0.0" but it hasn't updated "SomeComponent" and "SomeComponent" still depends on "Widgets.2.0.0". If SomeComponent requires exactly Widgets.2.0.0 then "MyApp" will fail to compile. If SomeComponent says it requires [2.0, *) then MyApp compiles and all nuget dependencies resolve to Widgets.3.0.0. (and SomeComponent should only do that if it's able to be recompiled from source ... which maybe we can do for .NET things that are less ABI-dependent).

Ignoring that, there's still more things I worry about. For example, I believe that the activatable class entries list simply the dll name and not the full path. So if both Widgets framework packages say their implementation dll is "Widgets.dll" then LoadLibrary will just pick the first one. We would need to change the dll names too, in addition to the namespace.

And then in practice, how many of our libraries would reasonably work side-by-side like this? Interop is going to come up where I want to pass a "Muffin" output from SomeComponent into OtherComponent and I'll get weird errors like "Muffin is not a Muffin".

Is there a concrete scenario from Reunion that's driving this requirement?

[DefaultRyan]

@jeffstall Agreed on 1+2. 3 is unclear

1. It doesn't need to support a single DLL that refers to both Muffin4 and Muffin5

I'm not following

As in, I the component author don't need to make 1 dll providing both Muffin4 and Muffin5 objects?

Or, I the component consumer don't need to make 1 dll consuming both Muffin4 and Muffin5 objects?

If the latter, is it I don't need to write a dll that directly consumes both Muffin4 and Muffin5 objects? e.g. I don't need to write

var m4 = new Foo._v4_.Muffin();
var m5 = new Foo._v5_.Muffin();

Or is it I don't need to write a dll that directly or indirectly consumes both Muffin4 and Muffin5 objects? e.g. I don't need to write

fee.DoThis(); //calls code that calls code that eventually does stuff with Muffin4
fie.DoThat(); //calls code that calls code that eventually does stuff with Muffin5

Please clarify

@DrusTheAxe -- It applies to authoring a single dll (or exe) that produces or consumes both Muffin4 and Muffin5.

[DefaultRyan]

@jevansaks I'll leave it to @jeffstall to chime in and clarify the need and driving scenarios, as he's my primary (sole?) customer of this proposal.

Regarding your versioning scenario, I've been informed that the convention for framework packages making a breaking major version change is to incorporate the major version number into the package name. So, in you scenario, Widgets.2.0.0 and Widgets.3.0.0 are entirely different packages. In this case, I'm not aware of anything blocking compilation currently, but I wouldn't expect any activation to work.

For requirement 2 stated by @jeffstall , I was trying to figure out what happens if we get a situation where two different assemblies name the same fully-qualified type, and I learned about the extern alias C# feature. So even if the types were to collide, there are ways to mitigate it.

[DrusTheAxe]

So if both Widgets framework packages say their implementation dll is "Widgets.dll"
Correct. For multi-version access to work you need to rename ALL identifiers. What that means varies depending on tech:

• Flat-C DLL exports -- unique DLL filenames e.g. Muffins4.dll vs Muffings5.dll (1)
• InProc COM -- unique DLL filenames and unique coclass and (esp) interface symbol names in IDL (2)
• InProc WinRT -- unique DLL filenames and unique ACIDs
• OOP COM -- unique coclass and (esp) interface symbol names in IDL (2)
• OOP WinRT -- unique ACIDs

Plus of course package family names must differ, when these are in MSIX Framework packages. Optional+Resource packages too. Only Main packages don't matter, since 1 user cannot have 2 Main packages in the same package family registered at the same time, thus no dupe-name collision.

Visual C++'s solved this for decades by including a version identifier in the CRT's filenames e.g. MSVCRT42.dll (yeah, I'm old :P)

(1) You can get around the same-dll-filename-doesn't-load-twice issue using SxS Fusion manifests, in some cases. That doesn't work for UWP apps as SxS's DLL redirector isn't part of the DLL Search Order for Universal packaged processes. [It is for PackagedClassicApp aka DesktopBridge processes, and of course non-packaged processes]. But this is a weird game to begin with. SxS is good for RegFreeCOM, RegFreeWinRT and some oddballs settings (elevation and HighDPI support come to mind) but the DLL Redirector is generally unpopular. Comctrl32 is the primary case I know of its use. VC2005+2008 CRTs relied on it too (and VS found the net experience sufficiently unrewarding they backed away from it in later releases)

(2) Technically interfaces are uniquely identified by IID, but it's hard to write source code with identical symbols for different things, e.g. when IMuffin v4 and v5 both say "interface IMuffin" in Muffin.idl...

[DrusTheAxe]

convention for framework packages making a breaking major version change is to incorporate the major version number into the package name

Yup. Common technique since waaay back in Win8 when MSIX (aka APPX) was invented. WinJS used to consider every release (include previews) a breaking change so they even had package names like microsoft.winjs.1.0.rc1. (One concern they had was monkey patching so implementation changes were effectively breaking changes)

Unique source names are necessary-but-insufficient to concurrently reference multiple versions

[BenJKuhn]

I've started a pull request in the xlang repo with a more detailed specification and specific design recommendations for the various tools & projections that contribute to the experience.

Let's continue the discussion here: microsoft/xlang#697