WebApplication does not observe changes made with WebApplicationFactory<T> in tests

[martincostello]

Describe the bug

When using WebApplicationFactory/WebApplication in conjunction with WebApplicationFactory<T> for integration tests, the WebApplication created by WebApplicationFactory does not observe configuration changes made to the IWebHostBuilder by WebApplicationFactory<T>.

First, if the configuration is changed using ConfigureAppConfiguration(), the changes are not visible in the IConfiguration resolved by the WebApplication. This applies to adding new sources as well as clearing the existing sources. In both cases, the original configuration remains with the built application, and the changes applied for the tests by the WebApplicationFactory<T> are not visible to the application.

Calling serviceProvider.GetServices<IConfiguration>().Count() within the application returns a value of 2. It appears that both two IConfiguration implementations, one for the web host and another for WebApplication are registered with the service provider, with the WebApplication one being resolved by calls to GetService<IConfiguration>() meaning that the changes are not observed.

This can be worked around by using ConfigureServices() to clear the duplicate registration to leave the one required by the tests:

builder.ConfigureServices(services =>
{
    var descriptors = services.Where(p => p.ServiceType == typeof(IConfiguration)).ToList();
    services.Remove(descriptors[1]);
});

Secondly, changes to the content root are not observed by the application. If UseContentRoot() is used to change the directory to serve content by WebApplicationFactory<T>, the change is not visible to the WebApplication instance, which continues to use the path associated with where the test process was launched from.

This causes static content to not be found when requested.

To Reproduce

An application that reproduces this issue is available at https://github.com/martincostello/WebApplicationFactory-Config-Issues. To reproduce the issue:

1. Clone https://github.com/martincostello/WebApplicationFactory-Config-Issues.git
2. Run build.ps1

The following test failures are then observed:

1. The Can_Override_Configuration test fails because the original value of Value is used from configuration ("a") instead of the one overridden by the tests ("b").
2. The Can_Override_ContentRoot tests fails because a 404 is returned due to the application being unable to find the test.txt file to serve the static content.
3. The Can_Remove_Configuration test fails because the original value of Value is used from configuration ("a") instead of an empty value of "" because all of the configuration sources were removed.

Running the application under the debugger in Visual Studio 2022 and requesting the /api or /test.txt resources in a browser show the application works as expected when run outside of the test project.

Similarly the Application_Returns_Defaults test also works correctly as no overrides are provided.

Further technical details

• .NET SDK: 6.0.100-preview.6.21324.1 (I tried to use the latest preview 7 build but had issues running it in Visual Studio 2022)
• Visual Studio 2022 Preview 1.1

[martincostello]

I'm still having a dig around trying to work out what's going on with this, but I think it might be to do with the fact that WebApplicationBuilder has its own IWebHostBuilder implementation which isn't the one that the configuration callback in WebApplicationFactory<T> gets (it gets a Microsoft.AspNetCore.Hosting.GenericWebHostBuilder), so the test config sort of gets applied to the wrong thing.

[davidfowl]

These issues are related to WebApplicationBuilder/WebApplication and might be related to #33082 but @halter73 might be able to pin point as he's working on this area right now.

The same logic works if you use top level statements with the HostBuilder directly:

Host.CreateDefaultBuilder()
    .ConfigureWebHostDefaults(builder =>
    {
        builder.Configure(app =>
        {
            app.UseStaticFiles();
            app.UseRouting();

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapGet("/api", (IConfiguration config, IHostEnvironment environment, IServiceProvider serviceProvider) =>
                {
                    return new
                    {
                        configurations = serviceProvider.GetServices<IConfiguration>().Count(),
                        environment = environment.EnvironmentName,
                        message = config["Message"],
                        value = config["Value"],
                    };
                });
            });
        });
    })
    .Build()
    .Run();

I had a make a couple of tweaks to your tests but they all passed.

[martincostello]

I think I'll stop banging my head against this one for now and see what Stephen makes of it.

[davidfowl]

@martincostello keep finding these bugs, they are super helpful!

[martincostello]

I think I've gotten as far as I can with my sample app for minimal actions that's how I've flushed out the various issues I've logged. I'll come back to it again when there's fixes available and/or later official preview builds released.

The changes are here if it helps you with any debugging/investigation: martincostello/dotnet-minimal-api-integration-testing#2

Two outstanding issues I've got that I've not logged yet as they're only apparent after several layers of workarounds and hacks:

1. The tests always want to seem on ports 5000 and 5001 despite using builder.UseUrls("https://127.0.0.1:0") - I'm not sure if this is because I've broken something, or it's another symptom of this issue. This was a bug in my app's tests.
2. The Razor page at the root URL no longer renders and just gives a 404 - again, not sure if I broke something or if it's due to the content root not being applied despite the call to UseContentRoot() on the IWebHostBuilder.

Thanks for contacting us.

We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.