Enable snapshotting / auto-generating ExpectedHtml

[egil]

I could be useful to enable snapshotting in the library. That would could help make tests even easier to create in a "test-after" scenario, and will make it easier to do automated regression testing. How it should work needs to be designed, but these are my initial thought:

• <Fact/> gets an parameter EnableSnapshotting, and when set to true, it will automatically generate the <ExpectedHtml> section inside the <Fact/> component, if its doesn't exist already.
• <ExpectedHtml> could get an an parameter that indicates when it was generated (commit id, date time?)

Two challenges with this:

1. How to get the location of the Test.razor file. This problem seems very environment dependent, e.g. the .razor files might be in different locations depending on the environment (local dev computer, build server, etc.). We need a good strategy here!
2. Parse and replace the proper sections of the Test.razor. This part doesn't seem hard, using some search and replace for the element.

This is originally an feature request from @chanan.

[egil]

The first challenge of finding the source files can be solved by not running the snapshot generation during test execution, but during build, using a MSBuild task or as a dotnet tool. That actually seems like a better solution, since it should be a one time thing to generate snapshots, while running tests is something that happens over and over again, also on CI servers.

One issue with a MSBuild task, if it runs on every build, is that with many test razor files in the project, the build time could be affected, if all files has to be inspected on every build. There are probably ways around that, but maybe a simple tool that you run on-demand is the simplest approach for now.

[egil]

Consider using @SimonCropp's Verify to enable this functionality if possible.

Verify enables snapshot like testing in C# based tests, so it does not seem to be an overlap here, since this type of snapshot testing is based in Razor files. @SimonCropp, feel free to correct me.

Also, it might be worth extracting razor based snapshot testing into its own sub library, since it is currently web specific, or at least place it in bunit.web (see #75).

[SimonCropp]

@egil more than happy for u to take some code from verify. note that some is already packaged up to be re-used by other snapshot libs, eg https://github.com/SimonCropp/Verify/blob/master/docs/diff-tool.md#diffengine https://github.com/SimonCropp/TextCopy https://github.com/SimonCropp/emptyfiles

happy to get on a call to discuss if u want

[SimonCropp]

another option is i could transfer ownership of the current verify.bunit project+package to you https://github.com/SimonCropp/Verify/tree/master/src/Verify.Bunit.

[egil]

...
happy to get on a call to discuss if u want

@SimonCropp thanks, I think I will take you up on the offer once I get a better understanding of how snapshotting should actually work. I have not spend much time with that feature, and its great that your library now has support such that I can get some experience.

another option is i could transfer ownership of the current verify.bunit project+package to you https://github.com/SimonCropp/Verify/tree/master/src/Verify.Bunit.

I think it is better that it stays with you and the Verify project, at least while bunit doesn't have a dependency on Verify. Thanks for the offer though.

[egil]

Based on my preliminary work on https://github.com/egil/SourceFileFinder, I think there is a possible solution for auto generating the expected output, that can work like this:

1. User creates a .razor test component with one or more snapshot tests inside, where only the test input is defined, e.g.: <SnapshotTest><TestInput>....</TestInput></SnapshotTest>

2. When the test runs, the source file is found using SourceFileFinder, and the rendered output from is added as <ExpectedOutput> inside each test.

   The rendered output should probably be cleaned up/normalized, and have the special blazor attributes removed, e.g. for tracking event handler bindings, which the differ will ignore anyway.

3. When the test runs again, the runner sees that there is an <ExpectedOutput> output, and does a comparison instead, and report any errors as usual.

To update a snapshot, e.g. the expected output, users just have to delete the existing expected output markup. Will that be enough or should we think of a mass-update feature?

@chanan, @SimonCropp, both of you have more experience with snapshot testing than I do. What do you think about this?

[SimonCropp]

@egil sorry i am having trouble groc'ing this. perhaps a vid or a call would be better to show your idea?

[egil]

@SimonCropp that would be lovely. Lets see if we can set that up if needed. Here is a little more background:

Instead of writing tests in .cs files, you can write them in .razor components. The basic structure is this:

@inherits TestComponentBase

<SnapshotTest Description="Test 1">
  <TestInput>...</TestInput>
  <ExpectedOutput>...</ExpectedOutput>
</SnapshotTest>

<SnapshotTest Description="Test 2">
  <TestInput>...</TestInput>
  <ExpectedOutput>...</ExpectedOutput>
</SnapshotTest>

When the test runs, bunit uses AngleSharp Diffing to compare the rendered output of the <TestInput> element with that of the <ExpectedOutput> element.

This is a complete razor snapshot test:

@inherits TestComponentBase

<SnapshotTest Description="A todolist with one todo added should render correctly"
              Setup="() => Services.AddMockJsRuntime()"
              SetupAsync="() => Task.CompletedTask">
    <TestInput>
        <TodoList Label="My label" Items=@(new Todo[]{ new Todo{ Id=42, Text="Check out this new thing called Blazor" } })>
            <ItemsTemplate Context="todo">
                <TodoItem Todo="todo"></TodoItem>
            </ItemsTemplate>
        </TodoList>
    </TestInput>
    <ExpectedOutput>
        <form>
            <div class="input-group">
                <input type="text" class="form-control" placeholder="My label" aria-label="My label" value="" />
                <div class="input-group-append">
                    <button class="btn btn-secondary" type="submit">Add task</button>
                </div>
            </div>
        </form>
        <ol class="list-group">
            <li id:regex="todo-42" class="list-group-item list-group-item-action">
                <span>Check out this new thing called Blazor</span>
                <span class="float-right text-danger">(click to complete)</span>
            </li>
        </ol>
    </ExpectedOutput>
</SnapshotTest>

Currently, the <ExpectedOutput> part has to be created manually by the user. So it is missing the "snapshot" part of snapshot testing :)

[SimonCropp]

ok take this feedback with a grain of salt as i have been using the "first test run generates the approved file" for a few years.

I can ever see myself going to the effort of crafting the TestInput content. and i can see a way of easily generating that node when it is embedded in xml

[egil]

Indeed. It should be pretty easy to find the right test and insert the markup.

Adding it yourself isn't super hard though. Just run the test and copy the output of the test error message.

[SimonCropp]

Adding it yourself isn't super hard though. Just run the test and copy the output of the test error message.

if u think that is something people will be happy to maintain, then yep this looks like an acceptable approach.

[egil]

Adding it yourself isn't super hard though. Just run the test and copy the output of the test error message.

if u think that is something people will be happy to maintain, then yep this looks like an acceptable approach.

I think it is much easier than typing it out by hand. I also think that it should be as easy as possible. Thus, your input is very welcome indeed.

My current thinking is this:

Initial running of test:

1. User adds a snapshot test with only the test input provided, and possible setup. E.g.
   <SnapshotTest Description="Test 1"><TestInput>...</TestInput></SnapshotTest>
2. When the test runs the first time, the <ExpectedOutput> is added to the test case. In addition, a SnapshotGenerated="DateTime" field could be added to make it easy for the user to see when the snapshot was generated.
   This can be done using AngleSharp, which can parse the Razor file like it was HTML, and then allow us to query the elements in the HTML and update them. It is also able to generate a pretty output again, so that is probably the best option.

Component changes and test breaks:

The easy solution is to simply have the user delete the <ExpectedOutput> and run the test again to have a new one generated. This works well with the Visual Studio interface as is, without custom VS extensions.

I think it is possible to pass parameters to dotnet test that the test runner can pick up on, such that it could be possible to call dotnet test /p:UpdateSnapshotTests to have all failing snapshot tests updated without individually having to go through each test and delete the <ExpectedOutput>.

How does this sound to you?

[egil]

Consider replacing SourceFileFinder with C# Source Generators as the solution for this: https://devblogs.microsoft.com/dotnet/introducing-c-source-generators/

[egil]

Since I have deprecated the <SnapshotTest> components and am instead recommending leaning on Verify.Bunit instead, ill close this issue.