tests: static test data generation

[jku]

current state:

• most client tests use a repository that is generated live during the test (RepositorySimulator)
• there is (and should be) static test data as well, see tests/repository_data/
• test data was generated with scripts that used the legacy codebase but it's not clear if the scripts and the data are actually up to date (or if the data has been manually modified)
• scripts are now removed as legacy code is removed (Rm all legacy #1790)

We will want to modify the static files at some point (e.g. we should make the files consistent_snapshot), so we should have easy ability to do so. I believe that means we should

• generate the static files with a script that uses metadata API
• possibly verify at test runtime that the script result is unchanged -- this would give us additional confirmation that we are not changing things (even superficially) in our serialisation format.

[jku]

After seeing the 92% speedup 🚀 in test runtime when we drop legacy tests: we could even expand these static test sets, we clearly have the test runtime budget now.

[lukpueh]

Thanks for the effort, @jku! I meant to write a ticket with 1619290 but then thought, let's cross the bridge when we get there, i.e. when we need to regenerate those fixtures. All the better when there's already a ticket.

[jku]

Adding to this: Our serialization tests are pretty good, but they are very "unit testy". Making sure that the output of the whole deserialize-serialize pipeline does not change is not really easy -- and adding test data for complete Metadata objects really stretches the viability of the "test data within test code" approach.

So maybe we should have something like this:

• script that produces a bunch of metadata files (or maybe just bytes)
• test that ensures that deserializing and then serializing the data produces the same data
• test that the output of the script did not change unexpectedly:
  • either the metadata files are stored in git (so humans can easily see what is tested) and the script just checks the produced content is still same
  • or the hashes of the metadata bytes are stored and verified: seeing the test data would require running the script manually

To be clear, this plan only solves the serialization testing issue -- but nothing prevents these same metadata files from also forming real repository versions.

[MVrachev]

One question that comes to my mind is about the purpose of these generation scripts.
What priorities do we have for them:

1. simple easy to understand scripts that generate one set of new metadata sets
2. flexible scripts that can be used to generate a variety of metadata combinations
3. scripts that allow generating and publishing multiple metadata versions (but that does sound like something RepositorySimulator already does)

If 1 is the idea, then it’s clear.
If 2 is the focus here it’s good to think about what kind of metadata combinations do we want to generate.
If 3 is the case then I will use some ideas implemented inside the RepositorySimulator.

[jku]

There's clearly multiple options here. I would start with whatever seems most tangible and easiest to accomplish, and try to document the possible paths forward from there.

So a first draft could look like this:

• a single script that produces several individual metadata files (that do not have to form a repository or be in any way related to each other). The set of metadata does not need to be extensive: this can be left as future work. Documenting what is and what should be covered in the test data should be a priority
• The script and resulting files are both committed to git
• A test is added that runs the script and asserts the results are identical compared to the content of the files in git
• this is going to require at least a bunch of keys stored in git as well (to ensure they are identical): they could be in the script sources maybe?

WRT to your questions:

• flexibility as a goal seems unnecessary to me
• if we start producing actual repositories, that does indeed look like RepositorySimulator: with --dump we could even use it as is (or almost as is, the filenames probably aren't versioned right now for easier diffing). However, his is likely going to need more design, experiments and even new code (how would the repository versions work?)... and I'm not 100% sure if RepositorySimulator is the correct thing to use here: it just happens to be the only "repo implementation" we have in the code base at the moment

[MVrachev]

I made a small prototype in my branch: https://github.com/MVrachev/tuf/tree/test-statics-data-generation.
What do you think @jku?
Where should we go from here?

[jku]

some thoughts from looking at the proto:

• Re-generating the "expected metadata files" needs to be a single command -- developer should not have to guess how to do that: ./generate_test_data could just generate all of the current test files in the correct places. If possible the test failure should explain what happened and also mention how to re-generate the expected results when apropriate.
• because the above means script controls the metadata filenames, I think the tests do not need to know about metadata file names at all (so should not load the files): maybe the test could just call MetadataGenerator.verify() to ask MetadataGenerator check if all currently generated files match the files on disk.
• can you explain the sys.path.append? In tests we just just assume that someone else (tox) ensures that tuf module is the correct version, I think we should be able to do that here too
• With regards to code layout I think current proto is is fine as the initial version but if you want to make it ready for improvements think about what happens when we want to add more metadata? E.g. multiple different unrelated root metadata?
  • Using TOP_LEVEL_ROLE_NAMES as filenames is likely not a good idea (or you at least need a separate directory for each) as we're goign to want multiple versions. Realistically names like root_all_key_types.json or root_non_consistent.json could work?
  • Also unless we test the metadata as a repository we should not create files that look like exactly like a repository: different file naming could help here too?
  • splitting the metadata generation into methods somehow is probably a good idea: one option is a dynamic dispatch so that all methods with name generate_XYZ() will be automatically run when script is run (and could create files in directory generated_metadata_expected/XYZ/: this way a method could not mistakenly overwrite a filename used in another method). Maybe generate_XYZ() could then have a verify flag so the same code either writes files or verifies the same files: this way nothing but the specific method needs to handle the filenames it uses.

Many of the ideas above are things we could do later and might not be very fully fleshed out: getting the re-generation and test cycle working is the main thing here: they need to be something that does not get in the way.

[jku]

also another thought: now that we're creating more and more metadata from scratch, #1459 becomes more and more interesting: see #1459 (comment) especially