Skip to content

Design test strategy #1579

@joshuagl

Description

@joshuagl

Description of issue or feature request:

The following list captures some areas of interest for testing and some example test cases for each area. For each area it would be useful to:

  1. better flesh out what we are trying to test, what is not in scope and generate a(n initial) list of tests to implement
  2. research and propose an appropriate test paradigm for the area which prioritises readability and maintainability of the tests
  3. given 1 and 2, design appropriate test suite

Test areas:

  • Expected error cases
    • Specification defined error cases (need to extract from the specification)
    • The known attacks on package managers (define test scenarios for each attack)
    • Python-tuf implementation error cases (review new code and identify error sequences which are testable)
  • Specific sequences the client should be able to handle – different types of updates sequences that may exist in a repository and which the updater should be able to handle
    • Key rotations
    • Recovery from fast-forward attack
    • Client that has been offline for N expirations of root metadata
    • Air-gapped client outside of timestamp expiration window
    • Some of the previously reported security issues?
    • etc
  • Configuration permutations
    • Ensure we test all the relevant sequences of events with and without consistent snapshots
    • Identify permutations of default config to test (where they are not already adequately tested by the unit tests)
  • Table testing for a set of defined tests with different parameters, i.e.
    • target paths (input to get_one_valid_targetinfo())
    • root rotations
    • different complexity delegation trees – both the client and repository side of the delegation tree operations
      • idea: describe delegations in Graphviz dot format?

Metadata

Metadata

Assignees

Labels

backlogIssues to address with priority for current development goalstesting

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions