Test expired metadata from cache #1707

ivanayov · 2021-12-03T08:49:48Z

This tests that an expired timestamp/snapshot/targets when loaded
from cache is not stored as final but is used to verify the new
timestamp

Addresses #1681

The code follows the Code Style Guidelines
Tests have been added for the bug fix or new feature
[n/a] Docs have been added for the bug fix or new feature

ivanayov · 2021-12-03T08:49:58Z

I'm leaving this with the non-matching description on purpose, as the expected behaviour in
the issue description is to load expired snapshot/targets, which is not the case. Not loading them
is actually correct from what I understand from the spec - you shouldn't be able to use outdated timestamp for verification, thus you cannot verify snapshot and targets as well. Please, correct me if I misunderstand.
So we could ether reword the requirement, of if there's a bug we can fix it.

coveralls · 2021-12-03T08:56:17Z

Pull Request Test Coverage Report for Build 1865335270

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

2 of 2 (100.0%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 98.574%

Totals
Change from base Build 1852769022:	0.2%
Covered Lines:	1095
Relevant Lines:	1107

💛 - Coveralls

tests/test_updater_with_simulator.py

MVrachev · 2021-12-07T14:08:37Z

I have some ideas about this pr and will have a conversation with Ivana soon.

jku · 2021-12-08T14:03:24Z

the expected behaviour in
the issue description is to load expired snapshot/targets, which is not the case. Not loading them
is actually correct from what I understand from the spec

This may have been clarified already but I'l say it here:

We do load expired metadata with one big caveat: the expired metadata can only be used to validate a newer version of the same type of metadata. In practice:

local, possibly expired, metadata is loaded and used to validate newer downloaded metadata. "validate" here means signature verification for root and various version checks for timestamp, snapshot and targets.
even the downloaded versions may be expired (in practice only root is, because we download all root versions up to the current, hopefully non-expired version)
the python-tuf implementation will fail the update once the process moves to the next type of metadata and the previous metadata is still expired (e.g. if final root version is expired, refresh() will fail when it tries to validate timestamp)

jku · 2021-12-13T09:44:17Z

the core issue here is that verifying that local files are opened does not mean they are actually used to verify new downloaded metadata (this is made clear even in the test itself: it opens targets.json but we know that's not used for anything). This does not mean the test is worthless but at least some descriptions in the PR and commit are not describing it correctly. I would like to see a clear explanation of what the test does so that
A) we can be sure we don't already have a test that does the same checks
B) future readers won't be deceived by inaccurate descriptions
I get that this may be the hardest part of this PR: let's talk if you don't have a good idea already

Just to be clear: I think this is a useful test that probably does not exist yet... but test_updater_top_level_update.py has some clearly related tests like test_new_timestamp_expired(): maybe it should have a companion test_local_timestamp_expired()? I don't have strong opinions here but maybe @sechkova has ideas about where to put tests like this -- and whether other tests exist already?

Also (and this is a small detail in comparison), I've been trying to advocate not meddling with the client metadata cache manually: in this case I think it would be better to not do that and instead mock current time (pretend that time now > expiry). Search for mock_time in tests for examples of this.

tests/test_updater_with_simulator.py

sechkova · 2021-12-14T12:53:34Z

Just to be clear: I think this is a useful test that probably does not exist yet... but test_updater_top_level_update.py has some clearly related tests like test_new_timestamp_expired(): maybe it should have a companion test_local_timestamp_expired()? I don't have strong opinions here but maybe @sechkova has ideas about where to put tests like this -- and whether other tests exist already?

I admit I find it hard to follow the intentions here, I am sorry if I've put a misleading issue description.
If I understand correctly, test_trusted_root_expired from test_updater_top_level_update.py is doing something very similar and can be used as an example.

ivanayov · 2021-12-14T13:40:14Z

@sechkova the idea was to simulate an expired metadata in cache without manually modifying the cache files. I might have misunderstood this mock - the intention is to simulate change in system time, so that what's stored in cache is considered expired, without need of directly touching the local metadata.

jku · 2021-12-14T14:46:34Z

I admit I find it hard to follow the intentions here, I am sorry if I've put a misleading issue description. If I understand correctly, test_trusted_root_expired from test_updater_top_level_update.py is doing something very similar and can be used as an example.

It is pretty similar, except that I think we don't actually persist downloaded non-root metadata if it's expired, even though we probably could (rado pointed this out to me -- I had not noticed). This difference isn't that important for actual working repos but for testing it means we need to mock system time (or utcnow() as you point out) to "make the local metadata expired".

sechkova · 2021-12-14T15:34:04Z

I admit I find it hard to follow the intentions here, I am sorry if I've put a misleading issue description. If I understand correctly, test_trusted_root_expired from test_updater_top_level_update.py is doing something very similar and can be used as an example.

It is pretty similar, except that I think we don't actually persist downloaded non-root metadata if it's expired, even though we probably could (rado pointed this out to me -- I had not noticed). This difference isn't that important for actual working repos but for testing it means we need to mock system time (or utcnow() as you point out) to "make the local metadata expired".

Thanks, now I think I got all pieces together. So I agree, test_updater_top_level_update.py could host the additional tests for loading expired snapshot and timestamp and since expired metadata and timestamp are not persisted, mocking seems to be the way.
I think expired local "targets" should be treated the same way as an invalid (unsigned) local metadata and may be part of another set of tests?

jku · 2022-01-11T07:57:49Z

I'm marking this a draft since the issue pointed out by Teodora (mocking the wrong thing) is still there: I might have other comments but I've been postponing review until the basics are there. I'm available for a chat if anything is unclear.

ivanayov · 2022-02-03T22:19:10Z

@jku I changed to mocking datetime as Teodora advised, so it's now available for review.

jku

marking request changes so it's visible on PR list: we talked about this over zoom, and found an issue with the timeline of expiries and the mocked refresh: ivana promised to fix

ivanayov · 2022-02-18T16:38:43Z

I've pushed an update.

jku

Nice, all test code seems correct and clear.

Left some minor tweak suggestions in code
These tests should have a docstring to explain what they are testing: they are some of the trickiest tests we have. Something like "Local timestamp must be used in rollback checks even if it is expired" -- maybe the same text that's already before the actual assertRaises

Additionally maybe the dosctring should also include a timeline, example of one test:

  * day 0: first updater refresh
  * day 0 (a bit later): repository publishes timestamp v2
  * day 7: timestamp v1 expiry
  * day 18: second updater refresh: assert that rollback check uses expired v1 here
  * day 21: timestamp v2 expiry

not sure about that last thing though: it's a lot of lines... but may be helpful to understand what's going on? I'll leave decision to you

tests/test_updater_top_level_update.py

This tests that an expired timestamp/snapshot/targets when loaded from cache is not stored as final but is used to verify the new timestamp Fixes theupdateframework#1681 Signed-off-by: Ivana Atanasova <[email protected]>

This change verifies that when local metadata has expired, it is still used to verify new metadata that's pulled from remote Signed-off-by: Ivana Atanasova <[email protected]>

This change fixes the expired metadata tests to mock `datetime` as previously they mocked `time` incorrectly, which did not affect update methods, as they use `datetime.datetime.utcnow()` to calculate now Signed-off-by: Ivana Atanasova <[email protected]>

This change improves the logic of expired metadata tests, so that it is explicitly visible what the expiry time and the versions are and when update/refresh is called in that period Signed-off-by: Ivana Atanasova <[email protected]>

ivanayov · 2022-03-18T20:03:17Z

Thank you @jku , I added docstrings to all the test methods for clarification.

jku

my comment about reusing the datetime.datetime.utcnow() result is marked resolved but all functions still call utcnow() twice...

That is a trivial complaint so let's go with this version, I think it's a good set of very tricky tests. Thanks.

ivanayov · 2022-03-21T12:25:34Z

@jku ah, I resolved by inertia with the microseconds updates... will note to address in following changes. Thanks

MVrachev reviewed Dec 3, 2021

View reviewed changes

tests/test_updater_with_simulator.py Outdated Show resolved Hide resolved

MVrachev reviewed Dec 6, 2021

View reviewed changes

tests/test_updater_with_simulator.py Outdated Show resolved Hide resolved

MVrachev reviewed Dec 6, 2021

View reviewed changes

tests/test_updater_with_simulator.py Outdated Show resolved Hide resolved

MVrachev mentioned this pull request Dec 9, 2021

ngtests: test loading of cached metadata #1681

Closed

ivanayov force-pushed the test_expired_metadata branch 4 times, most recently from 8bd6065 to c86ad65 Compare December 11, 2021 21:36

ivanayov force-pushed the test_expired_metadata branch 2 times, most recently from 7b15487 to e12a4a6 Compare December 13, 2021 17:37

sechkova reviewed Dec 14, 2021

View reviewed changes

tests/test_updater_with_simulator.py Outdated Show resolved Hide resolved

ivanayov force-pushed the test_expired_metadata branch 2 times, most recently from ef136d4 to ae0ee2e Compare January 7, 2022 15:43

jku marked this pull request as draft January 11, 2022 07:57

ivanayov force-pushed the test_expired_metadata branch from ae0ee2e to 7e56b0c Compare February 3, 2022 22:17

ivanayov marked this pull request as ready for review February 3, 2022 22:17

jku requested changes Feb 18, 2022

View reviewed changes

ivanayov force-pushed the test_expired_metadata branch 2 times, most recently from ba6ef64 to ed0d0b8 Compare February 18, 2022 16:13

jku requested changes Feb 21, 2022

View reviewed changes

tests/test_updater_top_level_update.py Outdated Show resolved Hide resolved

tests/test_updater_top_level_update.py Show resolved Hide resolved

jku mentioned this pull request Feb 23, 2022

blog: "how we test ngclient compliance" #1885

Closed

ivanayov added 3 commits March 18, 2022 19:53

Test expired metadata from cache

15c8d80

This tests that an expired timestamp/snapshot/targets when loaded from cache is not stored as final but is used to verify the new timestamp Fixes theupdateframework#1681 Signed-off-by: Ivana Atanasova <[email protected]>

Verify validation is performed from local metadata

cab99f5

This change verifies that when local metadata has expired, it is still used to verify new metadata that's pulled from remote Signed-off-by: Ivana Atanasova <[email protected]>

Fix expired metadata tests

d8d0486

This change fixes the expired metadata tests to mock `datetime` as previously they mocked `time` incorrectly, which did not affect update methods, as they use `datetime.datetime.utcnow()` to calculate now Signed-off-by: Ivana Atanasova <[email protected]>

ivanayov force-pushed the test_expired_metadata branch from ed0d0b8 to 0f6c961 Compare March 18, 2022 19:58

Update expired metadata tests logic

8d4d9af

This change improves the logic of expired metadata tests, so that it is explicitly visible what the expiry time and the versions are and when update/refresh is called in that period Signed-off-by: Ivana Atanasova <[email protected]>

ivanayov force-pushed the test_expired_metadata branch from 0f6c961 to 8d4d9af Compare March 18, 2022 20:01

jku approved these changes Mar 21, 2022

View reviewed changes

jku merged commit d9f2d9d into theupdateframework:develop Mar 21, 2022

Test expired metadata from cache #1707

Test expired metadata from cache #1707

Uh oh!

Conversation

ivanayov commented Dec 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ivanayov commented Dec 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Dec 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 1865335270

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MVrachev commented Dec 7, 2021

Uh oh!

jku commented Dec 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jku commented Dec 13, 2021

Uh oh!

Uh oh!

sechkova commented Dec 14, 2021

Uh oh!

ivanayov commented Dec 14, 2021

Uh oh!

jku commented Dec 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sechkova commented Dec 14, 2021

Uh oh!

jku commented Jan 11, 2022

Uh oh!

ivanayov commented Feb 3, 2022

Uh oh!

jku left a comment

Choose a reason for hiding this comment

Uh oh!

ivanayov commented Feb 18, 2022

Uh oh!

jku left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ivanayov commented Mar 18, 2022

Uh oh!

jku left a comment

Choose a reason for hiding this comment

Uh oh!

ivanayov commented Mar 21, 2022

Uh oh!

Uh oh!

ivanayov commented Dec 3, 2021 •

edited

Loading

ivanayov commented Dec 3, 2021 •

edited

Loading

coveralls commented Dec 3, 2021 •

edited

Loading

jku commented Dec 8, 2021 •

edited

Loading

jku commented Dec 14, 2021 •

edited

Loading

jku left a comment •

edited

Loading