-
Notifications
You must be signed in to change notification settings - Fork 278
Fix handling consistent targets same as legacy updater #1591
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix handling consistent targets same as legacy updater #1591
Conversation
Pull Request Test Coverage Report for Build 1290255166Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
While working on this one multiple questions appeared:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the errors you raise both look like things we could handle earlier: the target path originally comes from the client so if we're validating, let's validate where it really is input -- get_one_valid_targetinfo.
The commit also uses os.path
to form a URL. It may be possible that Python has enough workarounds to make that work in practice in Windows but I don't like it. os.path
is for local file paths
I'll try to have a PR for that ASAP now that the blocker is merged. But feel free to include the directory creating commit until then
Yeah, that's one of the core advantages of proper table testing, we probably do.
I think we maybe don't need to validate that it exists or can be written to -- I think we can just mention OSError in the docstring. But should test that something reasonable really happens on those cases I already mentioned the target path checks (if we do them) should IMO happen much earlier.
👍 new issue might be appropriate |
Created an issue for that: #1594. |
2c78cdd
to
7620478
Compare
This pr is ready for review after the merge of #1587 and #1592.
I experimented with a new commit where I used the same decorator as
I won't focus on validation in this pr.
I changed that and instead used a manual file path and URL creation. @jku can you answer my last question too:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The implementation looks good but I have some reservations for the tests that I put in the comments. Also there isn't really a test for the hash prefix implementation anywhere, the tests rely on the assumption that the default workflow uses consistent targets and no errors were raised.
7620478
to
c645353
Compare
I will think about how can I enhance the test and leave a comment when I am ready. |
Sorry I don't understand the question? I don't see any write errors documented currently. Can file writes fail? Yes. Does our code raise exceptions in those cases? probably: someone needs to read the code and find out |
I think this might be fine: we definitely want the default setup to always be consistent_snapshot (test could assert that this is the case, or just add a comment that it's expected). But I have been thinking that RepositorySimulator could easily have some tracking on what gets requested/downloaded... so a test could then assert that e.g. a specific url was used. This sounds like a potential improvement but maybe a sidetrack for this issue? |
c645353
to
919eae4
Compare
Updated the pr with the following changes:
|
c0b93f5
to
d42a199
Compare
d42a199
to
b51c099
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems ok now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks fine to me.
Some observations
- The targets testing is still lacking in volume: we should have vastly more variations on this... but let's do that on it's own issue, as result of Design test strategy #1579
- I think all of the different DataSet types are probably unneeded now -- it was useful in the very crowded decorator implementation but now the different DataSets might confuse more than help 🤷
left two more comments but they are nits as well
Yea, maybe you are right, given that as you have pointed out in one of your comments I should create a |
The definition of consistent targets in the spec is ambiguous: "consistent target files should be written to non-volatile storage as digest.filename.ext" Additionally, the specification describes consistent targets when the client builds the download URL as follows: "The filename is of the form HASH.FILENAME.EXT". The issue is about how we interpreted those quotes. The legacy updater has decided this means a target path "a/b" will translate to a download url path "a/{HASH}.b". The ngclient however translates the target path "a/b" to a download url path "{HASH}.a/b". We decided we want to follow the same approach taken from the legacy updater and thus change how we construct the consistent targets. Additionally, we want to make sure we test for cases when the TARGETPATH is an empty string or points to a directory. Signed-off-by: Martin Vrachev <[email protected]>
Generalize the decorator used in test_metadata_serialization.py and move it inside tests/utils.py, so it can be reused in other similar situations. Signed-off-by: Martin Vrachev <[email protected]>
Reuse the decorator defined in tests/utils.py in order to receive more helpful messages when an assertion fails in test_tragets(). Signed-off-by: Martin Vrachev <[email protected]>
Make sure that hash prefixes are added when downloading a target through the repository simulator. Signed-off-by: Martin Vrachev <[email protected]>
b51c099
to
c3e746a
Compare
Note: This pr is dependent on pr #1592.
Fixes #1576
Description of the changes being introduced by the pull request:
The definition of consistent targets in the spec is ambiguous:
"consistent target files should be written to non-volatile storage
as digest.filename.ext"
Additionally, the specification describes consistent targets when the
client builds the download URL as follows:
"The filename is of the form HASH.FILENAME.EXT".
The issue is about how we interpreted those quotes.
The legacy updater has decided this means a target path
a/b
will translate to a download url patha/{HASH}.b
.The ngclient however translates the target path
a/b
to a download URL path{HASH}.a/b
.We decided we want to follow the same approach taken from the legacy
updater and thus change how we construct the consistent targets.
Additionally, we want to make sure we test for cases when the TARGETPATH
is an empty string or points to a directory.
Please verify and check that the pull request fulfills the following
requirements: