Skip to content

aws s3 sync command is working according to filesize change instead of md5sum change for identical filename file #8377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
YoongLoong opened this issue Nov 30, 2023 · 3 comments
Assignees
Labels
duplicate This issue is a duplicate.

Comments

@YoongLoong
Copy link

Describe the bug

Hi, i have a situation in my usage on aws s3 sync:
File A: abcdefg.txt (1234 Bytes) (md5sum: a954ea6b12db7d788350b4c008ee26fb)
File B: abcdefg.txt (1234 Bytes) (md5sum: bd84bb07f07a58e832bb3c72e3db4663)

After I aws s3 sync the File A to the s3 location and try to sync the File B to the same s3 location it will skipped. Sync is only taking care of "filesize change" only instead of md5sum change which defeat the purpose of "SYNC" and it will give the wrong impression that there is no change between File A and File B. Would this be fixed or taken out as it is confusing to the user that they have the identical files being "sync"?

Expected Behavior

It should "sync" File B and replace File A upon second aws s3 sync as it is the "different" file with the same filename (different md5sum).

Current Behavior

aws s3 sync command skipped the upload due to the "same" file being uploaded (same filesize different md5sum)

Reproduction Steps

echo 1 > A/test
echo 2 > B/test
aws s3 sync A/ s3://<S3_Bucket>/ --exclude "" --include "test" --no-progress
aws s3 sync B/ s3://<S3_Bucket>/ --exclude "
" --include "test" --no-progress

Possible Solution

ETag of "B/test" file should be calculated and compared with the s3://<S3_Bucket>/test to ensure it is the same file before skipped sync

Additional Information/Context

No response

CLI version used

2.13.6

Environment details (OS name and version, etc.)

ubuntu14.04

@YoongLoong YoongLoong added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Nov 30, 2023
@YoongLoong YoongLoong changed the title aws s3 sync command is working according to filesize change instead of md5sum change aws s3 sync command is working according to filesize change instead of md5sum change for identical filename file Nov 30, 2023
@RahulDubey391
Copy link

Hi @YoongLoong, I would like to have a look into the issue.

@tim-finnigan tim-finnigan self-assigned this Dec 7, 2023
@tim-finnigan tim-finnigan added duplicate This issue is a duplicate. and removed bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 7, 2023
@tim-finnigan
Copy link
Contributor

There is already a feature request to support checksums in high-level S3 commands: #6750. Please 👍 that issue to show support, and add any comments there regarding your use case.

Copy link

github-actions bot commented Dec 7, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue is a duplicate.
Projects
None yet
Development

No branches or pull requests

3 participants