-
Notifications
You must be signed in to change notification settings - Fork 51
[support-bundle] include for large log files #8771
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
[support-bundle] include for large log files #8771
Conversation
Created using jj-spr 1.3.6-beta.1
Created using jj-spr 1.3.6-beta.1
In this case @wfchandler noted the log file is mostly duplicate log lines and would likely compress nicely. My concern is a log file where this is not the case and we have to burn CPU cycles compressing it and then transferring it over the network to later be unpacked. I wanted to post this is a draft for feedback because regardless of how well it compresses the final result gets decompressed on the other side and we still use the default compression algorithm for the final zip file I believe. |
@@ -1153,7 +1153,7 @@ fn recursively_add_directory_to_zipfile( | |||
|
|||
let file_type = entry.file_type()?; | |||
if file_type.is_file() { | |||
let opts = FullFileOptions::default(); | |||
let opts = FullFileOptions::default().large_file(true); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not a blocker for this PR , but it has me wondering:
We basically want to pull out the contents of the "zipfile from the sled-agent" and just rename things within the user-visible zipfile that we're creating. I wonder if this would be possible with a slightly more advanced version of zip, that lets us transfer the "contents" of zipfiles around without unpacking them.
(Basically, move the contents of a zipfile entry to a new name, if the compression scheme is an exact match)
This seems like a dangerous operation, but could be a possible optimization if this is taking a while. It's potentially eating a chunk of disk space, but as long as we're using streaming operations to write to the zipfile, I think it's not (currently) consuming 30+ GiB of memory.
TL;DR: this current implementation is fine with me, as long as it doesn't have too-bad performance implications.
@@ -1153,7 +1153,7 @@ fn recursively_add_directory_to_zipfile( | |||
|
|||
let file_type = entry.file_type()?; | |||
if file_type.is_file() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also: I wonder if we should pick an explicit compression scheme here, so we aren't at the whim of zip changing defaults.
E.g., we expect it'll be supported by customers, maybe we just pick zstd-3
here too
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's probably not a good idea to use zstd
for the customer-accessible file because the standard zip tools on all major platforms don't support it. How would we access the file on Illumos, for example?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
according to https://docs.rs/zip/latest/zip/write/struct.FileOptions.html I think the default is "Deflate", and the level is either 24 or 6? https://docs.rs/zip/latest/zip/write/struct.FileOptions.html#method.compression_level
Good point re: zstd - just want to make sure an update of the zip crate doesn't change this for us
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah yeah, agreed it makes sense to define which algorithm we want to use.
@wfchandler noticed an error in a collected support bundle where we failed to copy a large (31GB) log file.
We need to turn on large file support if we wish to include these files.