Releases: cloudlinux/go-diskqueue
Releases · cloudlinux/go-diskqueue
go-diskqueue release
Max bytes per diskqueue (#3) * Keep track of the number of messages in the writeFile and the number of messages we read in the readFile. Also, reserve the last 4 bytes at the end of the file for the number of messages in that file. * Update tests to reflect changes made to the metadata file. Metadata file now contains readMessages and writeMessages. * Test thtat write messages resets when a new file is created. * Add a test that checks if meta data file is correct after writing a complete file and completely reading a file. * Allocate 8 bytes instead of 4 since writeMessages is in64, and reset writeMessages and readMessages in skipToNextRWFile. * Update comment about number of bytes reserved for writeMessages. * Allow user to choose whether to use new feature or not. * Separate testing from the original with the implementation of depth for disk space limit. * Revert to original code style. * Add comment for functions. * Explain that maxBytesDiskSpace is 0 when user is not using disk space limit feature. * Add explicit disk size limit feature flag. * Add a new line after if block. * Track the disk space the files tracked by DiskQueue takes up, and test that these numbers are accurate. * Update totalBytes and writePos when writing number of messages, and resolve the writeMessages off by 1 error. * Remove the additional 8 bytes to totalBytes. * Revert "Remove the additional 8 bytes to totalBytes." This reverts commit 652b548b97ac8c5eb808638fe6bdd80898b3b2b2. * Revert "Update totalBytes and writePos when writing number of messages, and resolve the writeMessages off by 1 error." This reverts commit 48c7a85bb1326c9b362a81545e0908af0807ff7a. * Revert changes that removed the additional 8 bytes. * Add comment to make code more readable. * Add test that completes the write file by meeting the file size limit exactly with a message less than 8 bytes. * Add extra testing to validate the increment/decrement of bytes in core code. * Update variable names. * Revert "Update variable names." This reverts commit e9b23ac5eab849aed8fc2972d06870fcbf243f09. * Revert "Merge branch 'TrackDiskSize' into DepthImpl" This reverts commit 76a0ddd1719105e1d13e57988557783b9d6af9bd, reversing changes made to 30c6ef662449c70d289a402bb95cf9759c642e1e. * Update variable names. * Replace 8 with a constant to improve readability. * Reset Diskqueue data during readError. * Track disk size (#5) * Remove readBytes and update testing. * Add comment for readMsgSize. * Modify disk size limit features only if disk size limit feature is being used . * Reset write bytes during read error if read file is write file. * Track writeBytes and readMsgSize. * Update test. * Test writeBytes is accurate. * Make code more readable. * Revert back to using msgSize instead of readMsgSize in order to make minimal changes. * Update max bad file size to include numFileMsgsBytes. * Update go.mod * Disk size limit (#6) * Remove readBytes and update testing. * Add comment for readMsgSize. * Modify disk size limit features only if disk size limit feature is being used . * Increase disk size limit in testing, and get MetaData file size. * Add comments to metaDataFileSize func for better readability. * Add logic to remove readFile if we are going to surpass the Disk Size Limit. * Abstract code from moveForward and create makeSpace function to be used when DiskQueue needs to make space. * Rename func to make it more readable. * Make space when the new writeMessage will surpass the disk size limit. Test that DiskQueue never surpasses the disk size limit. * Make space until there is enough space to write the message * Abstract code and make a reachFileSizeLimit flag. * Update testing to account for metadata file size. * Handle the deletion of .bad files if it exists. * Get oldest bad file info does not need to return an error. It either finds a bad file or does not. * Track size of bad files. * Decrease badBytes when we delete a bad file and move badBytes to check if we need to make disk space. * Remove badBytes from metadata and make it an atomic field. * Recalculate badBytes if it is a negative number. * Add comments and rename varaibles to improve readability. * Update metaData file size now that badBytes is removed from metaData file. * Prevent user from writing data that is bigger than disk size limit. * Make the code more readable by changing the order of checks in writeOne(). * Test the scenario when msgSize needs the deletion of several files. * Use regex when getting diskqueue files with .bad extension. * Update the tracking of .bad files to use regex. * Remove log messages. * Start writing a test to check if .bad files are deleted or accounted for correctly. * Test that DiskQueue deleted .bad files first in order to make Disk Space. * Update with to other branches - variable name change and adjust writeBytes, writeMsgs, and readMsgs in handReadError. * Adjust writeBytes as a result of the initial overestimation of the size of a bad file when we cannot get its accurate size. * Reset messages in handleReadError * Use MatchString as opposed to Match. * Throw error. * Instead of overestimating bad file size, underestime it. This ensures we will never go over disk limit. * Add a check to see if readFile is corrupted. * Remove DEBUG logs. * test .bad files. * In progress... corrupt file and then have readOne() deem it corrupted. * Make code more readable. * Ensure that diskqueue handles corrupted files correctly. * Remove badBytes and get the accurate writeBytes data every time a writeFile is deleted or a readFile turns into a bad file. * Abstract general function to walk through all of the files in the directory DiskQueue writes and reads in. * Abstract code to its own function. * Remove readMsgSize and make code more readable. * Make test code more readable. * Update go.mod * Make regexp constants and close metadatafile after getting its size. * Change writeBytes to totalDiskSpaceUsed and have it track writeBytes and badBytes for now. * Update panic messages. * Track the total disk size with one variable rather than several smaller ones. * Remove unnecessary comments. * Break up huge function into two smaller functions. * Ensure that global regExp are created on start. * Add testing of depth, add info logs, and test when disk size limit is too small (not enough space for meta data file). * Update name of dq objects to match function name. * Use ReadDir instead of WalkDir * Replace go v1.16 functions and objects with <v1.16 functions and objects. * Have metaData file size stay as 56 bytes and update testing. * Add the number of bytes that were removed when a file is removed. * Only get the size if the file was able to be removed successfully. * Add function to get filder size and update log when removing read file to make space. * Make the code more readable. * Extract chunk of code into a new function. * Improve readability of the code and update code from nsqio/go-diskqueue#29 issue 29 * totalDiskSpaceUsed does not change in handleReadError * Iterate through each file rather than relying on the totalDiskSpaceUsed variable. * Revert "Merge branch 'master' into DiskSizeLimit" This reverts commit 0c456b49e0f550e540caa46599c59e167e4d4158, reversing changes made to a821855461335029764927e2d929c178967a51b6. * Update README.md * Update README.md Updated the Build status, Go reference, and latest Github release * Read correct number of messages from end of file (#7) * trivial improvement of some checks * use file instead of bufio.Reader in readNumOfMessages from file end Signed-off-by: Leon Ziyang Zhang <[email protected]> * Update Readme and add description for the fork. * Update \d\d\d\d\d\d to \d+ * Fixed issue with shared regexps * Update go.mod * Fixed conflicting variable issue * Fixed issue with peekChan * Reverted switch to next file before maxBytesPerFile is reached * Fixed issue with .bad files * Added timeout between corruption checks * Added more timeout between corruption checks * Reduced timeouts * Timeout replaced * Fixed pattern of bad files * Ref * Check failures * Added error handling * Fixed tests Signed-off-by: Leon Ziyang Zhang <[email protected]> Co-authored-by: Kevin Cam <[email protected]> Co-authored-by: Kevin Cam <[email protected]> Co-authored-by: CatherineF-dev <[email protected]> Co-authored-by: Leon Ziyang Zhang <[email protected]> Co-authored-by: Leon Ziyang Zhang <[email protected]> Co-authored-by: Mikhail Faraponov <[email protected]>