Skip to content

Disk size limit #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 69 commits into from
Jul 8, 2021
Merged

Disk size limit #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
0d45478
Merge pull request #2 from kev1n80/DepthImpl
kev1n80 Jun 2, 2021
4f3dbc2
Remove readBytes and update testing.
kev1n80 Jun 2, 2021
0ab1275
Add comment for readMsgSize.
kev1n80 Jun 2, 2021
844e43b
Modify disk size limit features only if disk size limit feature is be…
kev1n80 Jun 3, 2021
5068de1
Increase disk size limit in testing, and get MetaData file size.
kev1n80 Jun 3, 2021
42686e9
Add comments to metaDataFileSize func for better readability.
kev1n80 Jun 3, 2021
5093a20
Add logic to remove readFile if we are going to surpass the Disk Size…
kev1n80 Jun 3, 2021
425d8d3
Abstract code from moveForward and create makeSpace function to be us…
kev1n80 Jun 4, 2021
4380639
Rename func to make it more readable.
kev1n80 Jun 4, 2021
aaba003
Make space when the new writeMessage will surpass the disk size limit…
kev1n80 Jun 4, 2021
9f86e28
Make space until there is enough space to write the message
kev1n80 Jun 4, 2021
24a5f68
Abstract code and make a reachFileSizeLimit flag.
kev1n80 Jun 4, 2021
60415d7
Update testing to account for metadata file size.
kev1n80 Jun 4, 2021
c39849f
Handle the deletion of .bad files if it exists.
kev1n80 Jun 7, 2021
c00921e
Get oldest bad file info does not need to return an error. It either …
kev1n80 Jun 7, 2021
0709e65
Track size of bad files.
kev1n80 Jun 7, 2021
65854ca
Decrease badBytes when we delete a bad file and move badBytes to chec…
kev1n80 Jun 8, 2021
2c879c8
Remove badBytes from metadata and make it an atomic field.
kev1n80 Jun 8, 2021
aedd503
Recalculate badBytes if it is a negative number.
kev1n80 Jun 8, 2021
621b1bc
Add comments and rename varaibles to improve readability.
kev1n80 Jun 8, 2021
9cde8b0
Update metaData file size now that badBytes is removed from metaData …
kev1n80 Jun 8, 2021
b215b85
Prevent user from writing data that is bigger than disk size limit.
kev1n80 Jun 8, 2021
e2f76a8
Make the code more readable by changing the order of checks in writeO…
kev1n80 Jun 8, 2021
f76ab78
Test the scenario when msgSize needs the deletion of several files.
kev1n80 Jun 8, 2021
02ebb4b
Use regex when getting diskqueue files with .bad extension.
kev1n80 Jun 9, 2021
e7f8522
Update the tracking of .bad files to use regex.
kev1n80 Jun 9, 2021
9792c99
Remove log messages.
kev1n80 Jun 9, 2021
2438eaf
Start writing a test to check if .bad files are deleted or accounted …
kev1n80 Jun 9, 2021
1fab695
Test that DiskQueue deleted .bad files first in order to make Disk Sp…
kev1n80 Jun 9, 2021
3ec5f40
Update with to other branches - variable name change and adjust write…
kev1n80 Jun 9, 2021
bfed51c
Adjust writeBytes as a result of the initial overestimation of the si…
kev1n80 Jun 10, 2021
d0a48bb
Reset messages in handleReadError
kev1n80 Jun 10, 2021
872148f
Use MatchString as opposed to Match.
kev1n80 Jun 10, 2021
a13c888
Throw error.
kev1n80 Jun 10, 2021
c84524f
Instead of overestimating bad file size, underestime it. This ensures…
kev1n80 Jun 11, 2021
e9cd28a
Add a check to see if readFile is corrupted.
kev1n80 Jun 11, 2021
3f2ff05
Remove DEBUG logs.
kev1n80 Jun 11, 2021
f926027
test .bad files.
kev1n80 Jun 11, 2021
eef61fb
In progress... corrupt file and then have readOne() deem it corrupted.
kev1n80 Jun 11, 2021
6bcee72
Make code more readable.
kev1n80 Jun 11, 2021
6e63d55
Ensure that diskqueue handles corrupted files correctly.
kev1n80 Jun 14, 2021
86ade8f
Remove badBytes and get the accurate writeBytes data every time a wri…
kev1n80 Jun 14, 2021
0ab256e
Abstract general function to walk through all of the files in the dir…
kev1n80 Jun 14, 2021
23f9369
Abstract code to its own function.
kev1n80 Jun 14, 2021
2c66983
Remove readMsgSize and make code more readable.
kev1n80 Jun 15, 2021
779f347
Make test code more readable.
kev1n80 Jun 15, 2021
78610d8
Update go.mod
kev1n80 Jun 16, 2021
878cc61
Make regexp constants and close metadatafile after getting its size.
kev1n80 Jun 16, 2021
3b8bd41
Change writeBytes to totalDiskSpaceUsed and have it track writeBytes …
kev1n80 Jun 16, 2021
d453ed4
Update panic messages.
kev1n80 Jun 16, 2021
87638f0
Track the total disk size with one variable rather than several small…
kev1n80 Jun 17, 2021
4f017a3
Remove unnecessary comments.
kev1n80 Jun 17, 2021
3d62b73
Break up huge function into two smaller functions.
kev1n80 Jun 17, 2021
069c7f8
Ensure that global regExp are created on start.
kev1n80 Jun 17, 2021
2e68813
Add testing of depth, add info logs, and test when disk size limit is…
kev1n80 Jun 17, 2021
be18a89
Update name of dq objects to match function name.
kev1n80 Jun 18, 2021
26577b6
Use ReadDir instead of WalkDir
kev1n80 Jun 23, 2021
3ffa67c
Replace go v1.16 functions and objects with <v1.16 functions and obje…
kev1n80 Jun 24, 2021
ed2255d
Have metaData file size stay as 56 bytes and update testing.
kev1n80 Jun 25, 2021
89ba90b
Add the number of bytes that were removed when a file is removed.
kev1n80 Jun 28, 2021
3077b4f
Only get the size if the file was able to be removed successfully.
kev1n80 Jun 28, 2021
0834f63
Add function to get filder size and update log when removing read fil…
kev1n80 Jun 29, 2021
3680769
Make the code more readable.
kev1n80 Jul 7, 2021
e9fe0b0
Extract chunk of code into a new function.
kev1n80 Jul 7, 2021
44b932a
Improve readability of the code and update code from nsqio/go-diskque…
kev1n80 Jul 7, 2021
a4f9d67
totalDiskSpaceUsed does not change in handleReadError
kev1n80 Jul 7, 2021
a821855
Iterate through each file rather than relying on the totalDiskSpaceUs…
kev1n80 Jul 7, 2021
0c456b4
Merge branch 'master' into DiskSizeLimit
kev1n80 Jul 8, 2021
6a640f3
Revert "Merge branch 'master' into DiskSizeLimit"
kev1n80 Jul 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading