Skip to content

Handle large write batches #215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Jan 29, 2019
Merged

Handle large write batches #215

merged 9 commits into from
Jan 29, 2019

Conversation

wilhuff
Copy link
Contributor

@wilhuff wilhuff commented Jan 26, 2019

This addresses #208.

While this works, I'm not yet sure this approach performs well enough to merge as-is. For example, we could make the two methods I changed to use direct load to use the inline method that others use.

I've sent this out to get feedback on this before I invest time in benchmarking.

@googlebot googlebot added the cla: yes Override cla label Jan 26, 2019
@wilhuff
Handle reading large mutation batches
9693726
@wilhuff
Rework lookupMutationBatch and getNextMutationBatchAfterBatchId to lo…
dd75bcd 
…ad directly
@wilhuff
format
dd86110
@wilhuff
Rework large blob handling to load in chunks
9bd9914 
Unfortunately, SQLiteStatement.executeForBlobFileDescriptor is
unreliable, sometimes returning null even when the row exists and the
blob has a value (!!).
@wilhuff
cleanup
fdb1ceb
@wilhuff wilhuff force-pushed the wilhuff/large-write-batches branch from 37b5f8a to fdb1ceb Compare January 27, 2019 22:31
@wilhuff
Copy link
Contributor Author

wilhuff commented Jan 27, 2019

While testing the approach with SQLiteStatement.executeForBlobFileDescriptor() it turned out that this approach wasn't reliable. For reasons I didn't get to the bottom of, this would occasionally return null even for rows that existed and had a value.

@wilhuff wilhuff mentioned this pull request Jan 27, 2019
Closed
@wilhuff
Copy link
Contributor Author

wilhuff commented Jan 27, 2019

/retest

mikelehen
mikelehen approved these changes Jan 28, 2019
View reviewed changes
Copy link
Contributor

@mikelehen mikelehen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay. This looks good except for some minor comment nits. Thanks for tracking all of this down!

Don't forget to port to the other platforms. 😛

@mikelehen mikelehen assigned wilhuff and unassigned mikelehen Jan 28, 2019
@var-const var-const removed their assignment Jan 28, 2019
This was referenced Jan 28, 2019
Merged
Merged
@google-oss-bot
Copy link
Contributor

@wilhuff: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
smoke-tests-debug 9ef85f0 link /test smoke-tests-debug

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@wilhuff wilhuff merged commit 39408f5 into master Jan 29, 2019
@wilhuff wilhuff deleted the wilhuff/large-write-batches branch January 29, 2019 01:20
@firebase firebase locked and limited conversation to collaborators Oct 12, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@var-const var-const var-const approved these changes

@mikelehen mikelehen mikelehen approved these changes

Assignees

@wilhuff wilhuff

Labels
cla: yes Override cla size/L
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@wilhuff @google-oss-bot @mikelehen @var-const @googlebot