An S3 backend. #23
An S3 backend. #23
Conversation
| (defrecord S3Backend [#_service bucket] | ||
| core/IBackend | ||
| (new-session [_] | ||
| (atom {:writes 0 :deletes 0})) |
There was a problem hiding this comment.
Does the s3 API tell you about how much money you're spending on I/O? This could be really interesting if we could track the # of bytes uploaded, for example :)
There was a problem hiding this comment.
The API doesn't track that, it's something you need to track manually.
However, all data transfer in to S3 is free, and there's a $0.005 fee per 1000 PUTs. It's also neat if you're fetching data from an EC2 instance in the same region, since the data transfer is also free, you just pay for requests.
There was a problem hiding this comment.
Interesting--does Amazonica batch writes? If not, the tracing GC algo would allow you to avoid needing any auxiliary objects, to reduce the number of puts.
There was a problem hiding this comment.
It shouldn't batch writes -- it's a thin shim on the Java SDK, and that doesn't do any batching. And yeah, doing something more sensible for GC would eliminate additional requests.
There was a problem hiding this comment.
I think confetti does some diffing for you though, code here: https://github.com/confetti-clj/s3-deploy/blob/master/src/confetti/s3_deploy.clj. This is a great idea, keep it up!
|
This is awesome, thank you! I see that you implemented bidirectional reference tracking--I am wondering whether a tracing GC might be more appropriate. To do this, we'd only need to store the unidirectional reference from index to child, as well as storing the set of anchored roots. Then, to do a GC, we'd start from each root, recursively storing all the IDs of nodes that are reachable from those roots. Finally, we would stream the list of all keys in the bucket, and delete anything we didn't store in the set we built up before. This could be really interesting to look at with B=1000+ and buffer-size=5000+--multi MB nodes could actually make this an interesting way to store indexed data w/o a huge overhead :) |
|
I was also considering doing some of the bookkeeping in (e.g.) DynamoDB instead, and not having empty marker objects, but just wanted to get something that sort-of works first. I'll think some more on a better approach. I have a use case where I'm going to be storing potentially lots of data, which might not ever be queried, and queries wouldn't necessarily need to worry about inconsistent latencies. But, we'd want storage to be as cheap as we can get away with, which is why I'm considering S3. So yeah, multi-MB nodes would fit this fairly well :) |
|
I just threw up a draft of a tracing GC at #24--I haven't tested it hardly at all, but at least it compiles. I think that a bit of polish, that could handle space reclamation without needing to bring in dynamo or use extra API calls. |
The test builds two random b-trees, and then uses one of them as a gc root, and including the other in the lazy sequence of "all keys" to be dealt with by the collector. After running a gc, we assert that the `deleted-fn` has been invoked against all the "dead" nodes.
|
Thanks to @cddr, #24 now has been tested and had some bugs ironed out. (plus #25 massively reduced the I/O cost of tracing GC) Would you like to take a stab at integrating that into the S3 backend? It should allow the S3 backend to avoid any explicit bookkeeping, and do so without too much IO overhead. To do so, you should be able to merge or rebase #24 into this branch, and then just call the tracing GC. |
# Conflicts: # project.clj
This pulls in datacrypt-project#24, since I'd like to test out the GC too.
|
It looks like the test depends on a program called |
|
I did some digging into the Travis docs, and I'm not sure if I can go ahead and give it a try, though :/ |
|
I think a gem install in the before_install of the .travis file should work
|
|
I came up with a better approach https://github.com/csm/konserve-ddb-s3, which builds off the work of https://github.com/replikativ/konserve, https://github.com/replikativ/hitchhiker-tree, and https://github.com/replikativ/datahike. Thanks so much for this code! |
This is all pretty experimental, and I'm not sure I understand everything a backend needs to do. It's totally naive, but this might be the start of something useful. Kind of cool that it only took ~100 lines of code to do 8-)