PR: Rewrite ls to use prefixes

[martindurant]

Designed to get around the case where some parts of a bucket are not accessible

Fixes #38

@jseabold , please see if this fixes things for you.

[mrocklin]

@jseabold can you recommend a few tests to ensure that the behavior you're looking for checks out?

[jseabold]

Will have a look. Working on getting the ACLs for you that would allow you to make a role with these settings. It sounds like our typical settings were best practices from AWS but still trying to chase that down.

[jseabold]

http://blogs.aws.amazon.com/security/post/Tx1P2T3LFXXCNB5/Writing-IAM-policies-Grant-access-to-user-specific-folders-in-an-Amazon-S3-bucke

[jseabold]

Maybe raise an informative error here if the path doesn't end in a forward slash? Also, key is really a prefix here, if things are called correctly. Maybe use that nomenclature?

[martindurant]

I would rather not raise here, I think it's reasonable to do ls('bucket/path') and expect to get keys below bucket/path/.
OK to calling it prefix.

[martindurant]

@jseabold , I'm sorry, I still can't reproduce your situation, where you can't even list some bucket/prefix. The link is about user-level access policies, but I don't have the privileges to create those. Is there any chance we can set up a little bucket which encapsulates this problem for testing? Hare you checked whether my changes above solve the problem for you?

[jseabold]

This didn't work on the first thing I tried, but there's a lot going on here, so I'm trying to pin down why. Fundamentally, there's something I don't yet understand with how you're instantiating the client. This works

import boto3
session = boto3.Session(profile_name='account')
client = session.client('s3')
client.get_object(Bucket=BUCKET, Key=KEY)

This doesn't

fs = s3fs.S3FileSystem(profile_name='account')
fs.s3.get_object(Bucket=BUCKET, Key=KEY)

[jseabold]

It's because you're calling ls('') in the instantiation to make sure that everything worked I guess. I can't do this, but I need still need to sign my requests.

[jseabold]

Does this list and cache the entire bucket on instantiation? Again still trying to understand what's going on here, but, if so, that can be a super expensive operation. Assuming that's right, I would start from the premise that everyone has a key-value store with hundreds of thousands of keys (if not more) and err on the side of avoiding trying to list those at all cost.

[jseabold]

Other than that, yes, this seems to fix my issues in some quick checks. Thanks!

Will let you know if I run in to anything else.

[martindurant]

Oh! Thank you for finding the root (ls("")) - so this is a IAM policy thing, not a bucket permission thing...

That can certainly be avoided - it exists for the anon=None default case, to see if the credentials are valid. I can't immediately see another way to check this (but the check itself could be optional?).

[jseabold]

so this is a IAM policy thing, not a bucket permission thing...

I don't follow this.

Seems like a heavy way to check if credentials are valid. Given that it will happen later, why check at all? If you really want to handhold, you could fail gracefully later and tell the users to check their credentials or use anon=True and don't try to automatically determine this for users, given that there isn't a straightforward way to do it. You could try to use the sts service to generate a session token since that's fairly light, but I don't know the full permissions and failure modes around this for users and it's probably not ok to assume that everyone can do this.

If you attach the session, users can call S3FileSystem.session.get_credentials() and look themselves. I don't see how to get these from the service clients.

[martindurant]

Excellent idea!
The check is now done via sts, and I've attached the session too, as you suggest.
Is your problem finally fixed?

Also, you may have seen in #48 that there may be a simpler way coming to access single files one at a time (so you don't get filesystem methods, but you do get file-like behaviour, buffering etc).

[jseabold]

lgtm