Ignore documents bigger than 400KB

[alexandruluca]

Is it possible to automatically ignore documents which are greater than 400KB in size?

This would not break synchronization process.

I can create a PR if you will accept it

[eli-darkly]

I don't know what you mean by "this would not break synchronization process"— I mean, discarding data is not something the SDK would normally ever do. Trying to use the DynamoDB integration when you have flags over that size will never work, so having this package just drop that data is just creating a somewhat different failure mode. I think we would need to talk a little more about what the desired behavior is.

[eli-darkly]

It's also not quite as straightforward to calculate that size limit as you might think. It's not just 400K for the serialized JSON data of the flag itself— the limit also includes any other attributes in the data item (in this case, the flag key and version number) as well as the names of those attributes. There's also an index entry for the item, and the size of that also counts toward the limit, but I haven't seen any documentation as to what determines the size of the index entry. So, we could try to estimate this, but I think there will always be a slight chance that DynamoDB will figure it differently than we do and so it may not be entirely possible to avoid an error.

[alexandruluca]

What I mean with "this would not break synchronization process", was that we have a lambda which periodically synchronizes feature flags to dynamodb (https://docs.launchdarkly.com/sdk/features/storing-data/dynamodb). I know that discarding data over 400KB is not ideal but at least other feature flag data would get synchronized to our store.

Right now if one document which is greater than 400KB, then the entire batch gets discarded. I hope it's a bit more clear now

[alexandruluca]

Understood, I thought the check would be a bit more easy to implement. For our use case, it would have been enough to check the serialized JSON's length

[eli-darkly]

it would have been enough to check the serialized JSON's length

Well, I'm not against doing something like this, we have already been thinking about it, but what I'm getting at is that it would not really be enough to just check the serialized JSON's length. I mean, maybe in your particular application all of your flags are either "definitely nowhere near the size limit" or "definitely far over the size limit" and not anywhere in the middle, but for anything that is close to the limit, it would be possible for us to think it's OK and then have it be rejected by DDB anyway.

[alexandruluca]

Yes. For our particular case, only one flag had excessive amount of data, roughly 130kb more. The problem was that this was causing the entire batch to be discarded.

Shall I keep the ticket open then?

[eli-darkly]

No, we're already working on addressing this, we are just internally deciding how. The ticket isn't necessary.

[eli-darkly]

We've released version 4.0.1 which improves the behavior in this case. See release notes.