Skip to content

Support for concatenated objects #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
catwell opened this issue Feb 20, 2012 · 2 comments · Fixed by #30
Closed

Support for concatenated objects #10

catwell opened this issue Feb 20, 2012 · 2 comments · Fixed by #30

Comments

@catwell
Copy link

catwell commented Feb 20, 2012

MessagePack allows binary blobs composed of several valid MessagePack objects concatenated. This is the case of the official MessagePack test data for instance (https://github.com/msgpack/msgpack/tree/master/test).

To support this kind of data you need an interface that:

  • returns the first object of the blob and the byte offset where the decoding stopped,
  • (optionally) allows you to specify an offset where the decoding should start.

An advantage of implementing this is that it would make it possible to use the "official" test data to test this module.

@antirez
Copy link
Owner

antirez commented Feb 20, 2012

I like that, thanks.

that's the idea for the API change:

  • cmsgpack.unpack() works like now with one argument, returning just the decoded object, and starting at index 0.
  • If a second argument is passed, it is used to start the decoding at the specified byte. Also two values are returned instead of one, that is the decoded object, and the offset at which the decoding ended (offset of next object).
  • Also, no error is returned in this "two arguments" mode if not all the string is consumed of course.

This makes the API for the obvious case, obvious. The more complex API will impact only more demanding users.

Thanks for the suggestion,
Salvatore

@catwell
Copy link
Author

catwell commented Feb 20, 2012

This API would work. I do the opposite for return values (offset, object) and I return nil in the offset if the decoding failed (which I use for error checking) but that's a matter of style. In the context of Redis your ordering is simpler.

@ghost ghost mentioned this issue Jun 18, 2012
mattsta added a commit to mattsta/lua-cmsgpack that referenced this issue Apr 7, 2014
Inspired by antirez#10, I added two new functions allowing you to
limit multiple return from unpack.

If you don't trust your msgpack input or want to manage flow control
of returned object generation yourself, you can now limit the objects
returned from unpack (but not with unpack() itself because it already returns
an arbitrary number of objects).

You can now unpack_one(msgpack) to get (next_offset, obj) returned; then
call unpack_one(msgpack, next_offset) to get another offset and one
more object>

You can now also unpack_limit(msgpack, N) to get (next_offset, obj1,
obj2, ..., objN) returned; then call unpack_limit(msgpack, K, next_offset)
to get another offset and K more objects.

This also refactors the copy/paste command addition in module bring-up.

Tests added for new commands and all tests pass:
TEST PASSED:    225
TEST FAILED:    0
TEST SKIPPED:   0

Closes antirez#10
mattsta added a commit to mattsta/lua-cmsgpack that referenced this issue Apr 7, 2014
Changes:
  - Improve table vs. array detection
  - Improve packing +-inf
  - Add multiple pack/unpack support
  - Add cmsgpack.safe module variant
  - Add local build infrastructure for easier testing
  - Add user-controlled unpack support limiting returned objects
  - Add Lua 5.3 compatibility
  - Remove an unnecessary malloc
  - Use Lua memory allocator instead of malloc for buffer creation

Issues involved:
  - closes antirez#16 - allow multi pack/unpack by default
  - closes antirez#10 - unpack one/limit API added
  - closes antirez#13 and closes antirez#20 - use Lua allocator
  - closes antirez#15 - (included in antirez#16)
  - ignores antirez#22 because it's confusing
  - closes antirez#23 - fixed elsewhere
  - closes antirez#26 - extracted some useful parts from a difficult commit
  - closes antirez#28 - we started tagging versions again recently
  - closes antirez#27 - that failure case works for me now

I merged commits with original author information where possible, but
each commit required manual cleanup of one or more of:
formatting fixes (no tabs, please), commit message fixes (more details
please), extracting contents from a single 300 line commit with 5
different logical changes merged together, and general correctness
checking after merging with newer code.

As of this commit, all tests pass on Lua 5.1.5 and Lua 5.3-work2.
mattsta added a commit to mattsta/lua-cmsgpack that referenced this issue Apr 7, 2014
Changes:
  - Improve table vs. array detection
  - Improve packing +-inf
  - Add multiple pack/unpack support
  - Add cmsgpack.safe module variant
  - Add local build infrastructure for easier testing
  - Add user-controlled unpack support limiting returned objects
  - Add Lua 5.3 compatibility
  - Remove an unnecessary malloc
  - Use Lua memory allocator instead of malloc for buffer creation

Issues involved:
  - closes antirez#16 - allow multi pack/unpack by default
  - closes antirez#10 - unpack one/limit API added
  - closes antirez#13 and closes antirez#20 - use Lua allocator
  - closes antirez#15 - (included in antirez#16)
  - ignores antirez#22 because it's confusing
  - closes antirez#23 - fixed elsewhere
  - closes antirez#26 - extracted some useful parts from a difficult commit
  - closes antirez#28 - we started tagging versions again recently
  - closes antirez#27 - that failure case works for me now

I merged commits with original author information where possible, but
each commit required manual cleanup of one or more of:
formatting fixes (no tabs, please), commit message fixes (more details
please), extracting contents from a single 300 line commit with 5
different logical changes merged together, and general correctness
checking after merging with newer code.

As of this commit, all tests pass on Lua 5.1.5 and Lua 5.3-work2.
mattsta added a commit to mattsta/lua-cmsgpack that referenced this issue Apr 7, 2014
Changes:
  - Improve table vs. array detection
  - Improve packing +-inf
  - Add multiple pack/unpack support
  - Add cmsgpack.safe module variant
  - Add local build infrastructure for easier testing
  - Add user-controlled unpack support limiting returned objects
  - Add Lua 5.3 compatibility
  - Remove an unnecessary malloc
  - Use Lua memory allocator instead of malloc for buffer creation

Issues involved:
  - closes antirez#16 - allow multi pack/unpack by default
  - closes antirez#10 - unpack one/limit API added
  - closes antirez#13 and closes antirez#20 - use Lua allocator
  - closes antirez#15 - (included in antirez#16)
  - ignores antirez#22 because it's confusing
  - closes antirez#23 - fixed elsewhere
  - closes antirez#26 - extracted some useful parts from a difficult commit
  - closes antirez#28 - we started tagging versions again recently
  - closes antirez#27 - that failure case works for me now

I merged commits with original author information where possible, but
each commit required manual cleanup of one or more of:
formatting fixes (no tabs, please), commit message fixes (more details
please), extracting contents from a single 300 line commit with 5
different logical changes merged together, and general correctness
checking after merging with newer code.

As of this commit, all tests pass on Lua 5.1.5 and Lua 5.3-work2.
mattsta added a commit to mattsta/lua-cmsgpack that referenced this issue Apr 7, 2014
Changes:
  - Improve table vs. array detection
  - Improve packing +-inf
  - Add multiple pack/unpack support
  - Add cmsgpack.safe module variant
  - Add local build infrastructure for easier testing
  - Add user-controlled unpack support limiting returned objects
  - Add Lua 5.3 compatibility
  - Remove an unnecessary malloc
  - Use Lua memory allocator instead of malloc for buffer creation

Issues involved:
  - closes antirez#16 - allow multi pack/unpack by default
  - closes antirez#10 - unpack one/limit API added
  - closes antirez#13 and closes antirez#20 - use Lua allocator
  - closes antirez#15 - (included in antirez#16)
  - ignores antirez#22 because it's confusing
  - closes antirez#23 - fixed elsewhere
  - closes antirez#26 - extracted some useful parts from a difficult commit
  - closes antirez#28 - we started tagging versions again recently
  - closes antirez#27 - that failure case works for me now

I merged commits with original author information where possible, but
each commit required manual cleanup of one or more of:
formatting fixes (no tabs, please), commit message fixes (more details
please), extracting contents from a single 300 line commit with 5
different logical changes merged together, and general correctness
checking after merging with newer code.

As of this commit, all tests pass on Lua 5.1.5 and Lua 5.3-work2.
mattsta added a commit to mattsta/lua-cmsgpack that referenced this issue Apr 7, 2014
Changes:
  - Improve table vs. array detection
  - Improve packing +-inf
  - Add multiple pack/unpack support
  - Add cmsgpack.safe module variant
  - Add local build infrastructure for easier testing
  - Add user-controlled unpack support limiting returned objects
  - Add Lua 5.3 compatibility
  - Remove an unnecessary malloc
  - Use Lua memory allocator instead of malloc for buffer creation

Issues involved:
  - closes antirez#16 - allow multi pack/unpack by default
  - closes antirez#10 - unpack one/limit API added
  - closes antirez#13 and closes antirez#20 - use Lua allocator
  - closes antirez#15 - (included in antirez#16)
  - ignores antirez#22 because it's confusing
  - closes antirez#23 - fixed elsewhere
  - closes antirez#26 - extracted some useful parts from a difficult commit
  - closes antirez#28 - we started tagging versions again recently
  - closes antirez#27 - that failure case works for me now

I merged commits with original author information where possible, but
each commit required manual cleanup of one or more of:
formatting fixes (no tabs, please), commit message fixes (more details
please), extracting contents from a single 300 line commit with 5
different logical changes merged together, and general correctness
checking after merging with newer code.

As of this commit, all tests pass on Lua 5.1.5 and Lua 5.3-work2.
mattsta added a commit to mattsta/lua-cmsgpack that referenced this issue Apr 7, 2014
Changes:
  - Improve table vs. array detection
  - Improve packing +-inf
  - Add multiple pack/unpack support
  - Add cmsgpack.safe module variant
  - Add local build infrastructure for easier testing
  - Add user-controlled unpack support limiting returned objects
  - Add Lua 5.3 compatibility
  - Remove an unnecessary malloc
  - Use Lua memory allocator instead of malloc for buffer creation

Issues involved:
  - closes antirez#16 - allow multi pack/unpack by default
  - closes antirez#10 - unpack one/limit API added
  - closes antirez#13 and closes antirez#20 - use Lua allocator
  - closes antirez#15 - (included in antirez#16)
  - ignores antirez#22 because it's confusing
  - closes antirez#23 - fixed elsewhere
  - closes antirez#26 - extracted some useful parts from a difficult commit
  - closes antirez#28 - we started tagging versions again recently
  - closes antirez#27 - that failure case works for me now

I merged commits with original author information where possible, but
each commit required manual cleanup of one or more of:
formatting fixes (no tabs, please), commit message fixes (more details
please), extracting contents from a single 300 line commit with 5
different logical changes merged together, and general correctness
checking after merging with newer code.

As of this commit, all tests pass on Lua 5.1.5 and Lua 5.3-work2.
mattsta added a commit to mattsta/lua-cmsgpack that referenced this issue Apr 7, 2014
Changes:
  - Improve table vs. array detection
  - Improve packing +-inf
  - Add multiple pack/unpack support
  - Add cmsgpack.safe module variant
  - Add local build infrastructure for easier testing
  - Add user-controlled unpack support limiting returned objects
  - Add Lua 5.3 compatibility
  - Remove an unnecessary malloc
  - Use Lua memory allocator instead of malloc for buffer creation

Issues involved:
  - closes antirez#16 - allow multi pack/unpack by default
  - closes antirez#10 - unpack one/limit API added
  - closes antirez#13 and closes antirez#20 - use Lua allocator
  - closes antirez#15 - (included in antirez#16)
  - ignores antirez#22 because it's confusing
  - closes antirez#23 - fixed elsewhere
  - closes antirez#26 - extracted some useful parts from a difficult commit
  - closes antirez#28 - we started tagging versions again recently
  - closes antirez#27 - that failure case works for me now

I merged commits with original author information where possible, but
each commit required manual cleanup of one or more of:
formatting fixes (no tabs, please), commit message fixes (more details
please), extracting contents from a single 300 line commit with 5
different logical changes merged together, and general correctness
checking after merging with newer code.

As of this commit, all tests pass on Lua 5.1.5 and Lua 5.3-work2.
mattsta added a commit to mattsta/lua-cmsgpack that referenced this issue Apr 11, 2014
Changes:
  - Improve table vs. array detection
  - Improve packing +-inf
  - Add multiple pack/unpack support
  - Add cmsgpack.safe module variant
  - Add local build infrastructure for easier testing
  - Add user-controlled unpack support limiting returned objects
  - Add Lua 5.3 compatibility
  - Remove an unnecessary malloc
  - Use Lua memory allocator instead of malloc for buffer creation

Issues involved:
  - closes antirez#16 - allow multi pack/unpack by default
  - closes antirez#10 - unpack one/limit API added
  - closes antirez#13 and closes antirez#20 - use Lua allocator
  - closes antirez#15 - (included in antirez#16)
  - ignores antirez#22 because it's confusing
  - closes antirez#23 - fixed elsewhere
  - closes antirez#26 - extracted some useful parts from a difficult commit
  - closes antirez#28 - we started tagging versions again recently
  - closes antirez#27 - that failure case works for me now

I merged commits with original author information where possible, but
each commit required manual cleanup of one or more of:
formatting fixes (no tabs, please), commit message fixes (more details
please), extracting contents from a single 300 line commit with 5
different logical changes merged together, and general correctness
checking after merging with newer code.

As of this commit, all tests pass on Lua 5.1.5 and Lua 5.3-work2.
mattsta added a commit to mattsta/lua-cmsgpack that referenced this issue Apr 11, 2014
Changes:
  - Improve table vs. array detection
  - Improve packing +-inf
  - Add multiple pack/unpack support
  - Add cmsgpack.safe module variant
  - Add local build infrastructure for easier testing
  - Add user-controlled unpack support limiting returned objects
  - Add Lua 5.3 compatibility
  - Remove an unnecessary malloc
  - Use Lua memory allocator instead of malloc for buffer creation

Issues involved:
  - closes antirez#16 - allow multi pack/unpack by default
  - closes antirez#10 - unpack one/limit API added
  - closes antirez#13 and closes antirez#20 - use Lua allocator
  - closes antirez#15 - (included in antirez#16)
  - ignores antirez#22 because it's confusing
  - closes antirez#23 - fixed elsewhere
  - closes antirez#26 - extracted some useful parts from a difficult commit
  - closes antirez#28 - we started tagging versions again recently
  - closes antirez#27 - that failure case works for me now
  - closes antirez#31 - fix comment typos

I merged commits with original author information where possible, but
each commit required manual cleanup of one or more of:
formatting fixes (no tabs, please), commit message fixes (more details
please), extracting contents from a single 300 line commit with 5
different logical changes merged together, and general correctness
checking after merging with newer code.

As of this commit, all tests pass on Lua 5.1.5 and Lua 5.3-work2.
mattsta added a commit to mattsta/lua-cmsgpack that referenced this issue Apr 14, 2014
Changes:
  - Improve table vs. array detection
  - Improve packing +-inf
  - Add multiple pack/unpack support
  - Add cmsgpack.safe module variant
  - Add local build infrastructure for easier testing
  - Add user-controlled unpack support limiting returned objects
  - Add Lua 5.3 compatibility
  - Remove an unnecessary malloc
  - Use Lua memory allocator instead of malloc for buffer creation

Issues involved:
  - closes antirez#16 - allow multi pack/unpack by default
  - closes antirez#10 - unpack one/limit API added
  - closes antirez#13 and closes antirez#20 - use Lua allocator
  - closes antirez#15 - (included in antirez#16)
  - ignores antirez#22 because it's confusing
  - closes antirez#23 - fixed elsewhere
  - closes antirez#26 - extracted some useful parts from a difficult commit
  - closes antirez#28 - we started tagging versions again recently
  - closes antirez#27 - that failure case works for me now
  - closes antirez#31 - fix comment typos

I merged commits with original author information where possible, but
each commit required manual cleanup of one or more of:
formatting fixes (no tabs, please), commit message fixes (more details
please), extracting contents from a single 300 line commit with 5
different logical changes merged together, and general correctness
checking after merging with newer code.

As of this commit, all tests pass on Lua 5.1.5 and Lua 5.3-work2.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants