Skip to content

feedback: Binary protocol | Tarantool #1661

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
TarantoolBot opened this issue Nov 27, 2020 · 18 comments
Closed

feedback: Binary protocol | Tarantool #1661

TarantoolBot opened this issue Nov 27, 2020 · 18 comments
Assignees
Labels
need feedback [special status] On hold, awaiting feedback

Comments

@TarantoolBot
Copy link
Collaborator

<…>ed as keys within MP_MAP maps.

Binary protocol – illustration
|To follow the examples in this section,
get a single Linux computer and start three command-line shells (“terminals”).|

– On terminal #1, Start monitoring port 3302 with tcpdump:
sud<…>

https://www.tarantool.io/en/doc/2.3/dev_guide/internals/box_protocol/#box-protocol-join

Неужели это все действительно нужно для того, чтобы понять примеры на этой страничке?. Мне кажется достаточно сказать, что это msgpack (линк msgpack.org) и дать пример этого msgpack.

@pgulutzan
Copy link
Contributor

Examples and illustrations are never necessary, they merely -- we hope -- make it easier to see exactly what is going back and forth. As for MessagePack, we refer to its "spec" page but we had to mention that there are slight modifications, and our list of constants is actually not from there.

@alyapunov
Copy link

I agree, examples are useful, there are more useful if they are understandable. It think that the example is not because 1)it uses definitions that will be introduced two pages later 2)actual tcpdump (I tried) prints quite different output 3)as for me, tcpdump itself is not obvious thing 4)it invites to read a couple of dozens of lines of net_box sources. Actually the select protocol seems to be easier that this example of it.
And I don't like that introduction grows from year to year on this page. This example will be good if placed in the end of page but we should give a user protocol specification on the top of page.

@Totktonada
Copy link
Member

I would describe the protocol just in formatted JSON / YAML (with constant names instead of actual values), provide contant name-value table in a separate section and add a link to the msgpack standard. This way it should be as easy as possible to understand.

Despite I know the protocol and the msgpack standard (not all codes, okay), it is hard for me to read the documentation. The protocol is much more simple comparing to the documentation.

It is hard to defend 'simper' and 'harder' in a formal way. Let's just ask several more developers. @Mons, @kyukhin, @Korablev77, @Gerold103, guys, I would appreciate your feedback.

@pgulutzan
Copy link
Contributor

pgulutzan commented Jan 21, 2021

alyapunov was correct that the tcpdump result was different from the illustration, I believe that is fixed now.
It is also correct that we invite protocol developers to look at net_box source lines, I do not regard that as a flaw.
I do not know what "two pages later" means.
There already is a link to the msgpack standard, but it is insufficient because our list of constants is actually not from there.
I think the suggestion "formatted JSON / YAML" would not be helpful to a developer who wants to work on (say) a new connector, since they want to see what is going back and forth on the wire, and that is a bunch of bytes, not a formatted JSON or YAML.
There are some places where I believe we should change MP_INT to MP_UINT, I emailed about this on August 16.

@Totktonada
Copy link
Member

Nope, most of time a developer work with an msgpack library, not 'with bytes'. Even if you implement the library yourself, you'll implement the library first and than work with integer/string/bytes/array/map — (almost) JSON types. Mixing of different abstraction levels will make the code harder to understand and maintain. And I propose to get rid of such mixing in our documentation.

JSON/YAML is the human readable and well known representation of (almost) the same set of values as msgpack offers.

The main purpose of the protocol description is to describe the protocol. Not msgpack.


I wrote several tiny examples. They don't give exhaustive descriptions and they could be even imprecise or wrong. I just want to propose a syntax that is easy to read and translate to/from msgpack.

Example (SQL insert without bindings):

Request:

<size>: msgpack(size(<header>) + size(<body>))
<header>: msgpack({
    IPROTO_REQUEST_TYPE: IPROTO_EXECUTE,
    IPROTO_SYNC: <integer>,
})
<body>: msgpack({
    IPROTO_SQL_TEXT: <string>,
})

Response:

<size>: msgpack(size(<header>) + size(<body>))
<header>: msgpack({
    IPROTO_SCHEMA_VERSION: <integer>,
    IPROTO_STATUS_KEY: IPROTO_OK,
    IPROTO_SYNC: <integer>,
})
<body>: msgpack({
    IPROTO_SQL_INFO: {
        IPROTO_SQL_INFO_ROW_COUNT: <integer>,
    }
})

Example of constant definitions:

Response / request keys Encoded as Applicable for
IPROTO_REQUEST_TYPE 0x00 Request
IPROTO_STATUS_KEY 0x00 Response
IPROTO_SYNC 0x01 Both
IPROTO_SCHEMA_VERSION 0x05 Both
IPROTO_SQL_TEXT 0x40 Request
IPROTO_SQL_INFO 0x42 Response
<...> <...> <...>
SQL info keys Encoded as
IPROTO_SQL_INFO_ROW_COUNT 0x00
<...> <...>
IPROTO_REQUEST_TYPE values Encoded as
IPROTO_EXECUTE 0x0b
<...> <...>
IPROTO_STATUS_KEY values Encoded as
IPROTO_OK 0x00
<...> <...>

@Korablev77
Copy link

During work on C++ connector I've faced troubles with documentation as well (had to scroll over the pages to find and understand the structure of request/response etc). It would be convenient having references (in tables or/and jsons) as Alexander proposed in the post above.

@pgulutzan pgulutzan self-assigned this Jan 21, 2021
@pgulutzan
Copy link
Contributor

I will make a large change trying to follow the suggestions above. I have assigned this to myself temporarily.

@pgulutzan
Copy link
Contributor

pgulutzan commented Jan 26, 2021

I have changed the section:
moved the byte-code illustrations to the end,
added a list of constants,
and instead of box diagrams we have YAML-like code descriptions
(although not exactly as suggested because the convention in our
manual is to use italics not <...> for replaceable code).
Now compare the changed version
https://www.tarantool.io/en/doc/2.6/dev_guide/internals/box_protocol/#box-protocol
with the unchanged version
https://www.tarantool.io/en/doc/2.5/dev_guide/internals/box_protocol/#box-protocol
and please comment: which looks better?
Previous comments were from @alyapunov + @Totktonada + @Korablev77.

@pgulutzan
Copy link
Contributor

@NickVolynkin: See my comment above, from 19 days ago.
There has been no answer from @alyapunov or @Totktonada or @Korablev77.
So I ask you to please comment: which looks better?

@Korablev77
Copy link

@pgulutzan For some reason content of changed version looks not formatted:
iproto1
iproto2

IMHO it's hard to read doc in such form (I'm using Google Chrome Version 80.0.3987.122 on Linux Mint 19.3 Cinnamon)...

@NickVolynkin
Copy link
Contributor

@Korablev77 we know about formatting and will fix it in the current sprint.

@pgulutzan
Copy link
Contributor

If it is hard to read because of the text colour and font size: I acknowledged that this is not what Totktonada suggested, I explained "because the convention in our manual is to use italics not <...> for replaceable code". The way I did that is by concatenating :samp: and :code: on separate lines (the customized :codeitalic: etc. stuff was not working and probably wouldn't look much different). I believe NickVolynkin is referring to Issue#1801 Update formatting in "Binary protocol".

Also, for the request to list all the IPROTO constants at the start, I did not put each constant on a separate line, so that people can scroll past it more quickly. Try to remember that the poor reader has already had to scroll past such a list because IPROTO constants are also in the "Index" at the top of the page, with separate lines.

@NickVolynkin
Copy link
Contributor

Hi Peter @pgulutzan

So I ask you to please comment: which looks better?

In my opinion, the new version is structurally better, and it solves the original issue.

Although, formatting makes it harder to read. Fixing it (#1801) can be a good first issue for our new trainee, @xuniq. Of course, you're welcome to comment and review the new changes.

@pgulutzan
Copy link
Contributor

@NickVolynkin: Okay, then we will not go back to the earlier version.
I have already commented, explaining that I used :samp: and :code: so that replaceable code will be in italics, which code-block:: yaml will not do. As I said in my reply to @Korablev77, I am guessing that this "hard to read' complaint has something to do with colour or font size, but I am just guessing. Since you decided that issue#1736 (Roles should be fixed or abandoned) should be fixed, and it was fixed, we can test this. You can look at https://www.tarantool.io/en/doc/latest/dev_guide/internals/box_protocol/
and search for this line: "Byte codes for the response to the box.space.space-name:insert{6} example".
You are seeing italic because of :codeitalic: and normal code because of :codenormal:. Is it less "hard to read"?

@NickVolynkin
Copy link
Contributor

NickVolynkin commented Feb 18, 2021

explaining that I used :samp: and :code: so that replaceable code will be in italics, which code-block:: yaml will not do

I see, highlighting replaceable code is important in this document.

@Onvember just helped me with a solution for this. We have a way to both highlight some text and have it in a code block:

..  cssclass:: highlight
..  parsed-literal::

    :samp:`box.space.{space-name}:create_index('{index-name}')`

Output looks like this:

image

The highlighting with italics isn't very noticeable at this moment. We will fix it later.

@pgulutzan
Copy link
Contributor

That is an intelligent suggestion, parsed-literal
https://docutils.sourceforge.io/docs/ref/rst/directives.html#parsed-literal
is easier to use for fixed-width indenting and italics,
although one must be careful with escaping special characters.
Our guideline
https://www.tarantool.io/en/doc/latest/contributing/docs/markup/
does not mention parsed-literal but it is used elsewhere in the manual.
To me white-on-black does not look better than the colour that is there now,
but if you and @Korablev77 agree that this makes the "hard to read" complaint
disappear, should this issue be closed?

@NickVolynkin
Copy link
Contributor

@pgulutzan we'll fix the styles soon, so that highlighting will look better. Maybe we'll use italics and color, or just color. For instance, I like this example:

image

The guideline doesn't mention it yet, right. Most of us didn't even know about parsed-literal until yesterday. :)

@pgulutzan pgulutzan added the need feedback [special status] On hold, awaiting feedback label Apr 29, 2021
@patiencedaur patiencedaur added this to the iproto milestone Jan 19, 2022
@patiencedaur patiencedaur removed this from the iproto milestone Feb 1, 2022
@patiencedaur
Copy link
Contributor

In fact, I'd like to see the Binary protocol article restructured. However, this is for another issue. Closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
need feedback [special status] On hold, awaiting feedback
Projects
None yet
Development

No branches or pull requests

7 participants