Support for Strongly Ordered Reciept and Transmission

[erikerikson]

For those who use ordering for its equivalence to consensus a potentially unordered transmission or receipt of CloudEvents could block use of this library. However, providing order guarantees has implications for performance and interaction models based on the targeted protocol and/or middleware. To be concrete, if every message must be acknowledged before the next can be sent, parallelism within an ordering scope cannot be achieved without some sort of reassembly or gap detection and retransmission mechanism. Without such a mechanism, senders and receivers must fallback to a rate of transmission that is the time of a round trip on the network rather than the time to mark a message for transmission. Because of the order of magnitude differences it will be unacceptable to require those without ordering requirements to accept the throughput implications of ordering but it will also be unacceptable not to offer an ordered transmission capability. Thus implementers of the mappers between the CloudEvents values and specific protocols that can support ordering guarantees should offer configuration controlling this behavior.

Original comment to which this issue relates.

[lance]

To validate this as an issue, I put together a little test. Here is an HTTP server that receives a number in a header, expecting it to monotonically increase with each incoming request. It then sets a timeout for a random amount of time up to one second and returns 200/OK after the timeout. If at any point, a number is received out of order

// server.js
const http = require('http');

let count = 0;
const server = http.createServer((req, res) => {
  const num = req.headers['x-num'];
  if (++count != num) {
    throw new Error(`Received request out of order. Expected ${count}. Got ${num}`)
  } else if (count % 1000 === 0) { console.log(count); }

  setTimeout(_ => res.end('OK'), getRandom(1000));
  res.end('OK')
});

function getRandom(max) {
  return Math.floor(Math.random() * Math.floor(max));
}

server.listen(8080);

Here is a simple client that sends a monotonically increasing number in the header, from 1 - 5000.

const http = require('http');

let i = 0;
while (++i < 5000) {
  http.get('http://localhost:8080', {
    headers: {
      'x-num': i
    }
  })
}

It doesn't take very long to see that this is a problem.

Error: Received request out of order. Expected 392. Got 393
    at Server.<anonymous> (/home/lanceball/src/scratch/ordering/server.js:7:11)
    at Server.emit (events.js:210:5)
    at parserOnIncoming (_http_server.js:745:12)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:115:17)

[lance]

Hmm. I should try the client using sh and curl. Node’s http.get() is also async.

[erikerikson]

I guess I thought that was exactly the point. By default HTTP operations are non-blocking so as not to starve other items on the eventloop. If they are used in the library, I would expect out of order sending/receiving eventually. Something like the following is required to guarantee in order submission:

const http = require('http');

const get = async num => new Promise(resolve => {
  http.get(
    'http://localhost:8080',
    { headers: { 'x-num': num } },
    () => resolve()
  )
})

const run = async () => {
  let i = 0;
  while (++i < 5000) {
    await get(i)
  }
}

run()

Note that I'm not doing anything with the responses but I have to wait for an acknowledgement by the server before I can be guaranteed that my message was received.

[erikerikson]

For the record, the above has issues but so long as the transmission of that x-num header is the extent to which order matters (as is the case in your server code), the above satisfies.

A minor note is that although this is marked with the transport/http tag, it should apply equally to any transport that is not already inherently ordered and for which providing delivery ordering guarantees isn't silly. For example, it would be silly to care where the middleware doesn't respect ordering.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days