-
Notifications
You must be signed in to change notification settings - Fork 163
Description
What would you like to be added:
To allow for a way to name processed correlation keys.
What I propose is to add a new key
or keyName
or whatever property to the correlationDef
object, so that multiple events can address different correlation keys:
...
events:
- name: ProductCreated
kind: consumed
source: ...
type: ...
correlation:
- contextAttributeName: subject
correlationKey: product # the processed key will be stored as 'product'
- name: ProductSold
kind: consumed
source: ...
type: ...
correlation:
- contextAttributeName: subject
correlationKey: product # the processed key will be stored as 'product'
- name: OrderCreated
kind: consumed
source: ...
type: ...
correlation:
- contextAttributeName: subject
correlationKey: order # the processed key will be stored as 'order'
...
Why is this needed:
Consider the following definition:
...
events:
- name: MyEvent1
kind: consumed
source: ...
type: ...
correlation:
- contextAttributeName: subject
- name: MyEvent2
kind: consumed
source: ...
type: ...
correlation:
- contextAttributeName: subject
- name: MyOtherEventThatHasNothingToDoWithTheTwoPrevious
kind: consumed
source: ...
type: ...
correlation:
- contextAttributeName: subject
...
Image we have a sequential flow such as the following:
- Consume MyEvent1 with subject 'foo'
- Consume MyEvent2 with subject 'foo'
- Consume MyOtherEventThatHasNothingToDoWithTheTwoPrevious with subject 'bar'
What would happen on Synapse is:
- Check for a match for a correlation mapping with the defined correlations for MyEvent1 (i.e. subject)
- No correlation mapping exist, so the event is consumed, and the mapping is set to 'subject: foo'
- Check for a match for a correlation mapping with the defined correlations for MyEvent2 (i.e. subject)
- Match exists, and is set to 'foo', so the event is correlated and therefore consumed
- Check for a match for a correlation mapping with the defined correlations for MyOtherEventThatHasNothingToDoWithTheTwoPrevious(i.e. subject)
- Match exists, but the value is not set to 'foo', so the event is consumed, and the mapping is set to 'subject: bar' ===> BOOOOOOOOM
I could very easily find a dirty work around that problem, but like I said, it would be dirty. This is, IMO, up to the workflow designer to decide how a newly created correlation key should be named.
Note: not setting the contextAttributeValue
(and that should be better explained in the spec) basically says that first event to come is gonna set the correlation key based on the defined context attribute (i.e. subject).
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Activity
cdavernas commentedon Sep 14, 2022
We could also possibly introduce a new
$CORRELATION
jq named parameter, which would allow workflows to retrieve and play with those keys.cdavernas commentedon Oct 27, 2022
@ricardozanini @tsurdilo May I proceed with a PR to address that?
It's a minor though very important feature for correlation.
cdavernas commentedon Oct 27, 2022
Ideally, we could also take the advantage of that feature to revamp correlation and add support for (discouraged but sometimes necessary) payload based (vs context attributes) correlation:
Correlate incoming event of drfined type and source using both (or maybe exclusive too?) event context attributes and payload:
cdavernas commentedon Oct 27, 2022
Finally, to support all use cases, we could also add a
condition
property on eventDef, which would allow to define whether or not to consume/produce events based on their attributes and or payload.Only consume defined event when the condition matches:
We would therefore leverage the full power of what we already have and of cloudevents, allowing for extremely complex correlations, which is IMHO one of the core features of the spec. Hell, I came across the spec back in the days because/thanks to cloudevents!!!
fjtirado commentedon Oct 27, 2022
@cdavernas
I was thinking a lot about correlation in the last months and Im not sure it is a good idea to try to simulate a correlation engine within the spec.
Wont it be better to actually get rid of correlation and simplify the spec?
I know the answer is no, but I need to try ;)
The reason Im proposing this apparently radical idea is because I feel we are trying to do the job of a correlation engine, which should be typically done by another process (Im thinking on Napoleon and his divide&conquer strategy) and because I think we should focus on clarifying the way a flow is started through events (I know Tiho is going to write something on that regard).
Anyway, if we are getting deep into correlation (as Im afraid we are going to do ;)), then my first impression is that rather than use byContextAttribute or byPayloadAttribute properties we should just use JQ expression to select the attributes that should be used for the correlation. If we are interested in a payload atttribute we should just write
.data.<payload attribute>
(basically what you proposed in your last post)Also, I think we need to rewrite some bits of the spec to clarify many tricky scenarios related with correlations, the one you describe is just one of them, but there are certainly more (and some of them are related with start event stated vs non start event states and how to identify an existing flow that needs to be notified)
Finally, I think this topic deserve a face to face preliminary meeting to discuss all the angles and possibilities, wdyt?
ricardozanini commentedon Nov 14, 2022
Complex event processing scenarios are a widespread use case and a notable research topic called Complex Event Processing. So I'd say we must be aware of that and stay away if possible. Otherwise, we end up writing a specification inside a specification.
I'd say that we need some correlation features, but they should be limited.
cdavernas commentedon Nov 14, 2022
@ricardozanini What I'm proposing is, I believe, reasonable and easily implemtable. I agree with @fjtirado, however, that the feature should be optional. It is imho a requirement for (complex, that is more than two even states) event based workflows.
cdavernas commentedon Nov 14, 2022
Ill work on a PR proposal that shapes my ideas as optional add ins to the spec if that ok with you guys
ricardozanini commentedon Nov 14, 2022
@cdavernas, absolutely! Nothing against this proposal; we just need to be aware that if we keep adding use cases and features on top of correlations we will build another beast. :D
cdavernas commentedon Nov 14, 2022
@ricardozanini yeah you are right. I'll keep it as concise as possible!
github-actions commentedon Dec 30, 2022
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
22 remaining items