From fa2b35026899ca4d7937e4b96fc9b991f6bc79a9 Mon Sep 17 00:00:00 2001 From: pwseg Date: Tue, 6 Aug 2024 17:28:50 -0400 Subject: [PATCH 1/6] hotfix --- src/unify/data-graph/data-graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/unify/data-graph/data-graph.md b/src/unify/data-graph/data-graph.md index ce10db7b9e..c179539fee 100644 --- a/src/unify/data-graph/data-graph.md +++ b/src/unify/data-graph/data-graph.md @@ -1,4 +1,4 @@ -[--- +--- title: Data Graph plan: unify beta: true From ba1bb9fb48a1a302cc775ef8334af3054277df1f Mon Sep 17 00:00:00 2001 From: Jatin Sahu Date: Fri, 9 Aug 2024 15:25:55 +0530 Subject: [PATCH 2/6] Added example for partial batch failure --- .../functions/destination-functions.md | 76 ++++++++++++++++--- 1 file changed, 64 insertions(+), 12 deletions(-) diff --git a/src/connections/functions/destination-functions.md b/src/connections/functions/destination-functions.md index 915f9f6d7d..5525557bb6 100644 --- a/src/connections/functions/destination-functions.md +++ b/src/connections/functions/destination-functions.md @@ -278,7 +278,61 @@ The [Public API](/docs/api/public-api) Functions/Preview endpoint also supports ### Handling batching errors -Standard [function error types](/docs/connections/functions/destination-functions/#destination-functions-error-types) apply to batch handlers. Segment attempts to retry the batch in the case of Timeout or Retry errors. For all other error types, Segment discards the batch. It's also possible to report a partial failure by returning status of each event in the batch. Segment retries only the failed events in a batch until those events are successful or until they result in a permanent error. +Standard [function error types](/docs/connections/functions/destination-functions/#destination-functions-error-types) apply to batch handlers. Segment attempts to retry the whole batch in the case of Timeout or Retry errors. For all other error types, Segment discards the batch. + +### Handling partial batch failures + +It's also possible to report a partial failure by returning status of each event in the batch. Segment retries only the failed events in a batch until those events are successful or until they result in a permanent error. + +```js +async function onBatch(events, settings) { + const responseArray = []; + for (let i = 0; i < events.length; i++) { + try { + await processEvent(events[i], settings); + responseArray.push({ + status: 200 // success + }); + } catch (error) { + let response; + switch (error.name) { + case 'ValidationError': + response = { + status: 400, // Bad request - non-retryable + errormessage: error.message + }; + break; + case 'RetryError': + response = { + status: 500, // Retry error + errormessage: error.message + }; + break; + default: + response = { + status: 500, // Internal server error + errormessage: 'An unexpected error occurred' + }; + break; + } + + responseArray.push(response); + } + } + console.log(responseArray); + return responseArray; +} + +async function processEvent(event, settings) { + // business logic + if (event.properties.shouldFail) { + throw new ValidationError('Validation Error: invalid field'); + } + if (event.properties.retry) { + throw new RetryError('Retry Error'); + } +} +``` ```json [ @@ -287,18 +341,18 @@ Standard [function error types](/docs/connections/functions/destination-function }, { "status": 400, - "errormessage": "Bad Request" + "errormessage": "Validation Error: invalid field" }, { "status": 200 }, { "status": 500, - "errormessage": "Error processing request" + "errormessage": "Retry Error" }, { "status": 500, - "errormessage": "Error processing request" + "errormessage": "An unexpected error occurred" }, { "status": 200 @@ -308,15 +362,13 @@ Standard [function error types](/docs/connections/functions/destination-function For example, after receiving the responses above from the `onBatch` handler, Segment only retries **event_4** and **event_5**. -| Error Type | Result | +| Event Status | Result | | ---------------------- | ------- | -| Bad Request | Discard | -| Invalid Settings | Discard | -| Message Rejected | Discard | -| RetryError | Retry | -| Timeout | Retry | -| Unsupported Event Type | Discard | - +| 200 (Sucess) | Success | +| 400 (ValidationError) | Discard | +| 200 (Sucess) | Success | +| 500 (RetryError) | Retry | +| 500 (Unsupported Event Type) | Retry | ## Save and deploy the function From c75d0f49a85fc251254047c1713d961597d14357 Mon Sep 17 00:00:00 2001 From: pwseg Date: Thu, 13 Feb 2025 14:35:21 -0600 Subject: [PATCH 3/6] reworded some stuff for clarity --- .../functions/destination-functions.md | 26 ++++++++++++------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/src/connections/functions/destination-functions.md b/src/connections/functions/destination-functions.md index 5525557bb6..cd826a81bc 100644 --- a/src/connections/functions/destination-functions.md +++ b/src/connections/functions/destination-functions.md @@ -278,11 +278,13 @@ The [Public API](/docs/api/public-api) Functions/Preview endpoint also supports ### Handling batching errors -Standard [function error types](/docs/connections/functions/destination-functions/#destination-functions-error-types) apply to batch handlers. Segment attempts to retry the whole batch in the case of Timeout or Retry errors. For all other error types, Segment discards the batch. +Standard [function error types](/docs/connections/functions/destination-functions/#destination-functions-error-types) apply to batch handlers. Segment handles errors differently depending on whether they affect the entire batch or individual events: ### Handling partial batch failures -It's also possible to report a partial failure by returning status of each event in the batch. Segment retries only the failed events in a batch until those events are successful or until they result in a permanent error. +If your destination can process some events while rejecting others, you can return a structured response indicating the status of each event. Segment retries only the failed events. + +The following example shows how to process each event separately and return a structured response: ```js async function onBatch(events, settings) { @@ -334,6 +336,8 @@ async function processEvent(event, settings) { } ``` +Here's an example response from the `onBatch` handler: + ```json [ { @@ -360,15 +364,17 @@ async function processEvent(event, settings) { ] ``` -For example, after receiving the responses above from the `onBatch` handler, Segment only retries **event_4** and **event_5**. +After receiving these responses from `onBatch`, Segment only retries **event_4** and **event_5**. + +| Event Status | Result | +| ---------------------------- | ------- | +| 200 (Sucess) | Success | +| 400 (ValidationError) | Discard | +| 200 (Sucess) | Success | +| 500 (RetryError) | Retry | +| 500 (Unsupported Event Type) | Retry | -| Event Status | Result | -| ---------------------- | ------- | -| 200 (Sucess) | Success | -| 400 (ValidationError) | Discard | -| 200 (Sucess) | Success | -| 500 (RetryError) | Retry | -| 500 (Unsupported Event Type) | Retry | +This approach lets you handle errors at the event level while still benefiting from batching efficiency. ## Save and deploy the function From 1437e22975eb68ac144d20f33a838cf6fd3c2c34 Mon Sep 17 00:00:00 2001 From: pwseg Date: Thu, 13 Feb 2025 14:36:01 -0600 Subject: [PATCH 4/6] style status codes in table --- .../functions/destination-functions.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/src/connections/functions/destination-functions.md b/src/connections/functions/destination-functions.md index cd826a81bc..615ca54c40 100644 --- a/src/connections/functions/destination-functions.md +++ b/src/connections/functions/destination-functions.md @@ -366,13 +366,14 @@ Here's an example response from the `onBatch` handler: After receiving these responses from `onBatch`, Segment only retries **event_4** and **event_5**. -| Event Status | Result | -| ---------------------------- | ------- | -| 200 (Sucess) | Success | -| 400 (ValidationError) | Discard | -| 200 (Sucess) | Success | -| 500 (RetryError) | Retry | -| 500 (Unsupported Event Type) | Retry | +| Event Status | Result | +| ----------------------- | ------- | +| `200` (Success) | Success | +| `400` (ValidationError) | Discard | +| `200` (Success) | Success | +| `500` (`RetryError`) | Retry | +| `500` (Internal Error) | Retry | + This approach lets you handle errors at the event level while still benefiting from batching efficiency. From 7e7eb4d6f55fd22ef862c0b611d1628fe878b7a4 Mon Sep 17 00:00:00 2001 From: pwseg Date: Thu, 13 Feb 2025 14:37:37 -0600 Subject: [PATCH 5/6] added intro table to sentence --- src/connections/functions/destination-functions.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/connections/functions/destination-functions.md b/src/connections/functions/destination-functions.md index 615ca54c40..80809dc4da 100644 --- a/src/connections/functions/destination-functions.md +++ b/src/connections/functions/destination-functions.md @@ -366,6 +366,8 @@ Here's an example response from the `onBatch` handler: After receiving these responses from `onBatch`, Segment only retries **event_4** and **event_5**. +The following table shows how Segment processes different event statuses. Segment retries events with a retryable error and discards others or marks them as successfully processed: + | Event Status | Result | | ----------------------- | ------- | | `200` (Success) | Success | From 62700d23aac8659888f6196d1daed09228840f24 Mon Sep 17 00:00:00 2001 From: pwseg Date: Thu, 13 Feb 2025 14:38:42 -0600 Subject: [PATCH 6/6] make this an h4 --- src/connections/functions/destination-functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/functions/destination-functions.md b/src/connections/functions/destination-functions.md index 80809dc4da..46da9750c8 100644 --- a/src/connections/functions/destination-functions.md +++ b/src/connections/functions/destination-functions.md @@ -280,7 +280,7 @@ The [Public API](/docs/api/public-api) Functions/Preview endpoint also supports Standard [function error types](/docs/connections/functions/destination-functions/#destination-functions-error-types) apply to batch handlers. Segment handles errors differently depending on whether they affect the entire batch or individual events: -### Handling partial batch failures +#### Handling partial batch failures If your destination can process some events while rejecting others, you can return a structured response indicating the status of each event. Segment retries only the failed events.