Skip to content

feat(data-classes): Add KinesisFirehoseEvent #1059

Closed
@michaelbrewer

Description

@michaelbrewer

Is your feature request related to a problem? Please describe.

AWS Lambda Powertools provides a nearly complete set of utilities for batch processing, but not Kinesis Data Firehose. Within our org having support for this in the Python version would be very useful. And in the long run including support for a unified batch processing.

Describe the solution you'd like

Add support for a KinesisFirehoseEvent data class for request and response. As well has some payload decoding utilities

  • data - can include support for payload decoding and caching
  • invocationId - can be added as a correlation path
  • result - in the response can be an enum ( 'Ok', 'Dropped', 'ProcessingFailed')

Describe alternatives you've considered

Building an unofficial data class, which does not make sense as their are already official ones for Java and Go.

Additional context

  • Amazon Kinesis Data Firehose Request message event
{
  "invocationId": "invoked123",
  "deliveryStreamArn": "aws:lambda:events",
  "region": "us-west-2",
  "records": [
    {
      "data": "SGVsbG8gV29ybGQ=",
      "recordId": "record1",
      "approximateArrivalTimestamp": 1510772160000,
      "kinesisRecordMetadata": {
        "shardId": "shardId-000000000000",
        "partitionKey": "4d1ad2b9-24f8-4b9d-a088-76e9947c317a",
        "approximateArrivalTimestamp": "2012-04-23T18:25:43.511Z",
        "sequenceNumber": "49546986683135544286507457936321625675700192471156785154",
        "subsequenceNumber": ""
      }
    },
    {
      "data": "SGVsbG8gV29ybGQ=",
      "recordId": "record2",
      "approximateArrivalTimestamp": 151077216000,
      "kinesisRecordMetadata": {
        "shardId": "shardId-000000000001",
        "partitionKey": "4d1ad2b9-24f8-4b9d-a088-76e9947c318a",
        "approximateArrivalTimestamp": "2012-04-23T19:25:43.511Z",
        "sequenceNumber": "49546986683135544286507457936321625675700192471156785155",
        "subsequenceNumber": ""
      }
    }
  ]
}
  • Amazon Kinesis Data Firehose Request ApacheLog
{
  "invocationId": "invocationIdExample",
  "region": "us-east-1",
  "records": [
    {
      "recordId": "49546986683135544286507457936321625675700192471156785154",
      "approximateArrivalTimestamp": 1495072949453,
      "data": "NjQuMjQyLjg4LjEwIC0gLSBbMDcvTWFyLzIwMDQ6MTY6MTA6MDIgLTA4MDBdICJHRVQgL21haWxtYW4vbGlzdGluZm8vaHNkaXZpc2lvbiBIVFRQLzEuMSIgMjAwIDYyOTE=="
    }
  ]
}
  • Response example
{
  "records": [
    {
      "recordId": "49572672223665514422805246926656954630972486059535892482",
      "result": "Ok",
      "data": "SEVMTE8gV09STEQ=",
      "metadata": {
        "partitionKeys": {
          "customerId": "customerId",
          "year": "2017",
          "month": "01",
          "date": "01",
          "hour": "01",
          "minute": "01"
        }
      }
    }
  ]
}

Other language data classes:

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions