Closed
Description
Is your feature request related to a problem? Please describe.
AWS Lambda Powertools provides a nearly complete set of utilities for batch processing, but not Kinesis Data Firehose. Within our org having support for this in the Python version would be very useful. And in the long run including support for a unified batch processing.
Describe the solution you'd like
Add support for a KinesisFirehoseEvent
data class for request and response. As well has some payload decoding utilities
data
- can include support for payload decoding and cachinginvocationId
- can be added as a correlation pathresult
- in the response can be an enum ( 'Ok', 'Dropped', 'ProcessingFailed')
Describe alternatives you've considered
Building an unofficial data class, which does not make sense as their are already official ones for Java and Go.
Additional context
- Amazon Kinesis Data Firehose Request message event
{
"invocationId": "invoked123",
"deliveryStreamArn": "aws:lambda:events",
"region": "us-west-2",
"records": [
{
"data": "SGVsbG8gV29ybGQ=",
"recordId": "record1",
"approximateArrivalTimestamp": 1510772160000,
"kinesisRecordMetadata": {
"shardId": "shardId-000000000000",
"partitionKey": "4d1ad2b9-24f8-4b9d-a088-76e9947c317a",
"approximateArrivalTimestamp": "2012-04-23T18:25:43.511Z",
"sequenceNumber": "49546986683135544286507457936321625675700192471156785154",
"subsequenceNumber": ""
}
},
{
"data": "SGVsbG8gV29ybGQ=",
"recordId": "record2",
"approximateArrivalTimestamp": 151077216000,
"kinesisRecordMetadata": {
"shardId": "shardId-000000000001",
"partitionKey": "4d1ad2b9-24f8-4b9d-a088-76e9947c318a",
"approximateArrivalTimestamp": "2012-04-23T19:25:43.511Z",
"sequenceNumber": "49546986683135544286507457936321625675700192471156785155",
"subsequenceNumber": ""
}
}
]
}
- Amazon Kinesis Data Firehose Request ApacheLog
{
"invocationId": "invocationIdExample",
"region": "us-east-1",
"records": [
{
"recordId": "49546986683135544286507457936321625675700192471156785154",
"approximateArrivalTimestamp": 1495072949453,
"data": "NjQuMjQyLjg4LjEwIC0gLSBbMDcvTWFyLzIwMDQ6MTY6MTA6MDIgLTA4MDBdICJHRVQgL21haWxtYW4vbGlzdGluZm8vaHNkaXZpc2lvbiBIVFRQLzEuMSIgMjAwIDYyOTE=="
}
]
}
- Response example
{
"records": [
{
"recordId": "49572672223665514422805246926656954630972486059535892482",
"result": "Ok",
"data": "SEVMTE8gV09STEQ=",
"metadata": {
"partitionKeys": {
"customerId": "customerId",
"year": "2017",
"month": "01",
"date": "01",
"hour": "01",
"minute": "01"
}
}
}
]
}
Other language data classes: