Closed
Description
Updated topic to mention ElasticCache and general Redis as well.
https://aws.amazon.com/about-aws/whats-new/2021/08/amazon-memorydb-redis/
Metadata
Metadata
Assignees
Type
Projects
Status
Shipped
Relationships
Development
Participants
Activity
gmcrocetti commentedon Sep 2, 2021
I'm putting myself available to tackle this issue.
heitorlessa commentedon Sep 4, 2021
danielloader commentedon Sep 4, 2021
Isn't it meant to be the same client implementation for both?
heitorlessa commentedon Sep 4, 2021
heitorlessa commentedon Sep 4, 2021
whardier commentedon Sep 4, 2021
Redis in general would be fairly rad to have available via an extra requirement. I wasn't entirely sure if a broad implementation in aws-lambda-powertools for redis support would ultimately support both ElasticCache and MemoryDB.
[-]Add support for MemoryDB as Idemopotency backend[/-][+]Add support for MemoryDB/ElasticCache/Redis as Idemopotency backend[/+]gmcrocetti commentedon Sep 9, 2021
@heitorlessa . Things have changed, won't have bandwidth to tackle this issue :/. I hope someone else will volunteer. Enjoy your time ! Obrigado 👍
20 remaining items
heitorlessa commentedon Dec 6, 2023
We're reviewing the last edge case on concurrency and docs tomorrow - hoping to get this released next week
github-actions commentedon Jan 10, 2024
This issue is now closed. Please be mindful that future comments are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.
github-actions commentedon Jan 19, 2024
This is now released under 2.32.0 version!
leandrodamascena commentedon Jan 30, 2024
To enable other runtimes and customers to benefit from the research and implementation carried out in this pull request, I have provided the Redis implementation text below.
Many thanks to @roger-zhangg for the partnership and deep dive into this extensive and fantastic work, which undoubtedly raised the bar for this project!
API Design
We kept the same user experience when switching from DynamoDB to Redis. This is very important because one of the core principles of Powertools is to ensure the developer experience is as seamless as possible.
By keeping the persistence abstraction layer interchangeable between DynamoDB and Redis, we enabled a smooth transition that required minimal code changes for developers. The interfaces remained consistent, reducing friction when migrating the data storage technology.
Connection
In the initial version, we provided a dedicated Connection Class to assist our customers in creating Redis connections. This class wrapped the Redis client, both standalone and cluster, enabling customers to provide their Redis connection details (host, port, passwords) for establishing connections. Once the connection was established, this class could be passed to the Idempotency Layer for use. However, this design had a few significant drawbacks. Firstly, it was challenging for this design to support Redis sentinel connections, as sentinel connections are set up differently from standalone and cluster connections. Secondly, the default Redis client had more than 40 parameters. In our connection design, we chose to support only the most commonly used ones, passing all other parameters using
**kwargsdirectly to the wrapped Redis Client. This could result in a less-than-optimal experience for our customers, as they would need to figure out the parameter names without IDE typing hints when passing parameters using**kwargs.Keeping these considerations in mind, we drafted the second version of the Redis Connection. We made a few changes to the logic in the Idempotency Layer class so that it can now accept an established Redis client. With this design, customers can pass in any Redis client they prefer, as long as it adheres to the schema defined in the protocol class. Additionally, using the original Redis clients allows customers to leverage their prior experience with Redis and easily transfer and adapt their existing code. However, after some discussions, we concluded that this design may not be user-friendly for individuals without prior Redis experience. Therefore, we believe it's still beneficial to provide a helper class for creating connections to assist such users.
In the third and final design, we have opted to implement a helper connection class that assists customers in creating Redis connections with only the most commonly used Redis parameters (host, port, username, password, db_index, url, ssl). Simultaneously, we enable our customers to bring their own Redis connection if they prefer. One common use case, for example, is when customers want to use Redis with their certificates. This added flexibility allows them to establish secure Redis connections using their custom certificates while benefiting from the simplified connection setup provided by our helper class.
Example
Orphan Records
Each idempotency record includes attributes such as
expire_time, andinprogress_expire_time. These records should be automatically deleted when the current time reachesexpire_timein the "completed" status orinprogress_expire_timein the "in_progress" status.However, due to factors like Lambda handler timeouts, exceptions, or potential Redis expiration issues, there may be instances where idempotency records persist in Redis even after the current time has exceeded
inprogress_expire_timewhile the record status is still "in_progress," or the current time has surpassedexpire_timewhile the record status is "completed." We refer to these invalid records as "Orphan Records."It's important to note that the method we implement to address these orphan records must be executed with caution to avoid potential race conditions. Further details on this issue will be elaborated upon in the following two paragraphs.
Redis HSET vs SET
In the idempotency workflow, we need to store idempotency records with multiple attributes in Redis. This can be achieved by using
HSETwith the idempotency key followed by attributes. Alternatively, we can encapsulate the idempotency key and all its attributes into a JSON format and employSETto store the entire JSON structure.In the initial design, we employed
HSETto set idempotency records.HSEToffers several advantages overSET: hash lookups are typically faster, andHSETallows us to set multiple attributes using the same hash key. This aligns perfectly with the requirements of idempotency records, as we need to store theidempotencykey,expire_time,in_progress_expire_time,status, andpayloadunder a single key. Thus, the utilization ofHSETfor storing idempotency records appears to be an optimal choice for this project.However, there are two major drawbacks using this method.
Firstly,
SETallows us to set an expiration time when creating the record, butHSETdoesn't offer this capability. To set an expiration time, we must make an additional Redis call usingEXPIREfor the respective key after usingHSET. This could lead to reduced performance since it requires sending two Redis calls, significantly increasing latency due to the double Round Trip Time (RTT) to Redis. This also introduces the potential for orphan records. For example, if the Lambda handler times out immediately after creating the Idempotency record but before setting the expiration time, the record will become an orphan record as it won't automatically expire in Redis. One way to optimize this issue is to use Redis commands like a pipeline to combineHSETandEXPIREinto a single Redis request. However, this workaround also increases the complexity of the code.Secondly,
SETallows us to use the "nx" (non-exist) parameter to write only on keys that do not exist. However,HSETdoes not support this "nx" parameter. "nx" plays a crucial role in preventing race conditions, which will be explained in the following section on Redis Race Conditions.During our experiments, we used
pipelineswithHSETto reduce the Round Trip Time (RTT). However,pipelinesintroduce complexity that can make code writing and testing more challenging:pipelinefailure, it becomes more challenging to identify which specific command in the batch caused the issues. Extra logging must be implemented.pipe = r.pipeline(transaction=True)Due to
HSETlacking support for "EX" (expiration time) and "NX" (non-exist), and the challenges associated withpipelines, we have made the decision to transition to usingSETin our final design.Redis Race condition
There are two potential race conditions in the Redis Idempotency workflow. Although the probability of these race conditions occurring is low, if not addressed effectively, certain payloads may bypass the idempotency layer. This could lead to the underlying Lambda handler executing more than once for identical payloads, which is an undesirable scenario for our customers when utilizing our idempotency layer.
The first potential race condition occurs when two Lambda handlers simultaneously perform
SET/HSEToperations without using 'nx' on a non-existent record. Consider a scenario where, at the same time, two lambda handlers with the same payload both useGETorHGETto verify if a particular record is empty. While they both assume the record is empty, they both decide to write to the record usingSET/HSET, resulting in the record being updated twice. Eventually, both Lambda handlers with the same payload pass the idempotency check and proceed to the execution stage. In this case, the idempotency layer would fails to prevent the handler from running twice on the same payload. See graph below.This race condition can be resolved by adding
nx=Truewhen usingSET. Once we applynx=TruetoSET, even if two handlers executeSETat the exact same time, only one of them will succeed, and the other handler will recognize that the record exists and return accordingly.The second potential race condition is similar to the first one but more complicated. This scenario occurs when two lambda handers are both trying to fix the same orphan record they found at the same time. In our idempotency workflow, if a Lambda handler encounters an existing idempotency record and identifies it as a corrupted or orphaned record, it proceeds to overwrite it with a valid record. However, in this case, we are attempting to overwrite a record in Redis, so we cannot use the
nx=Trueflag inSETas we are modifying an existing record. Consequently, this scenario introduces the possibility of a race condition where twoSEToperations are executed simultaneously by two different Lambda handlers, potentially resulting in the record being updated twice, and both lambda handlers advancing to the execution stage. This is a situation that the idempotency mechanism should prevent. One solution for this is to useSETon a key withnxto serve as a lock, ensuring that only the handler that successfully acquires the lock proceeds to update the orphan record. See graph below showing each scenario:without lock
with lock
Tests
We had challenges while writing tests to test the Redis interface and functionality, primarily because testing the Protocol without establishing a real connection is a little bit hard. Our initial approach was create an integration testing by running Redis locally in Docker containers, but spinning up container environments locally has some downsides, such as:
Our solution was to shift towards more functional testing by injecting a fake "Redis client" class instead of using a real Redis connection. This approach offers the following advantages:
Thanks