Skip to content

fix(idempotency): Log nested exception message #1813

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

mploski
Copy link
Contributor

@mploski mploski commented Jan 5, 2023

**Issue number: #1772

Summary

Raise IdempotencyPersistenceLayerError exception with additional context taken from raised downstream exception

Changes

Please provide a summary of what's being changed

  • Modify idempotency core logic to pass downstream exception to newly raised IdempotencyPersistenceLayerError
    exception
  • Create base exception class that formats output visible in stdout and inherit from it in idempotency exceptions classes
    Before this change:
[ERROR] IdempotencyPersistenceLayerError: ('Failed to save in progress record to idempotency store', ClientError('An error occurred (AccessDeniedException) when calling the PutItem operation: User: <REMOVED> is not authorized to perform: dynamodb:PutItem on resource: <REMOVED> because no identity-based policy allows the dynamodb:PutItem action'))

After:

[ERROR] IdempotencyPersistenceLayerError: Failed to save in progress record to idempotency store: (An error occurred (AccessDeniedException) when calling the PutItem operation: User: <REMOVED> is not authorized to perform: dynamodb:PutItem on resource:<REMOVED> because no identity-based policy allows the dynamodb:PutItem action)

This change is not mandatory. I'm happy to hear your opinion whether we should reformat exception message or not and what format would be the best

  • Adjust tests

User experience

Before then change all downstream exception details was hidden from user, making debugging harder.
Now downstream details are available
For details please follow: #1772

Checklist

If your change doesn't seem to apply, please leave them unchecked.

Is this a breaking change?

RFC issue number:

Checklist:

  • Migration process documented
  • Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@mploski mploski requested a review from a team as a code owner January 5, 2023 12:17
@mploski mploski requested review from rubenfonseca and removed request for a team January 5, 2023 12:17
@boring-cyborg boring-cyborg bot added the tests label Jan 5, 2023
@pull-request-size pull-request-size bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jan 5, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jan 5, 2023

⚠️Large PR detected⚠️

Please consider breaking into smaller PRs to avoid significant review delays. Ignore if this PR has naturally grown to this size after reviews.

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Jan 5, 2023

⚠️Large PR detected⚠️

Please consider breaking into smaller PRs to avoid significant review delays. Ignore if this PR has naturally grown to this size after reviews.

@mploski mploski force-pushed the fix/idempotency-print-nested-exception branch from 01b8fa5 to da141a6 Compare January 5, 2023 12:36
@pull-request-size pull-request-size bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 5, 2023
@mploski mploski force-pushed the fix/idempotency-print-nested-exception branch from da141a6 to 42acd50 Compare January 5, 2023 12:37
@pull-request-size pull-request-size bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 5, 2023
@leandrodamascena leandrodamascena self-assigned this Jan 5, 2023
else:
super().__init__()


Copy link
Contributor

@leandrodamascena leandrodamascena Jan 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @mploski! This is a huge improvement indeed. I remember the first time I needed to use the idempotency utility, I got an error due to permissions and spent a lot of time figuring out the reason for the "Failed to save record in progress to idempotency store" message. Creating a specific class to capture and return detailed errors makes life and adoption easier!!

I was wondering if we really need to instantiate the parent Exception class or if we could change this code to directly return the error message and make the code more clean. But it's just an idea to think about, I'm not sure and would just like to hear your side.

class BaseError(Exception):
     """
    Base error class that overwrites the way exception and extra information is printed.
    See https://github.com/awslabs/aws-lambda-powertools-python/issues/1772
    """

    def __init__(self, *args: str):
        self.message = args[0] if args else ""
        self.args = args

    def __str__(self):
        """
        Return all arguments formatted or original message
        """
        if self.args:
            return f"{' - '.join(str(arg) for arg in self.args)}"
        return self.message

Copy link
Contributor Author

@mploski mploski Jan 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @leandrodamascena , thx for the feedback!
I like your proposal and I refactored my code to follow your suggestion i.e to modify how exception is printed in str dunder method. This approach also has additional benefit that we keep initial attributes ( in args list) in object itself so we can easily see how exception object was initialized during debugging if needed.

As for the super class calling. In theory it is suggested to call it as explained in Python Cookbook book: https://books.google.pl/books?id=f7wGeA71_eUC&pg=PA579&dq=To+illustrate+the+use+of+.args,+consider+this+interactive+session+with+the+built-in+RuntimeError+exception,+and+notice+how+any+number+of+arguments+can+be+used+with+the+raise+statement:&hl=en&sa=X&ved=2ahUKEwinm_Dy0bX8AhWMposKHQNHB3oQ6AF6BAgBEAI#v=onepage&q=To%20illustrate%20the%20use%20of%20.args%2C%20consider%20this%20interactive%20session%20with%20the%20built-in%20RuntimeError%20exception%2C%20and%20notice%20how%20any%20number%20of%20arguments%20can%20be%20used%20with%20the%20raise%20statement%3A&f=false.
I dove deep more to see what happens when we call Exception init method. It seems this behavior changed somewhere in newer python 3 releases and args list initialization was moved to new dunder method so all needed stuff is initialized either way even without calling superclass init method ( as oppose to python 2.x and 3.0 for example) . So I think we can safely omit it.
here is a c code for new method in exception class in python 3 main branch: https://github.com/python/cpython/blob/3.7/Objects/exceptions.c#L47 - you can see that we copy args to instance attribute
and in Python 3.0:
https://github.com/python/cpython/blob/v3.0/Objects/exceptions.c#L49 - you can see that this action is done only in init not in new

Can I ask you to review it once again? :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @mploski! Thank you very much for your feedback and all the links provided, it was very helpful to understand more deeply how the exception is handled and it arguments.

From my side everything is ok and we are ready to merge!

@leandrodamascena
Copy link
Contributor

I did all the tests with this code and simulated several scenarios and in all I was successful. Really a big improvement this @mploski. I just commented one file and I will use your feedback to learn more about Python :)

image

@codecov-commenter
Copy link

codecov-commenter commented Jan 7, 2023

Codecov Report

Base: 97.58% // Head: 97.59% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (8e471b6) compared to base (e9aacfb).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #1813   +/-   ##
========================================
  Coverage    97.58%   97.59%           
========================================
  Files          141      141           
  Lines         6425     6434    +9     
  Branches       442      444    +2     
========================================
+ Hits          6270     6279    +9     
  Misses         123      123           
  Partials        32       32           
Impacted Files Coverage Δ
...ws_lambda_powertools/utilities/idempotency/base.py 100.00% <100.00%> (ø)
...bda_powertools/utilities/idempotency/exceptions.py 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Contributor

@leandrodamascena leandrodamascena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

else:
super().__init__()


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @mploski! Thank you very much for your feedback and all the links provided, it was very helpful to understand more deeply how the exception is handled and it arguments.

From my side everything is ok and we are ready to merge!

@leandrodamascena leandrodamascena merged commit e2cad31 into aws-powertools:develop Jan 9, 2023
@mploski mploski deleted the fix/idempotency-print-nested-exception branch January 9, 2023 13:42
@rubenfonseca rubenfonseca changed the title Fix(idempotency): Log nested exception message fix(idempotency): Log nested exception message Jan 12, 2023
@github-actions github-actions bot added the bug Something isn't working label Jan 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants