Access defined runtime environment variables from powertools

[Fahadmo]

Is your feature request related to a problem? Please describe.

AWS Lambda has defined runtime environment variables as shown here: https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html#configuration-envvars-runtime

In order to get the name of the variable we often need to go back to the documentation which can be frustrating.

Describe the solution you'd like

I would like to have something that would let me access all defined environment variables.
For example, this could be a class with a list of cached properties for each defined runtime variable.

@cache 
@property
def AWS_REGION(self):
    return os.environ.get("AWS_REGION")

Describe alternatives you've considered
Currently, we just run the following for each environment variable we need.

x = os.environ.get(<ENV VARIABLE NAME>)

[michaelbrewer]

@heitorlessa :)

I would like this by like this: https://www.starlette.io/config/

[saragerion]

@Fahadmo personally speaking, I think this idea is brilliant. Thanks for sharing it with us.

AppConfig is outside the scope of this issue, but curious to hear about how do you see this functionality working together with AppConfig (another configuration source for Lambda)?
https://aws.amazon.com/blogs/mt/introducing-aws-appconfig-lambda-extension-deploying-application-configuration-serverless/
Asking this not bring this AppConfig functionality into this issue, but just to understand better what is the desired DX considering that there might be more than 1 data source in the future.

[michaelbrewer]

@heitorlessa - So far i have found a couple interesting libraries as an inspiration for a simple framework for environment variables, defaults (.env or json settings) and parameters (appconfig, ssm, secrets manager etc..):

• python decouple and prettyconf - what i like about these libraries is how simple and lightweight they are. Prettyconf does have an example of using SSM.
• Starlette - config - is much like python decouple.
• dynaconf is super fancy and allows for nested properties and automatic type casting.

Personally i kind of likely the simpler prettyconf or Starlette - config api:

config = Configuration(loaders=[Environment(), EnvFile(filename=env_file), ParameterStore(), ...])
DEBUG = config('DEBUG', cast=bool, default=False)
DATABASE_URL = config('DATABASE_URL', cast=databases.DatabaseURL)
SECRET_KEY = config('SECRET_KEY', cast=Secret)
ALLOWED_HOSTS = config('ALLOWED_HOSTS', cast=CommaSeparatedStrings)

• loaders - could be environment then a json config file. Or SSM Parameter Store and then environment variables.
• cast - could be list, dict, json etc..

[ran-isenberg]

For people who use the parser and pydantic, there's already a very simple solution.
You can see the implementation here: https://isenberg-ran.medium.com/aws-lambda-environment-variables-best-practices-f760384c23ed. I have a branch and tests ready to go if you want.
It does not require adding any new libraries or dependencies.

Pydantic also has support for secrets and loading configuration from files.

Just a thought ;)
@heitorlessa

[michaelbrewer]

Simple but I want a zero dependency version.

[ran-isenberg]

So you dont want to add prettyconf or Starlette ? just implement a leaner version?

[michaelbrewer]

no. just the UX.

[michaelbrewer]

"couple interesting libraries as an inspiration for a simple framework"

[ran-isenberg]

I respect that.
I'm usually on the other side - I will add external dependencies if they are reliable, have good docs, the UX is great and somebody is maintaining them. If that does not exist, I will write the code myself.

[michaelbrewer]

i don't see the need for 70mb dependency i don't want for a simple environment utility which can be performs using vanilla python 3.7+. When a lambda is >5mb in size the performance drops too much for me use case.

Hopefully v2 of powertools is closer on the roadmap to allow for even smaller lambdas.

[michaelbrewer]

Reference: https://mikhail.io/serverless/coldstarts/aws/

[ran-isenberg]

From my experience, the init time for my lambdas is in the regular time, no performance issues (and i monitor them).
Cold starts are solved by using provisioned concurrency.
And you can still add any of the other tools you mentioned above, you dont have to use pydantic. I'm sure they are not 70 megas too.

And about pydantic - I use it in so many other use cases that's it's as integral as powertools. But to each his own.

[michaelbrewer]

i just don't want to use pydantic :(

[ran-isenberg]

and that's just fine :)

[michaelbrewer]

provisioned concurrency is akin to switching to fargate

[michaelbrewer]

@heitorlessa - would it be possible to expand on what Feature: Expose Lambda Env Vars to prevent additional hops to the documentation means? Is this some kind of class that abstracts the environment variables much like the typescript version does this? Maybe once the requirements are agreed to, a "Good first issue" label could be added for contributors to look at :)

[ran-isenberg]

You can see a full implementation (with tests and full documentation) in my best practices github project.
Here's the env parser : https://github.com/ran-isenberg/aws-lambda-handler-cookbook/blob/main/service/handlers/utils/env_vars_parser.py
the docs:
https://ran-isenberg.github.io/aws-lambda-handler-cookbook/best_practices/environment_variables/

The schema for the env vars: https://github.com/ran-isenberg/aws-lambda-handler-cookbook/blob/main/service/handlers/schemas/env_vars.py

and a blog explaining why this is important ;)
https://www.ranthebuilder.cloud/post/aws-lambda-cookbook-environment-variables

@heitorlessa FYI in case you want it for the extras package

[michaelbrewer]

@ran-isenberg zero dependencies?

Coz I thought this would be part of core library, to reduce the use of static constants and cut down on the boilerplate.

As Powertools uses environment variables for a lot of things, log level, feature toggles or overriding service names. It should just be very something and lightweight

1. No additional dependencies
2. Some basic handling of str to bool, or str to enum, or str to int.
3. An easier way to document these internal env names and types
4. An easier way to discover what settings are supported by the internal libraries.

If can of coz be extended to do extra stuff in your BYO implementations. Or just to append your own set of environment variables.

[ran-isenberg]

@ran-isenberg zero dependencies?

Coz I thought this would be part of core library, to reduce the use of static constants and cut down on the boilerplate.

As Powertools uses environment variables for a lot of things, log level, feature toggles or overriding service names. It should just be very something and lightweight

1. No additional dependencies
2. Some basic handling of str to bool, or str to enum, or str to int.
3. An easier way to document these internal env names and types
4. An easier way to discover what settings are supported by the internal libraries.

If can of coz be extended to do extra stuff in your BYO implementations. Or just to append your own set of environment variables.

No, like I said, it uses Pydantic for parsing, so it can only used in the extras. For those who already use pydantic, this is a no brainer.

[michaelbrewer]

Yep. Definitely a plus for the Pydantic specific lambdas and possible an entire theme on its own. Currently it looks like the project will be on lock down until the end of July. So hopefully after that there is some directions on

1. V2 - yah / nah
2. Modularization - yah / nah
3. Official lambda layer without boto3 (after integration testing is down)
4. What would be the way to vote / request / RFCs on new modules etc.

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.

[heitorlessa]

related: #1172

We're following closely Pydantic v2 (~17x faster and much smaller package) which could address this in a better way as defined in #1172.

[ran-isenberg]

@heitorlessa FYI: https://github.com/ran-isenberg/aws-lambda-handler-cookbook/blob/main/service/handlers/utils/env_vars_parser.py

[leandrodamascena]

Hello everybody! To add this feature, we need to support Pydantic and validate environment variables against a Pydantic model. We need to serialize different types like string, int, classes and others. To optimize Lambda execution, we need to cache models and values to avoid running Pydantic operations every time, and other things.

The things mentioned above make creating this utility complex and at this moment we are focusing on other things like OpenAPI specifications, external observability provider, new backend for Idempotency and others. Alternatively, we suggest using the AWS Lambda Environment Variables Modeler (Python) project. @ran-isenberg is one of the Powertools for AWS Lambda (Python) contributors and created this project. I'm sure Ran will be happy to address feedback, implement new features, and fix bugs if they exist.

AWS Lambda Environment Variable Modeler (Python) Features:

• Validates the environment variables against a Pydantic model: define both semantic and syntactic validation.
• Serializes the string environment variables into complex classes and types.
• Provides means to access the environment variables safely with a global getter function in every part of the function.
• Provides a decorator to initialize the environment variables before executing a function.
• Caches the parsed model for performance improvement for multiple 'get' calls.

Thank you and I'm closing this issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.