Add ability to override configuration settings using environment variables

[Wing924]

What this PR does:

You can use environment variable references in the config file to set values that need to be configurable during deployment.
To do this, use:

${VAR}

Where VAR is the name of the environment variable.

Each variable reference is replaced at startup by the value of the environment variable.
The replacement is case-sensitive and occurs before the YAML file is parsed.
References to undefined variables are replaced by empty strings unless you specify a default value or custom error text.

To specify a default value, use:

${VAR:default_value}

Where default_value is the value to use if the environment variable is undefined.

Which issue(s) this PR fixes:

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pracucci]

Thanks for your contribution. Could you describe your use case please? Cortex configuration can be overridden via CLI flags and CLI flags are usually the expected way to override some config using values from environment variables.

[Wing924]

@pracucci
Now I use kustomize to manage k8s manifest. As I asked on the slack, it's better to put all configs on one config file (as a configmap) compared to set them by flags. The problem is that we can't change/override values on configmap per overlays.
Some values like db-url are different from each overlay. Some values like password are secrets, which don't want to put on plaintext.

As you say, put them on flags with environment variables is an option, but I don't like this because many components need to set the same flags:

# cassandra settings
- -cassandra.addresses=$(CASSANDRA_ADDRESSES)
- -cassandra.username=$(CASSANDRA_USERNAME)
- -cassandra.password=$(CASSANDRA_PASSWORD)

it looks very redundant to me.

[tomwilkie]

I'm not averse to this myself; having multiple ways of overriding config settings (flags and environment variables) is a oft requested feature in the Prometheus world, and the "config management is someone else's problem" approach has gone down particularly badly. Plus this PR includes docs and tests, so I can't find anything to nit pick!

That being said I can see this being a little divisive, so I want to get a feeling from the other maintainers. But I'm +1 on this.

[pracucci]

I agree you did a great job and I'm not against this change as well. I can see a value. However, I've the feeling it may be a possible breaking change (see the comment) reason why I'm wondering if we should enable this only via CLI flag.

[pracucci]

I would replace configuration settings with YAML config file settings to be more clear.

[pracucci]

You did a very good job! LGTM.

I'm going to ping other maintainers as well, to see if there's any strong opinion about not doing env vars expansion in the config.

[bboreham]

lgtm

[gouthamve]

++1