Support other block storage options as a storage backend for runtime configuration file

[anna-tran]

Is your feature request related to a problem? Please describe.
Currently Cortex reads the runtime configuration file from filesystem. Cortex should be able to support other storage backends that Cortex can read the configuration file from (e.g. S3) in order to better keep track of versioning and easier access.

This is already supported today for other Cortex configurations like
blocks storage config.

Describe the solution you'd like
Inline the storage bucket config with the runtime configuration Config as done for other components like rule store.

By default the storage backend will be Filesystem, the same as it is today.

[marianafranco]

This can be useful for k8s cortex installations with many per-user limits overrides which are unable to use a ConfigMap as the storage solution for runtime configs given its 1 MiB limit.

[marianafranco]

@anna-tran The idea is to allow something like this, right?

runtime_config:
  # existent field
  [period: <duration> | default = 10s]

  # existent field
  [file: <string> | default = ""]

  [backend: <string> | default = "filesystem"]

  s3: 
      [endpoint: <string> | default = ""]

      [region: <string> | default = ""]

      [bucket_name: <string> | default = ""]
       ...

  filesystem:
      [dir: <string> | default = ""]

   gcs:
       [bucket_name: <string> | default = ""]
       
       [service_account: <string> | default = ""]

[anna-tran]

@marianafranco Yes, for now we can start with only s3 and filesystem as options for the backend and extend to other storage/cloud services. We can keep the runtime_config.file as a legacy field and copy over the configuration to the runtime_config.filesystem.dir if it's not configured.

[marianafranco]

Don't we need the runtime_config.file since the S3 config don't have a way to specify the file name?

cortex/pkg/storage/bucket/s3/config.go

Lines 60 to 72 in 3d1315a

	type Config struct {
	Endpoint string `yaml:"endpoint"`
	Region string `yaml:"region"`
	BucketName string `yaml:"bucket_name"`
	SecretAccessKey flagext.Secret `yaml:"secret_access_key"`
	AccessKeyID string `yaml:"access_key_id"`
	Insecure bool `yaml:"insecure"`
	SignatureVersion string `yaml:"signature_version"`
	BucketLookupType string `yaml:"bucket_lookup_type"`
	SSE SSEConfig `yaml:"sse"`
	HTTP HTTPConfig `yaml:"http"`
	}

[yeya24]

Don't we need the runtime_config.file since the S3 config don't have a way to specify the file name?

Yeah I think we can reuse that field for the filename for S3. Just wanna double check, do we have any use case of using dynamic file name? If not, then this should be good enough.

[alvinlin123]

Should we start with a proposal PR?

[anna-tran]

do we have any use case of using dynamic file name

I think to keep it simple we’ll require the file name to be specified like we do today.

@alvinlin123 I will start the proposal PR for it today