Templates in use are being removed by alertmanager cleanup logic

[dpericaxon]

Describe the bug
With the implementation of Cortex with a sidecar, templates continuously get removed every 15seconds. Through a discussion on slack, the issue is that the local storage implementation doesn't fill in the Templates slice. So the cleanup logic marks all the templates as stale.

To Reproduce
Steps to reproduce the behavior:

1. Deploy Helm Chart https://github.com/cortexproject/cortex-helm-chart
2. Follow the instructions here to set up the sidecar configuration for alertmanager.
3. Add some templates into configmaps that the sidecar is watching. It will mount those configmaps onto the filesystem and then within 15 seconds they will be deleted.

Expected behavior
The cleanup logic should not delete templates in use.

Environment:

• Infrastructure: Kubernetes(AKS)
• Deployment tool: Cortex Helm Chart https://github.com/cortexproject/cortex-helm-chart

Additional Context
Our Configuration:

alertmanager:
  external_url: /api/prom/alertmanager
  enable_api: true
  data_dir: /data/
  storage:
    type : local
    local:
      path: /data

[alvinlin123]

Initial suspicions is that the local storage implementation does not fill in the Templates slice of alertspb.AlertConfigDesc struct, which is used to detect which template is in-use during staled template removal phase.

[alvinlin123]

We should cut a fix release for this bug.

[alanprot]

Hi..

Can you share where those templates files end up?

Lets say u have 2 users, with 1 templates each, seems that the folder structure should be:

/data/user1.yaml
/data/user1/templates/template-yyy-tpl
/data/user2.yaml
/data/user2/templates/template-xxx-tpl

Is that correct?