AlertManager/ruler : Getting 400 error while alert is posting from ruler to alert manager

[venkatchinmay]

Describe the bug
we are getting 400 error while ruler is posting the alert to alert manager

To Reproduce
configurations used for alert manager

alertmanager:
data_dir: /tmp/cortex/alerts
external_url: "http://XXX.XXX.XXX.XXX:8094/"
storage:
type : local
local:
path: /tmp/cortex/alerts
enable_api: true
configurations used in ruler
ruler:
enable_api: true
enable_sharding: false
alertmanager_url: "http://XXX.XXX.XXX.XXX:8094"
storage:
type: local
local:
directory: /tmp/cortex/rules

ruler group(ruler1.yml)

groups:

• name: system_cpu_utilization
  rules:
  • alert: RESOURCE
    expr: sum(rate(system_cpu_utilization[5m])) > 0
    for: 1m
    labels:
    severity: critical
    annotations:
    summary: "getting alert from promethus"
    description: "getting critical alert"

Steps to reproduce the behavior:

1. Start Cortex with the target alert manager (v1.6.0)
2. Start Cortex with the target all (v1.6.0)
3. we are using cassandra storage for ingester data
4. we are using storage local for ruler and alert manager
5. we are storing ruler1.yml in /tmp/cortex/rules//rules1.yaml as mentioned in documentation
6. data is posting to cortex ingester data is populating but ruler is unable to post the alert

Expected behavior
we need to get the alert to alert manager
Environment:

• Infrastructure: laptop
• Deployment tool: docker
• docker run -d --name=cortex --network=opsramp_network -v /home/chinmay/cortex/docs/configuration/single-process-config.yaml:/etc/single-process-config.yaml -v /tmp:/tmp -p 9009:9009 quay.io/cortexproject/cortex:v1.6.0 -config.file=/etc/single-process-config.yaml
• docker run -d --name=cortex-alert-manager --network=opsramp_network -v /home/chinmay/cortex/docs/configuration/alert-manager/single-process-config.yaml:/etc/single-process-config.yaml -v /tmp:/tmp -p 8094:8094 quay.io/cortexproject/cortex:v1.6.0 -config.file=/etc/single-process-config.yaml

Storage Engine

• local

Additional Context
logs in ruler
level=error ts=2021-03-08T05:07:35.959544712Z caller=notifier.go:527 user=client_5 alertmanager=http://xxx.xxx.xxx.xxx:8094/api/v1/alerts count=1 msg="Error sending alert" err="bad response status 400 Bad Request" level=error ts=2021-03-08T05:09:35.318821594Z caller=notifier.go:527 user=client_5 alertmanager=http://xxx.xxx.xxx.xxx:8094/api/v1/alerts count=1 msg="Error sending alert" err="Post \"http://xxx.xxx.xxx.xxx:8094/api/v1/alerts\": EOF"
logs in alert manager
level=error ts=2021-03-08T05:07:35.958754394Z caller=api.go:93 component=MultiTenantAlertmanager org_id=client_5 traceID=4bebdda8804a1e63 msg="error marshalling YAML Alertmanager config" err="yaml: unmarshal errors:\n line 1: cannot unmarshal !!seq into alertmanager.UserConfig" level=error ts=2021-03-08T05:11:35.330058676Z caller=api.go:93 component=MultiTenantAlertmanager org_id=client_5 traceID=15874a85a25f01a4 msg="error marshalling YAML Alertmanager config" err="yaml: unmarshal errors:\n line 1: cannot unmarshal !!seq into alertmanager.UserConfig"

[gotjosh]

You're trying to send the alerts to the configuration API endpoint, instead of the Alertmanager.

The config value of:

alertmanager_url: "http://XXX.XXX.XXX.XXX:8094"

should be

alertmanager_url: "http://XXX.XXX.XXX.XXX:8094/alertmanager"

[venkatchinmay]

@gotjosh - it is trying hit and getting 404 error
http://XXX.XXX.XXX.XXX:8094/alertmanager/api/v1/alerts count=1 msg="Error sending alert" err="bad response status 404 Not Found
please give me the details of what payload ruler hit s the /api/v1/alerts (post) to alert manager. so that it will help to fix my issue
Thanks in advance

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[dcharbonnier]

got exactly the same issue, /alertmanager is for the api v2 but I also got a 404 and with v1 I got a 400

[arnaldomf]

@dcharbonnier I had the same issue, unfortunately the documentation is not really clear at this point. After a while, I solved the problem.

My environment looks like this:

ruler: 172.20.255.205:9009
alertmanager: 172.20.255.208:9009

In the alertmanager configuration file I set the external_url option:

alertmanager:
  external_url: "http://172.20.255.208:9009/alertmanager"

In the ruler configuration file I set the alertmanager_url option with the same value of external_url in the alertmanager configuration:

ruler:
  alertmanager_url: "http://172.20.255.208:9009/alertmanager"