Skip to content

Change config flags, add flags to disable Alertmanager notifications methods: email & webhoook #2187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Mar 4, 2020
13 changes: 12 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,14 @@
* [CHANGE] Updated Prometheus dependency to v2.16.0. This Prometheus version uses Active Query Tracker to limit concurrent queries. In order to keep `-querier.max-concurrent` working, Active Query Tracker is enabled by default, and is configured to store its data to `active-query-tracker` directory (relative to current directory when Cortex started). This can be changed by using `-querier.active-query-tracker-dir` option. Purpose of Active Query Tracker is to log queries that were running when Cortex crashes. This logging happens on next Cortex start. #2088
* [CHANGE] Experimental TSDB: TSDB head compaction interval and concurrency is now configurable (defaults to 1 min interval and 5 concurrent head compactions). New options: `-experimental.tsdb.head-compaction-interval` and `-experimental.tsdb.head-compaction-concurrency`. #2172
* [CHANGE] Remove fluentd-based billing infrastructure and flags such as `-distributor.enable-billing`. #1491
* [CHANGE] Renamed Configs configuration options. #2187
* configuration options
* `-database.*` -> `-configs.database.*`
* `-database.migrations` -> `-configs.database.migrations-dir`
* config file
* `configdb.uri:` -> `configs.database.uri:`
* `configdb.migrationsdir:` -> `configs.database.migrations_dir:`
* `configdb.passwordfile:` -> `configs.database.password_file:`
* [CHANGE] Experimental TSDB: the querier in-memory index cache used by the experimental blocks storage shifted from per-tenant to per-querier. The `-experimental.tsdb.bucket-store.index-cache-size-bytes` now configures the per-querier index cache max size instead of a per-tenant cache and its default has been increased to 1GB. #2189
* [CHANGE] If you are vendoring Cortex and use its components in your project, be aware that many Cortex components no longer start automatically when they are created. You may want to review PR and attached document. #2166
* [CHANGE] Cortex now has /ready probe for all services, not just ingester and querier as before. In single-binary mode, /ready reports 204 only if all components are running properly. #2166
Expand All @@ -34,8 +42,11 @@
* `--experimental.distributor.user-subring-size`
* [FEATURE] Added flag `-experimental.ruler.enable-api` to enable the ruler api which implements the Prometheus API `/api/v1/rules` and `/api/v1/alerts` endpoints under the configured `-http.prefix`. #1999
* [FEATURE] Added sharding support to compactor when using the experimental TSDB blocks storage. #2113
* [FEATURE] Add ability to override YAML config file settings using environment variables. #2147
* [FEATURE] Added ability to override YAML config file settings using environment variables. #2147
* `-config.expand-env`
* [FEATURE] Added flags to disable Alertmanager notifications methods. #2187
* `-configs.notifications.disable-email`
* `-configs.notifications.disable-webhook`
* [FEATURE] Add /config HTTP endpoint which exposes the current Cortex configuration as YAML. #2165
* [FEATURE] Allow Prometheus remote write directly to ingesters. #1491
* [FEATURE] Add flag `-experimental.tsdb.stripe-size` to expose TSDB stripe size option. #2185
Expand Down
38 changes: 24 additions & 14 deletions docs/configuration/config-file-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,9 +118,8 @@ Where default_value is the value to use if the environment variable is undefined
# The ruler_config configures the Cortex ruler.
[ruler: <ruler_config>]

# The configdb_config configures the config database storing rules and alerts,
# and used by the 'configs' service to expose APIs to manage them.
[configdb: <configdb_config>]
# The configs_config configures the Cortex Configs DB and API.
[configs: <configs_config>]

# The alertmanager_config configures the Cortex alertmanager.
[alertmanager: <alertmanager_config>]
Expand Down Expand Up @@ -2081,22 +2080,33 @@ The `fifo_cache_config` configures the local in-memory cache.
[validity: <duration> | default = 0s]
```

### `configdb_config`
### `configs_config`

The `configdb_config` configures the config database storing rules and alerts, and used by the 'configs' service to expose APIs to manage them.
The `configs_config` configures the Cortex Configs DB and API.

```yaml
# URI where the database can be found (for dev you can use memory://)
# CLI flag: -database.uri
[uri: <string> | default = "postgres://[email protected]/configs?sslmode=disable"]
database:
# URI where the database can be found (for dev you can use memory://)
# CLI flag: -configs.database.uri
[uri: <string> | default = "postgres://[email protected]/configs?sslmode=disable"]

# Path where the database migration files can be found
# CLI flag: -database.migrations
[migrationsdir: <string> | default = ""]
# Path where the database migration files can be found
# CLI flag: -configs.database.migrations-dir
[migrations_dir: <string> | default = ""]

# File containing password (username goes in URI)
# CLI flag: -database.password-file
[passwordfile: <string> | default = ""]
# File containing password (username goes in URI)
# CLI flag: -configs.database.password-file
[password_file: <string> | default = ""]

api:
notifications:
# Disable Email notifications for Alertmanager.
# CLI flag: -configs.notifications.disable-email
[disable_email: <boolean> | default = false]

# Disable WebHook notifications for Alertmanager.
# CLI flag: -configs.notifications.disable-webhook
[disable_webhook: <boolean> | default = false]
```

### `configstore_config`
Expand Down
5 changes: 3 additions & 2 deletions pkg/alertmanager/alerts/configdb/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,16 @@ package configdb
import (
"context"

"github.com/cortexproject/cortex/pkg/configs/userconfig"

"github.com/cortexproject/cortex/pkg/alertmanager/alerts"
"github.com/cortexproject/cortex/pkg/configs"
"github.com/cortexproject/cortex/pkg/configs/client"
)

// Store is a concrete implementation of RuleStore that sources rules from the config service
type Store struct {
configClient client.Client
since configs.ID
since userconfig.ID
alertConfigs map[string]alerts.AlertConfigDesc
}

Expand Down
63 changes: 47 additions & 16 deletions pkg/configs/api/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ package api
import (
"database/sql"
"encoding/json"
"errors"
"flag"
"fmt"
"html/template"
"io/ioutil"
Expand All @@ -18,20 +20,46 @@ import (
amconfig "github.com/prometheus/alertmanager/config"
"github.com/weaveworks/common/user"

"github.com/cortexproject/cortex/pkg/configs"
"github.com/cortexproject/cortex/pkg/configs/db"
"github.com/cortexproject/cortex/pkg/configs/userconfig"
"github.com/cortexproject/cortex/pkg/util"
)

var (
ErrEmailNotificationsAreDisabled = errors.New("email notifications are disabled")
ErrWebhookNotificationsAreDisabled = errors.New("webhook notifications are disabled")
)

// Config configures Configs API
type Config struct {
Notifications NotificationsConfig `yaml:"notifications"`
}

// NotificationsConfig configures Alertmanager notifications method.
type NotificationsConfig struct {
DisableEmail bool `yaml:"disable_email"`
DisableWebHook bool `yaml:"disable_webhook"`
}

// RegisterFlags adds the flags required to configure this to the given FlagSet.
func (cfg *Config) RegisterFlags(f *flag.FlagSet) {
f.BoolVar(&cfg.Notifications.DisableEmail, "configs.notifications.disable-email", false, "Disable Email notifications for Alertmanager.")
f.BoolVar(&cfg.Notifications.DisableWebHook, "configs.notifications.disable-webhook", false, "Disable WebHook notifications for Alertmanager.")
}

// API implements the configs api.
type API struct {
db db.DB
http.Handler
db db.DB
cfg Config
}

// New creates a new API
func New(database db.DB) *API {
a := &API{db: database}
func New(database db.DB, cfg Config) *API {
a := &API{
db: database,
cfg: cfg,
}
r := mux.NewRouter()
a.RegisterRoutes(r)
a.Handler = r
Expand Down Expand Up @@ -124,7 +152,7 @@ func (a *API) setConfig(w http.ResponseWriter, r *http.Request) {
}
logger := util.WithContext(r.Context(), util.Logger)

var cfg configs.Config
var cfg userconfig.Config
switch parseConfigFormat(r.Header.Get("Content-Type"), FormatJSON) {
case FormatJSON:
if err := json.NewDecoder(r.Body).Decode(&cfg); err != nil {
Expand All @@ -147,7 +175,7 @@ func (a *API) setConfig(w http.ResponseWriter, r *http.Request) {
return
}

if err := validateAlertmanagerConfig(cfg.AlertmanagerConfig); err != nil && cfg.AlertmanagerConfig != "" {
if err := validateAlertmanagerConfig(cfg.AlertmanagerConfig, a.cfg.Notifications); err != nil && cfg.AlertmanagerConfig != "" {
level.Error(logger).Log("msg", "invalid Alertmanager config", "err", err)
http.Error(w, fmt.Sprintf("Invalid Alertmanager config: %v", err), http.StatusBadRequest)
return
Expand Down Expand Up @@ -180,7 +208,7 @@ func (a *API) validateAlertmanagerConfig(w http.ResponseWriter, r *http.Request)
return
}

if err = validateAlertmanagerConfig(string(cfg)); err != nil {
if err = validateAlertmanagerConfig(string(cfg), a.cfg.Notifications); err != nil {
w.WriteHeader(http.StatusBadRequest)
util.WriteJSONResponse(w, map[string]string{
"status": "error",
Expand All @@ -194,27 +222,30 @@ func (a *API) validateAlertmanagerConfig(w http.ResponseWriter, r *http.Request)
})
}

func validateAlertmanagerConfig(cfg string) error {
func validateAlertmanagerConfig(cfg string, noCfg NotificationsConfig) error {
amCfg, err := amconfig.Load(cfg)
if err != nil {
return err
}

for _, recv := range amCfg.Receivers {
if len(recv.EmailConfigs) != 0 {
return fmt.Errorf("email notifications are not supported in Cortex yet")
if noCfg.DisableEmail && len(recv.EmailConfigs) > 0 {
return ErrEmailNotificationsAreDisabled
}
if noCfg.DisableWebHook && len(recv.WebhookConfigs) > 0 {
return ErrWebhookNotificationsAreDisabled
}
}

return nil
}

func validateRulesFiles(c configs.Config) error {
func validateRulesFiles(c userconfig.Config) error {
_, err := c.RulesConfig.Parse()
return err
}

func validateTemplateFiles(c configs.Config) error {
func validateTemplateFiles(c userconfig.Config) error {
for fn, content := range c.TemplateFiles {
if _, err := template.New(fn).Parse(content); err != nil {
return err
Expand All @@ -224,14 +255,14 @@ func validateTemplateFiles(c configs.Config) error {
return nil
}

// ConfigsView renders multiple configurations, mapping userID to configs.View.
// ConfigsView renders multiple configurations, mapping userID to userconfig.View.
// Exposed only for tests.
type ConfigsView struct {
Configs map[string]configs.View `json:"configs"`
Configs map[string]userconfig.View `json:"configs"`
}

func (a *API) getConfigs(w http.ResponseWriter, r *http.Request) {
var cfgs map[string]configs.View
var cfgs map[string]userconfig.View
var cfgErr error
logger := util.WithContext(r.Context(), util.Logger)
rawSince := r.FormValue("since")
Expand All @@ -244,7 +275,7 @@ func (a *API) getConfigs(w http.ResponseWriter, r *http.Request) {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
cfgs, cfgErr = a.db.GetConfigs(r.Context(), configs.ID(since))
cfgs, cfgErr = a.db.GetConfigs(r.Context(), userconfig.ID(since))
}

if cfgErr != nil {
Expand Down
Loading