Make job names configurable

[jtlisi]

This PR allows the operator to configure the job names for the generated dashboards. This allows users to configure the mixin for non-default deployments of cortex and configure the mixin for use with the single binary deployment of cortex. A single binary deployment of cortex will not have unique job s for all of its modules. Instead, $namespace/cortex will be the job name in most deployments. Exposing the job constant used in the cortex dashboards will allow the user to configure and generate dashboards that work best for them.

For example:
Assuming a default micro-service deployment of cortex with the standard job names and a single binary deployment with the job name cortex. The following config could be set to allow for dashboards that accommodate both deployments.

{
  _config: {
    job_names: {
      ingester: '(ingester|cortex$)',
      distributor: ('distributor|cortex$)',
      querier: '(querier|cortex$)',
      query_frontend: '(query-frontend|cortex$)',
      table_manager: '(table-manager|cortex$)',
      store_gateway: '(store-gateway|cortex$)',
    },
  }
}

• Later this can ideally be simplified into a smaller selection of configs

[cstyan]

LGTM, have you tried this in QA or anywhere to ensure those dashboards still work?

and for your example component|cortex$, the $ is jsonnet specific syntax?

[tomwilkie]

Looks like your trying to make this work for the single binary - does my previous effort not work here?

[tomwilkie]

Looks like you’re trying to make this work for the single binary - does my previous PR not do the trick?

[jtlisi]

@cstyan I deployed it to the development cluster and it worked pretty well there. I want to deploy it again and double check.

@tomwilkie We are still running into two issues. First, is that the job names in the dashboard are still hard coded to ingester / distributor / etc. When we run a single binary internally the job name for the scrape will be cortex. Second, internally we run both microservices and single binary deployments. Which means our dashboards need to somehow accommodate both. By exposing the job names as variables we can inject a regex that works for both microservices or singleBinary. However, the config would be much simpler if we just deployed one or the other.

[tomwilkie]

Can we generate & install two version of the mixin then?

The challenge with this approach is it won't work for people not on k8s. And I'd rather avoid have 3 versions of the mixin if possible.

[jtlisi]

@tomwilkie For people not on k8s they can still set the singleBinary flag to true and that will remove label selectors for cluster and namespace and just leave the selectors for the job names. So if someone was running on bare metal they could just set that flag and update all of the job names to be cortex if they are running on a single binary or leave them as is if they are running microservices on baremetal. The singleBinary config option acts more as a enableK8sLabels config option.

cortex-jsonnet/cortex-mixin/dashboards/dashboard-utils.libsonnet

Lines 34 to 56 in 958fb89

	if $._config.singleBinary
	then d.addMultiTemplate('job', 'cortex_build_info', 'job')
	else d
	.addMultiTemplate('cluster', 'cortex_build_info', 'cluster')
	.addMultiTemplate('namespace', 'cortex_build_info', 'namespace'),
	},
	// The ,ixin allow specialism of the job selector depending on if its a single binary
	// deployment or a namespaced one.
	jobMatcher(job)::
	if $._config.singleBinary
	then 'job=~"$job"'
	else 'cluster=~"$cluster", job=~"($namespace)/%s"' % job,
	namespaceMatcher()::
	if $._config.singleBinary
	then 'job=~"$job"'
	else 'cluster=~"$cluster", namespace=~"$namespace"',
	jobSelector(job)::
	if $._config.singleBinary
	then [utils.selector.noop('cluster'), utils.selector.re('job', '$job')]
	else [utils.selector.re('cluster', '$cluster'), utils.selector.re('job', '($namespace)/%s' % job)],

[jtlisi]

LGTM, have you tried this in QA or anywhere to ensure those dashboards still work?

and for your example component|cortex$, the $ is jsonnet specific syntax?

The $ is just to ensure the regex terminates after cortex otherwise the cortex-gw would always match as well.