reduce the lock use times in ValidateLabels method

[songjiayang]

Is your feature request related to a problem? Please describe.

In default case, distributor will validate every series lables, but when validate one field it will resolve user runtime config once, will also lock one time.

func ValidateLabels(cfg LabelValidationConfig, userID string, ls []cortexpb.LabelAdapter, skipLabelNameValidation bool) ValidationError {
	if cfg.EnforceMetricName(userID) {
		unsafeMetricName, err := extract.UnsafeMetricNameFromLabelAdapters(ls)
		if err != nil {
			DiscardedSamples.WithLabelValues(missingMetricName, userID).Inc()
			return newNoMetricNameError()
		}

		if !model.IsValidMetricName(model.LabelValue(unsafeMetricName)) {
			DiscardedSamples.WithLabelValues(invalidMetricName, userID).Inc()
			return newInvalidMetricNameError(unsafeMetricName)
		}
	}

	numLabelNames := len(ls)
	if numLabelNames > cfg.MaxLabelNamesPerSeries(userID) {
		DiscardedSamples.WithLabelValues(maxLabelNamesPerSeries, userID).Inc()
		return newTooManyLabelsError(ls, cfg.MaxLabelNamesPerSeries(userID))
	}

	maxLabelNameLength := cfg.MaxLabelNameLength(userID)
	maxLabelValueLength := cfg.MaxLabelValueLength(userID)
	lastLabelName := ""
	maxLabelsSizeBytes := cfg.MaxLabelsSizeBytes(userID)
	...
}

func (om *Manager) GetConfig() interface{} {
	om.configMtx.RLock()
	defer om.configMtx.RUnlock()

	return om.config
}

When there are more series, the CPU takes more time，see the pprof:

Describe the solution you'd like

remove the lock in GetConfig or cache override limits at ValidateLabels method.

[alvinlin123]

Thanks for reporting this @songjiayang, would you be open to submit a PR for this? I don't think removing the lock in GetConfig would be a safe thing to do, but caching the limit values in ValidateLabels does seem like a viable solution.