Elastic Stack Docker Compose

Docker Compose configuration to run the Elastic Stack components.

This is meant to be a development and demo environment to solve these problems:

• Work on development of OpenTelemetry-related features in Elastic Observability
• Run pre-release builds of components to preview upcoming features
• Get an Elastic environment that can collect OpenTelemetry data up and running quickly

Runs:

• Elasticsearch
• The Elastic Distribution of OpenTelemetry Collector
• Kibana

Requirements

• Docker Compose

Usage

• Clone this repository
• cd elastic-stack-docker-compose
• docker compose up
• Open https://localhost:5601
• Log in with elastic/changeme
• Adjust configuration in the files in this repository for your needs
• If you have improvements or fixes, open a pull request to https://github.com/smith/elastic-stack-docker-compose

OpenTelemetry Demo

To send data from the OpenTelemetry Demo to this cluster, check out a copy of the demo, and update src/otel-collector/otelcol-config-extras.yml:

exporters:
  otlp:
    endpoint: "http://host.docker.internal:4317"

service:
  pipelines:
    metrics:
      exporters: [otlp, debug]
    logs:
      exporters: [otlp, debug]

docker compose up to start the demo. It will send all data from the demo's collector over OTLP to ours.

Elastic Universal Profiling

Universal Profiling services are not started by default. Use the profiles to start them:

docker compose --profile profiling up

Kibana TLS

Kibana is configured for HTTP2 by default. The HTTPS connection will not be trusted unless you configure your operating system to trust the certificate.

To get a copy out of the certificate from a running environment run:

docker compose run setup_certs cat config/certs/elasticsearch/elasticsearch.crt > cert

On MacOS you can add this certificate to the trusted store with:

sudo security add-trusted-cert -d -r trustAsRoot -p ssl -k /Library/Keychains/System.keychain cert

You'll need to do this again if the volume for the certificates gets recreated.

Docker Compose metrics

To collect Docker Compose metrics, set the environment variable DOCKER_CLI_OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317. These metrics go to the metrics-generic-default data stream with service.name=docker and service.name=docker-compose.

Selecting which services to run

If you don't need a particular service (let's say you already have Kibana running in dev mode), you can add scale: 0 to the service definition to prevent it from being started when docker compose up is run. You can also run docker compose stop SERVICE_NAME to stop an individual service.

Elastic APM Server

Elastic APM Server is included for testing with legacy scenarios.

To start the APM server run:

docker compose --profile apm up

To configure the OpenTelemetry collector to send data to the APM Server, uncomment the traces/fromsdk pipeline under the # Send traces to APM server comment.

Updating image

In .env the default image variables look like this:

KIBANA_IMAGE=docker.elastic.co/kibana/kibana:9.0.0-SNAPSHOT

When you first start, it will download the latest snapshot image. to update to the latest image, for, for example, Kibana, run:

docker compose stop kibana
docker compose rm -f kibana
docker compose pull kibana

You can omit kibana to update all images.

Data collected

• Any OpenTelemetry log or metric data sent to localhost:4317-4318 (use host.docker.internal from containers.)
• HTTP checks for Elasticsearch and Kibana. These metrics go to the metrics-generic-default data stream with http and httpcheck fields.
• Host metrics. These are processed with the Elastic Infra Metrics Processor
• OpenTelemetry collector internal logs and metrics
• Logs from Elasticsearch and Kibana

Upstream docker-compose.yml configurations

This compose.yaml is based on these sources:

• Elasticsearch
• OpenTelemetry Demo

Why doesn't this use Kubernetes?

If you need Kubernetes, you should use it. See the Elastic Kubernetes documentation. This is a simpler environment meant to run on a single computer.