|
| 1 | +--- |
| 2 | +mapped_pages: |
| 3 | + - https://www.elastic.co/guide/en/ecs-logging/overview/current/intro.html |
| 4 | + - https://www.elastic.co/guide/en/ecs-logging/overview/current/index.html |
| 5 | +--- |
| 6 | + |
| 7 | +# ECS logging libraries [intro] |
| 8 | + |
| 9 | +Centralized application logging with the Elastic stack made easy. |
| 10 | + |
| 11 | + |
| 12 | + |
| 13 | + |
| 14 | +## What is ECS? [_what_is_ecs] |
| 15 | + |
| 16 | +Elastic Common Schema (ECS) defines a common set of fields for ingesting data into Elasticsearch. For more information about ECS, visit the [ECS Reference Documentation](ecs://docs/reference/index.md). |
| 17 | + |
| 18 | + |
| 19 | +## What is ECS logging? [_what_is_ecs_logging] |
| 20 | + |
| 21 | +ECS loggers are plugins for your favorite logging library. They make it easy to format your logs into ECS-compatible JSON. For example: |
| 22 | + |
| 23 | +```json |
| 24 | +{"@timestamp":"2019-08-06T12:09:12.375Z", "log.level": "INFO", "message":"Tomcat started on port(s): 8080 (http) with context path ''", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.boot.web.embedded.tomcat.TomcatWebServer"} |
| 25 | +{"@timestamp":"2019-08-06T12:09:12.379Z", "log.level": "INFO", "message":"Started PetClinicApplication in 7.095 seconds (JVM running for 9.082)", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.samples.petclinic.PetClinicApplication"} |
| 26 | +{"@timestamp":"2019-08-06T14:08:40.199Z", "log.level":"DEBUG", "message":"init find form", "service.name":"spring-petclinic","process.thread.name":"http-nio-8080-exec-8","log.logger":"org.springframework.samples.petclinic.owner.OwnerController","transaction.id":"28b7fb8d5aba51f1","trace.id":"2869b25b5469590610fea49ac04af7da"} |
| 27 | +``` |
| 28 | + |
| 29 | + |
| 30 | +## Get started [_get_started] |
| 31 | + |
| 32 | +Refer to the installation instructions of the individual loggers: |
| 33 | + |
| 34 | +* [.NET](ecs-dotnet://docs/reference/setup.md) |
| 35 | +* Go: [zap](ecs-logging-go-zap://docs/reference/setup.md), [logrus](ecs-logging-go-logrus://docs/reference/setup.md), [zerolog](ecs-logging-go-zerolog://docs/reference/setup.md) |
| 36 | +* [Java](ecs-logging-java://docs/reference/setup.md) |
| 37 | +* Node.js: [morgan](ecs-logging-nodejs://docs/reference/morgan.md), [pino](ecs-logging-nodejs://docs/reference/pino.md), [winston](ecs-logging-nodejs://docs/reference/winston.md) |
| 38 | +* [PHP](ecs-logging-php://docs/reference/setup.md) |
| 39 | +* [Python](ecs-logging-python://docs/reference/installation.md) |
| 40 | +* [Ruby](ecs-logging-ruby://docs/reference/setup.md) |
| 41 | + |
| 42 | + |
| 43 | +## Why ECS logging? [_why_ecs_logging] |
| 44 | + |
| 45 | +**Simplicity: no manual parsing** |
| 46 | +: Logs arrive pre-formatted, pre-enriched and ready to add value, making problems quicker and easier to identify. No more tedious grok parsing that has to be customized for every application. |
| 47 | + |
| 48 | + |
| 49 | +**Decently human-readable JSON structure** |
| 50 | +: The first three fields are `@timestamp`, `log.level` and `message`. This lets you easily read the logs in a terminal without needing a tool that converts the logs to plain-text. |
| 51 | + |
| 52 | + |
| 53 | +**Enjoy the benefits of a common schema** |
| 54 | +: Use the Kibana [Logs app](docs-content://solutions/observability/logs/explore-logs.md) without additional configuration. |
| 55 | + |
| 56 | +Using a common schema across different services and teams makes it possible create reusable dashboards and avoids [mapping explosions](docs-content://manage-data/data-store/mapping.md#mapping-limit-settings). |
| 57 | + |
| 58 | + |
| 59 | +**APM Log correlation** |
| 60 | +: If you are using an [Elastic APM agent](docs-content://reference/ingestion-tools/apm/apm-agents.md), you can leverage the log correlation feature without any additional configuration. This lets you jump from the [Span timeline in the APM UI](docs-content://solutions/observability/apps/trace-sample-timeline.md) to the [Logs app](docs-content://solutions/observability/logs/explore-logs.md), showing only the logs which belong to the corresponding request. Vice versa, you can also jump from a log line in the Logs UI to the Span Timeline of the APM UI. For more information about the log correlation feature, refer to [](docs-content://solutions/observability/logs/stream-application-logs.md). |
| 61 | + |
| 62 | +### Additional advantages when using in combination with Filebeat [_additional_advantages_when_using_in_combination_with_filebeat] |
| 63 | + |
| 64 | +We recommend shipping the logs with Filebeat. Depending on the way the application is deployed, you may log to a log file or to stdout (for example in Kubernetes). |
| 65 | + |
| 66 | +Here are a few benefits to this over directly sending logs from the application to Elasticsearch: |
| 67 | + |
| 68 | +**Resilient in case of outages** |
| 69 | +: [Guaranteed at-least-once delivery](beats://docs/reference/filebeat/how-filebeat-works.md#at-least-once-delivery) without buffering within the application, thus no risk of out of memory errors or lost events. There’s also the option to use either the JSON logs or plain-text logs as a fallback. |
| 70 | + |
| 71 | + |
| 72 | +**Loose coupling** |
| 73 | +: The application does not need to know the details of the logging backend (URI, credentials, etc.). You can also leverage alternative [Filebeat outputs](beats://docs/reference/filebeat/configuring-output.md), like Logstash, Kafka or Redis. |
| 74 | + |
| 75 | + |
| 76 | +**Index Lifecycle management** |
| 77 | +: Leverage Filebeat’s default [index lifecycle management settings](beats://docs/reference/filebeat/ilm.md). This is much more efficient than using daily indices. |
| 78 | + |
| 79 | + |
| 80 | +**Efficient Elasticsearch mappings** |
| 81 | +: Leverage Filebeat’s default ECS-compatible [index template](beats://docs/reference/filebeat/configuration-template.md). |
| 82 | + |
| 83 | + |
| 84 | + |
| 85 | +## Field mapping [_field_mapping] |
| 86 | + |
| 87 | + |
| 88 | +### Default fields [_default_fields] |
| 89 | + |
| 90 | +These fields are populated by the ECS loggers by default. Some of them, such as the `log.origin.*` fields, may have to be explicitly enabled. Others, such as `process.thread.name`, are not applicable to all languages. Refer to the documentation of the individual loggers for more information. |
| 91 | + |
| 92 | +| ECS field | Description | Example | |
| 93 | +| --- | --- | --- | |
| 94 | +| [`@timestamp`](ecs://docs/reference/ecs-base.md) | The timestamp of the log event. | `"2019-08-06T12:09:12.375Z"` | |
| 95 | +| [`log.level`](ecs://docs/reference/ecs-log.md) | The level or severity of the log event. | `"INFO"` | |
| 96 | +| [`log.logger`](ecs://docs/reference/ecs-log.md) | The name of the logger inside an application. | `"org.example.MyClass"` | |
| 97 | +| [`log.origin.file.name`](ecs://docs/reference/ecs-log.md) | The name of the file containing the source code which originated the log event. | `"App.java"` | |
| 98 | +| [`log.origin.file.line`](ecs://docs/reference/ecs-log.md) | The line number of the file containing the source code which originated the log event. | `42` | |
| 99 | +| [`log.origin.function`](ecs://docs/reference/ecs-log.md) | The name of the function or method which originated the log event. | `"methodName"` | |
| 100 | +| [`message`](ecs://docs/reference/ecs-base.md) | The log message. | `"Hello World!"` | |
| 101 | +| [`error.type`](ecs://docs/reference/ecs-error.md) | Only present for logs that contain an exception or error. The type or class of the error if this log event contains an exception. | `"java.lang.NullPointerException"` | |
| 102 | +| [`error.message`](ecs://docs/reference/ecs-error.md) | Only present for logs that contain an exception or error. The message of the exception or error. | `"The argument cannot be null"` | |
| 103 | +| [`error.stack_trace`](ecs://docs/reference/ecs-error.md) | Only present for logs that contain an exception or error. The full stack trace of the exception or error as a raw string. | `"Exception in thread "main" java.lang.NullPointerException\n\tat org.example.App.methodName(App.java:42)"` | |
| 104 | +| [`process.thread.name`](ecs://docs/reference/ecs-process.md) | The name of the thread the event has been logged from. | `"main"` | |
| 105 | + |
| 106 | + |
| 107 | +### Configurable fields [_configurable_fields] |
| 108 | + |
| 109 | +Refer to the documentation of the individual loggers on how to set these fields. |
| 110 | + |
| 111 | +| ECS field | Description | Example | |
| 112 | +| --- | --- | --- | |
| 113 | +| [`service.name`](ecs://docs/reference/ecs-service.md) | Helps to filter the logs by service. | `"my-service"` | |
| 114 | +| [`service.version`](ecs://docs/reference/ecs-service.md) | Helps to filter the logs by service version. | `"1.0"` | |
| 115 | +| [`service.environment`](ecs://docs/reference/ecs-service.md) | Helps to filter the logs by environment. | `"production"` | |
| 116 | +| [`service.node.name`](ecs://docs/reference/ecs-service.md) | Allow for two nodes of the same service, on the same host to be differentiated. | `"instance-0000000016"` | |
| 117 | +| [`event.dataset`](ecs://docs/reference/ecs-event.md) | Enables the [log rate anomaly detection](docs-content://solutions/observability/logs/inspect-log-anomalies.md). | `"my-service"` | |
| 118 | + |
| 119 | + |
| 120 | +### Custom fields [_custom_fields] |
| 121 | + |
| 122 | +Most loggers allow you to add additional custom fields. This includes both, static and dynamic ones. Examples for dynamic fields are logging structured objects, or fields from a thread local context, such as `MDC` or `ThreadContext`. |
| 123 | + |
| 124 | +When adding custom fields, we recommend using existing [ECS fields](ecs://docs/reference/ecs-field-reference.md) for these custom values. If there is no appropriate ECS field, consider prefixing your fields with `labels.`, as in `labels.foo`, for simple key/value pairs. For nested structures, consider prefixing with `custom.`. This approach protects against conflicts in case ECS later adds the same fields but with a different mapping. |
| 125 | + |
0 commit comments