Ecs logs integration

[felixbarny]

What does this PR do?

• Adds a ECS Logs integration aiming at simplifying the life of users who want to ingest a log file that's produced by one of our ECS logging libraries.
• Currently uses the logfile input (should use filestream later)
  • Uses defaults optimized for ECS JSON log files:
  • Enables all relevant metadata processors
  • Enables multiline settings appropriate for the stackTraceAsArray setting of ecs-logging-java
  • Excludes .gz files by default from file patterns that include a wildcard at the end to match rolled over files. Example: app.log, app.log.1, app.log.2.gz
• Logs are sent to the logs-ecs_router-default data stream
  • This data stream has a default ingest pipeline which does the following things:
    • Parsing the JSON
    • Routing the logs to a different data stream, based on the content of the log: logs-${data_stream.dataset:generic}-${data_stream.namespace:default}. So if the logs contain data_stream.dataset: foo (or event.dataset: foo), they will be sent to logs-foo-default.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

TODOs/open questions

• Use filestream instead of logfile input
• Add default multi line setting for the stackTraceAsArray option (if line does not start with {)
• Use include/exclude to include rolled over files but exclude .gz
• Should we build on top of [Filestream] New Input package #2735 so that we only apply defaults (dataset, processors, multiline, include/exclude)?
• End-to-end test with local agent

Blocked by

• [Change Proposal] Ability to add privileges for all data streams of a specific type package-spec#315

Follow ups

• Handle mapping conflicts
  • Custom fields conflict with ECS fields. Corrupts the mapping so that docs with proper ECS fields fail. Hard to recover from. Optionally use ECS conventions for dynamic mappings elasticsearch#85692
  • Custom fields conflicting with other custom fields, such as foo foo.bar (foo used both as scalar and object type). https://github.com/elastic/observability-dev/issues/1661
• Data stream router processor to avoid expensive mustache script Add reroute processor elasticsearch#76511

Author's Checklist

How to test this PR locally

Related issues

• closes [Feature Request] Enable ECS log detection by default using Custom Log integration #1332

Screenshots

[elasticmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-09-12T08:55:41.342+0000

• Duration: 10 min 23 sec

Steps errors

Expand to view the steps failures

Checks and builds Go sources

• Took 0 min 23 sec . View more details here
• Description: mage -debug check

Google Storage Download

• Took 0 min 0 sec . View more details here

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!