{m365_defender,microsoft_defender_endpoint}: Add mapping and transform for CDR workflows #14809

kcreddy · 2025-08-05T07:10:28Z

Proposed commit message

Add support for CDR Cloud Native Vulnerability Management (CNVM)[1] workflow by 
adding necessary mappings and latest transform.

[1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Run pipeline tests:
m365_defender:

--- Test results for package: m365_defender - START ---
╭───────────────┬───────────────┬───────────┬───────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM   │ TEST TYPE │ TEST NAME                                         │ RESULT │ TIME ELAPSED │
├───────────────┼───────────────┼───────────┼───────────────────────────────────────────────────┼────────┼──────────────┤
│ m365_defender │ vulnerability │ pipeline  │ (ingest pipeline warnings test-vulnerability.log) │ PASS   │    477.276ms │
│ m365_defender │ vulnerability │ pipeline  │ test-vulnerability.log                            │ PASS   │  104.08175ms │
╰───────────────┴───────────────┴───────────┴───────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: m365_defender - END   ---
Done

microsoft_defender_endpoint:

--- Test results for package: microsoft_defender_endpoint - START ---
╭─────────────────────────────┬───────────────┬───────────┬───────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE                     │ DATA STREAM   │ TEST TYPE │ TEST NAME                                         │ RESULT │ TIME ELAPSED │
├─────────────────────────────┼───────────────┼───────────┼───────────────────────────────────────────────────┼────────┼──────────────┤
│ microsoft_defender_endpoint │ vulnerability │ pipeline  │ (ingest pipeline warnings test-vulnerability.log) │ PASS   │ 434.634375ms │
│ microsoft_defender_endpoint │ vulnerability │ pipeline  │ test-vulnerability.log                            │ PASS   │  96.629375ms │
╰─────────────────────────────┴───────────────┴───────────┴───────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: microsoft_defender_endpoint - END   ---
Done

Run system tests:
m365_defender:

--- Test results for package: m365_defender - START ---
╭───────────────┬───────────────┬───────────┬───────────┬────────┬─────────────────╮
│ PACKAGE       │ DATA STREAM   │ TEST TYPE │ TEST NAME │ RESULT │    TIME ELAPSED │
├───────────────┼───────────────┼───────────┼───────────┼────────┼─────────────────┤
│ m365_defender │ vulnerability │ system    │ default   │ PASS   │ 2m13.608947917s │
╰───────────────┴───────────────┴───────────┴───────────┴────────┴─────────────────╯
--- Test results for package: m365_defender - END   ---
Done

microsoft_defender_endpoint:

--- Test results for package: microsoft_defender_endpoint - START ---
╭─────────────────────────────┬───────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE                     │ DATA STREAM   │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────────────────────────┼───────────────┼───────────┼───────────┼────────┼───────────────┤
│ microsoft_defender_endpoint │ vulnerability │ system    │ default   │ PASS   │ 39.797611416s │
╰─────────────────────────────┴───────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: microsoft_defender_endpoint - END   ---
Done

Related issues

Screenshots

elastic-vault-github-plugin-prod · 2025-08-05T07:37:28Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

…nt-cdr

elasticmachine · 2025-08-05T14:27:38Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kcreddy · 2025-08-05T14:38:03Z

...s/microsoft_defender_endpoint/elasticsearch/transform/latest_cdr_vuln/fields/base-fields.yml

@maxcold, for microsoft_defender_endpoint integration, I had to shorten the transform name as latest_cdr_vuln because if not, the system tests would throw an error during package installation:

Error: failed to setup system runner: can't install the package: there was an apply error: installation failed: can't install the package: could not zip-install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"action_request_validation_exception\n\tRoot causes:\n\t\taction_request_validation_exception: Validation Failed: 1: The id cannot contain more than 64 characters.;"}

Its coming from Kibana (logs below):

2025-08-05 19:18:07 [2025-08-05T13:48:07.229+00:00][WARN ][plugins.fleet] Error during execution of state "install_transforms" with status "failed": action_request_validation_exception 2025-08-05 19:18:07 Root causes: 2025-08-05 19:18:07 action_request_validation_exception: Validation Failed: 1: The id cannot contain more than 64 characters.; 2025-08-05 19:18:07 [2025-08-05T13:48:07.246+00:00][DEBUG][plugins.fleet] Created transform: logs-microsoft_defender_endpoint.latest_action-default-1.0.0 2025-08-05 19:18:07 [2025-08-05T13:48:07.311+00:00][DEBUG][plugins.fleet] Started transform: logs-microsoft_defender_endpoint.latest_action-default-1.0.0 2025-08-05 19:18:08 [2025-08-05T13:48:08.138+00:00][WARN ][plugins.fleet] Failure to install package [microsoft_defender_endpoint]: [ResponseError: action_request_validation_exception 2025-08-05 19:18:08 Root causes: 2025-08-05 19:18:08 action_request_validation_exception: Validation Failed: 1: The id cannot contain more than 64 characters.;] 2025-08-05 19:18:09 [2025-08-05T13:48:09.163+00:00][ERROR][plugins.fleet] Uninstalling microsoft_defender_endpoint-3.0.0 after error installing: [ResponseError: action_request_validation_exception 2025-08-05 19:18:09 Root causes: 2025-08-05 19:18:09 action_request_validation_exception: Validation Failed: 1: The id cannot contain more than 64 characters.;] with install type: install

Hope the shortened name is okay.

kcreddy · 2025-08-05T16:08:37Z

packages/m365_defender/manifest.yml

@@ -11,7 +11,7 @@ conditions:
  elastic:
    subscription: basic
  kibana:
-    version: "^8.18.0 || ^9.0.0"
+    version: "^8.19.0 || ^9.1.0"


Currently both integrations are not added to permissions list in Elasticsearch.
I created the PR: elastic/elasticsearch#132445 to add them. Once approved and backported, I will update these versions accordingly. Until then the system tests will fail because of permissions error inside transform.

cc: @maxcold

I added run_as_kibana_system: false for now to test. Will send you the credentials for testing.

elastic-sonarqube · 2025-08-05T16:47:19Z

Quality Gate failed

Failed conditions
26.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

chrisberkhout

Looks good. I made a few comments in microsoft_defender_endpoint, but they apply to both.

chrisberkhout · 2025-08-08T09:04:20Z

...rosoft_defender_endpoint/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

      field: event.id
      tag: set_event_id_from_microsoft_defender_endpoint_vulnerability_affected_machine_id
-      value: '{{{microsoft_defender_endpoint.vulnerability.affected_machine.id}}}-{{{_ingest.timestamp}}}'
-      if: ctx.microsoft_defender_endpoint?.vulnerability?.affected_machine?.id != null
+      copy_from: microsoft_defender_endpoint.vulnerability.affected_machine.id
+      ignore_empty_value: true


Seems odd to set the vulnerability event ID to the affected machine ID.
I see it's set to a vulnerability ID below so I guess this is just the fallback.

Actually, this field indicates a finding id of sorts, example: 86c0491db8ff7e8dcad520288b7759fa27793ce1-_-CVE-2024-11168-_-red_hat-_-python-unversioned-command_for_linux-_-0:3.9.18-3.el9_4.6-_-.
It includes machineId, vulnerabilityId, packageName, packageVersion, etc.
https://github.com/elastic/integrations/blob/main/packages/m365_defender/data_stream/vulnerability/_dev/test/pipeline/test-vulnerability.log-expected.json#L60.

chrisberkhout · 2025-08-08T09:15:55Z

...rosoft_defender_endpoint/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

+      tag: grok_to_extract_vulnerability_score_version
+      patterns:
+        - '^CVSS:%{DATA:vulnerability.score.version}/%{GREEDYDATA}$'
+        - '^%{GREEDYDATA}$'


With the 2nd pattern, won't it match and ignore any string that didn't match the first?

If that's intentional, maybe it can get if: ctx.microsoft_defender_endpoint?.vulnerability?.cvss_vector instanceof String and avoid the on_failure.

Similar comment, as for m365_defender. Also time complexity of the first pattern can be reduced.

chrisberkhout · 2025-08-08T09:18:31Z

...rosoft_defender_endpoint/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

+  - set:
+      field: vulnerability.cve
+      tag: set_vulnerability_cve_from_vulnerability_id
+      copy_from: microsoft_defender_endpoint.vulnerability.id
+      ignore_empty_value: true
+      if: ctx.vulnerability?.id != null && ctx.vulnerability.id.toUpperCase().contains('CVE') == true
  - set:
      field: event.id
      tag: set_event_id_from_vulnerability_id
-      value: '{{{vulnerability.id}}}-{{{_ingest.timestamp}}}'
-      if: ctx.event?.id == null && ctx.vulnerability?.id != null
+      copy_from: microsoft_defender_endpoint.vulnerability.id
+      ignore_empty_value: true
+      if: ctx.event?.id == null


If the vulnerability ID is a CVE, I guess we may get many events with the same vulnerability ID.

What's the intention with removing the timestamp from event.id?

This is clarified with Cloud team, whether we really want uniqueness (on source indices) for this field, and it seems that we don't. Essentially the idea for this field is that we don't need to append timestamp as the same finding should have the same id over time.

chrisberkhout · 2025-08-08T09:24:16Z

...s/microsoft_defender_endpoint/elasticsearch/transform/latest_cdr_vuln/fields/base-fields.yml

The field mapping files in the transform that match what's in the data stream can removed by using the new elastic-package links functionality.

In case a file currently has both common and unique content, it'd be good to split that file.

The full links documentation: https://github.com/elastic/elastic-package/blob/main/docs/howto/links.md

I tested this today and this links feature doesn't seem to be working as expected for me.

Here is the commit that I tested against.

When I build the integration using elastic-package build and check the build folder, the linked files are actually generated inside package root directory.

I checked the mappings in Kibana as well. All the linked mappings are missing. Only explicitly defined mapping exists.

Also, the transform produced an error which also indicates the mapping doesn't work.

…nt-cdr

packages/m365_defender/_dev/build/docs/README.md

packages/m365_defender/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

packages/m365_defender/data_stream/vulnerability/fields/base-fields.yml

packages/m365_defender/data_stream/vulnerability/fields/ecs.yml

packages/m365_defender/elasticsearch/transform/latest_cdr_vulnerabilities/fields/ecs.yml

packages/microsoft_defender_endpoint/data_stream/vulnerability/fields/ecs.yml

packages/microsoft_defender_endpoint/elasticsearch/transform/latest_cdr_vuln/fields/ecs.yml

packages/microsoft_defender_endpoint/elasticsearch/transform/latest_cdr_vuln/fields/package.yml

...ages/microsoft_defender_endpoint/elasticsearch/transform/latest_cdr_vuln/fields/resource.yml

...microsoft_defender_endpoint/elasticsearch/transform/latest_cdr_vuln/fields/vulnerability.yml

…nt-cdr

elasticmachine · 2025-08-12T06:41:58Z

💔 Build Failed

Buildkite Build
Commit: 2f27420

Failed CI Steps

History

💔 Build #29916 failed 90bf705
💔 Build #29896 failed e340286
💔 Build #29881 failed 4473e4f
💔 Build #29632 failed eb9d41e

cc @kcreddy

efd6 · 2025-08-12T06:53:54Z

CI seems to be having problems with docker; I've been seeing problems in beats too.

kcreddy · 2025-08-12T07:04:40Z

CI seems to be having problems with docker; I've been seeing problems in beats too.

Actually I thought it was because images for 8.19.2 are not available yet (maybe soon).
I need 8.19.2 for supporting permissions added to Elasticsearch recently: elastic/elasticsearch#132629

kcreddy added 2 commits August 5, 2025 11:41

Add missing CDR mapping for m365_defender

99b4b49

Add missing CDR mapping for microsoft_defender_endpoint

9067166

Add more cloud ECS fields and changelog

4d117b6

andrewkroh added Integration:m365_defender Microsoft Defender XDR Integration:microsoft_defender_endpoint Microsoft Defender for Endpoint documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Aug 5, 2025

kcreddy mentioned this pull request Aug 5, 2025

Add {m365_defender,microsoft_defender_endpoint}.vulnerability indices to kibana_system role permissions elastic/elasticsearch#132445

Merged

kcreddy added 5 commits August 5, 2025 18:48

Add transform into m365_defender

ff15d2f

Add transform into microsoft_defender_endpoint

729c82e

Merge remote-tracking branch 'upstream/main' into defender_xdr_endpoi…

d5f886d

…nt-cdr

update readme

3c4a85e

Run system tests

51d2e3c

kcreddy marked this pull request as ready for review August 5, 2025 14:27

kcreddy requested a review from a team as a code owner August 5, 2025 14:27

kcreddy added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Aug 5, 2025

kcreddy self-assigned this Aug 5, 2025

kcreddy commented Aug 5, 2025

View reviewed changes

update min kibana stack versions to match permissions PR

50b5fd1

kcreddy requested a review from a team August 5, 2025 14:44

kcreddy added the Category: CDR label Aug 5, 2025

revert stack version

4c60986

kcreddy commented Aug 5, 2025

View reviewed changes

Fix retention intervals

eb9d41e

andrewkroh added the breaking change label Aug 7, 2025

chrisberkhout reviewed Aug 8, 2025

View reviewed changes

kcreddy added 2 commits August 11, 2025 16:49

Update kibana version constraints

4473e4f

Merge remote-tracking branch 'upstream/main' into defender_xdr_endpoi…

aee7a7f

…nt-cdr

run pipeline tests

e340286

efd6 reviewed Aug 12, 2025

View reviewed changes

kcreddy added 3 commits August 12, 2025 11:37

Address PR comments

e608bbb

Merge remote-tracking branch 'upstream/main' into defender_xdr_endpoi…

90bf705

…nt-cdr

Add new lines inside transform fields

2f27420

{m365_defender,microsoft_defender_endpoint}: Add mapping and transform for CDR workflows #14809

Are you sure you want to change the base?

{m365_defender,microsoft_defender_endpoint}: Add mapping and transform for CDR workflows #14809

Uh oh!

Conversation

kcreddy commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elastic-vault-github-plugin-prod bot commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

elasticmachine commented Aug 5, 2025

Uh oh!

kcreddy Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Aug 5, 2025

Quality Gate failed

Uh oh!

chrisberkhout left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

efd6 commented Aug 12, 2025

Uh oh!

kcreddy commented Aug 12, 2025

Uh oh!

Uh oh!

kcreddy commented Aug 5, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Aug 5, 2025 •

edited

Loading

kcreddy Aug 5, 2025 •

edited

Loading

kcreddy Aug 5, 2025 •

edited

Loading

elasticmachine commented Aug 12, 2025 •

edited

Loading