{microsoft_defender_endpoint, m365_defender}.vulnerability: New API implementation #15603

kcreddy · 2025-10-08T10:57:19Z

Proposed commit message

{microsoft_defender_endpoint, m365_defender}.vulnerability: New API implementation

Existing CEL program uses 3 API endpoints to fetch vulnerability
data. Although we fetch more fields using this approach, it doesn't 
scale well and hence is unusable even for few hundred machines.

This PR updates the vulnerability data stream with new 
SoftwareVulnerabilitiesExport API[1], which is recommended for 
larger workloads. While there are few data points missed in this 
new implementation[2], we maintain all the required fields for 
3rd party vulnerability workflow[3].

Other changes:
- Updates microsoft_defender_endpoint min stack version to "8.19.3"
  as the permissions for the transform were actually applied in 
  "8.19.3" version, and not in "8.19.2"[4].
- Add dataset filter to all visualisations of vulnerability dashboards.

[1]: https://learn.microsoft.com/en-us/defender-endpoint/api/get-assessment-software-vulnerabilities#2-export-software-vulnerabilities-assessment-via-files
[2]: https://github.com/elastic/integrations/issues/15521#issuecomment-3380969284
[3]: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide
[4]: https://github.com/elastic/elasticsearch/pull/132629

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Verified there are no prebuilt rules effected by this change.
Update dashboards to remove references to old fields.
Add video for upgrade flow.

How to test this PR locally

Pipeline tests and System tests pass.

--- Test results for package: m365_defender - START ---
╭───────────────┬───────────────┬───────────┬───────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM   │ TEST TYPE │ TEST NAME                                         │ RESULT │ TIME ELAPSED │
├───────────────┼───────────────┼───────────┼───────────────────────────────────────────────────┼────────┼──────────────┤
│ m365_defender │ vulnerability │ pipeline  │ (ingest pipeline warnings test-vulnerability.log) │ PASS   │ 480.561041ms │
│ m365_defender │ vulnerability │ pipeline  │ test-vulnerability.log                            │ PASS   │  87.309125ms │
╰───────────────┴───────────────┴───────────┴───────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: m365_defender - END   ---
Done

Related issues

Closes {microsoft_defender_endpoint, m365_defender}: vulnerability data stream scaling problem #15521

Screenshots

Updated dashboards:

microsoft_defender_endpoint-vulnerability_overview

Upgrade Flow

mde-vulnerability-3.1-to-3.2-upgrade-flow.mp4

elasticmachine · 2025-10-09T20:00:31Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

...s/m365_defender/kibana/visualization/m365_defender-4f3a6702-9642-4392-9b34-ceb1447e09a7.json

packages/m365_defender/changelog.yml

packages/microsoft_defender_endpoint/changelog.yml

packages/m365_defender/data_stream/vulnerability/agent/stream/cel.yml.hbs

packages/microsoft_defender_endpoint/_dev/deploy/docker/vulnerability-http-mock-config.yml

...endpoint/kibana/search/microsoft_defender_endpoint-89e2c263-f9c2-4f34-85d2-86b1d1b9106b.json

...t/kibana/visualization/microsoft_defender_endpoint-4f3a6702-9642-4392-9b34-ceb1447e09a7.json

packages/m365_defender/data_stream/vulnerability/agent/stream/cel.yml.hbs

packages/microsoft_defender_endpoint/data_stream/vulnerability/agent/stream/cel.yml.hbs

efd6 · 2025-10-10T06:30:13Z

packages/m365_defender/changelog.yml

+    - description: |
+        Fetch vulnerability data using SoftwareVulnerabilitiesExport API endpoint.
+        Following fields are no longer available in new implementation: "cloud.provider", "cloud.resource_id", 
+        "cloud.instance.id", "host.geo", "host.ip", "host.risk.calculated_level", "related.ip", 
+        "vulnerability.description", "vulnerability.published_date", "vulnerability.score.version".
+        If there are custom dashboards using these fields, they could be broken.
+      type: breaking-change
+      link: https://github.com/elastic/integrations/pull/15603


Suggested change

- description: |

Fetch vulnerability data using SoftwareVulnerabilitiesExport API endpoint.

Following fields are no longer available in new implementation: "cloud.provider", "cloud.resource_id",

"cloud.instance.id", "host.geo", "host.ip", "host.risk.calculated_level", "related.ip",

"vulnerability.description", "vulnerability.published_date", "vulnerability.score.version".

If there are custom dashboards using these fields, they could be broken.

type: breaking-change

link: https://github.com/elastic/integrations/pull/15603

- description: |

Fetch vulnerability data using SoftwareVulnerabilitiesExport API endpoint.

type: enhancement

link: https://github.com/elastic/integrations/pull/15603

- description: |

The following fields are no longer available in the new implementation: "cloud.provider", "cloud.resource_id",

"cloud.instance.id", "host.geo", "host.ip", "host.risk.calculated_level", "related.ip",

"vulnerability.description", "vulnerability.published_date", "vulnerability.score.version".

type: breaking-change

link: https://github.com/elastic/integrations/pull/15603

Updated in 2f1492e

efd6 · 2025-10-10T06:33:01Z

packages/microsoft_defender_endpoint/changelog.yml

+    - description: |
+        Fetch vulnerability data using SoftwareVulnerabilitiesExport API endpoint.
+        Following fields are no longer available in new implementation: "cloud.provider", "cloud.resource_id", 
+        "cloud.instance.id", "host.geo", "host.ip", "host.risk.calculated_level", "related.ip", 
+        "vulnerability.description", "vulnerability.published_date", "vulnerability.score.version".
+        If there are custom dashboards using these fields, they could be broken.
+      type: breaking-change
+      link: https://github.com/elastic/integrations/pull/15603


Suggested change

- description: |

Fetch vulnerability data using SoftwareVulnerabilitiesExport API endpoint.

Following fields are no longer available in new implementation: "cloud.provider", "cloud.resource_id",

"cloud.instance.id", "host.geo", "host.ip", "host.risk.calculated_level", "related.ip",

"vulnerability.description", "vulnerability.published_date", "vulnerability.score.version".

If there are custom dashboards using these fields, they could be broken.

type: breaking-change

link: https://github.com/elastic/integrations/pull/15603

- description: |

Fetch vulnerability data using SoftwareVulnerabilitiesExport API endpoint.

type: enhancement

link: https://github.com/elastic/integrations/pull/15603

- description: |

The following fields are no longer available in the new implementation: "cloud.provider", "cloud.resource_id",

"cloud.instance.id", "host.geo", "host.ip", "host.risk.calculated_level", "related.ip",

"vulnerability.description", "vulnerability.published_date", "vulnerability.score.version".

type: breaking-change

link: https://github.com/elastic/integrations/pull/15603

Updated in 2f1492e

efd6 · 2025-10-10T06:37:40Z

packages/m365_defender/data_stream/vulnerability/_dev/test/system/test-default-config.yml

 data_stream:
  vars:
-    batch_size: 2
+    sas_valid_hours: 2


Suggested change

sas_valid_hours: 2

sas_valid_hours: 2h

efd6 · 2025-10-10T06:37:53Z

...crosoft_defender_endpoint/data_stream/vulnerability/_dev/test/system/test-default-config.yml

 data_stream:
  vars:
-    batch_size: 2
+    sas_valid_hours: 2


Suggested change

sas_valid_hours: 2

sas_valid_hours: 2h

Thanks. Updated in 2f1492e

elastic-vault-github-plugin-prod · 2025-10-10T08:05:54Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2025-10-10T08:42:08Z

💚 Build Succeeded

Buildkite Build
Commit: 55b57a1

History

💔 Build #32583 failed 2f1492e
💔 Build #32578 failed c05ba2b
💚 Build #32564 succeeded f131c3e
💔 Build #32562 failed fe988d9
💔 Build #32561 failed 95dc8c8

cc @kcreddy

brijesh-elastic · 2025-10-11T06:40:39Z

packages/m365_defender/data_stream/vulnerability/agent/stream/cel.yml.hbs

-                    string(vulnerabilityResp.Body)
+                "code": string(resp.StatusCode),
+                "id": string(resp.Status),
+                "message": "GET "+ state.url.trim_right("/") + ":" + (


Suggested change

"message": "GET "+ state.url.trim_right("/") + ":" + (

"message": "GET /api/machines/SoftwareVulnerabilitiesExport :" + (

brijesh-elastic · 2025-10-11T06:44:08Z

Since there is a change in the transform schema, the fleet_transform_version and destination indices need to be updated.

kcreddy added 3 commits October 6, 2025 16:58

Move to new API

55c2a54

Update pipeline

9dc9320

Update fields

c52b766

kcreddy mentioned this pull request Oct 8, 2025

{microsoft_defender_endpoint, m365_defender}: vulnerability data stream scaling problem #15521

Open

kcreddy added 2 commits October 8, 2025 17:27

update dashboards

da3e73a

update navigation

6ca2bee

andrewkroh added Integration:microsoft_defender_endpoint Microsoft Defender for Endpoint dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Oct 8, 2025

kcreddy added 5 commits October 9, 2025 18:06

update dashboard

1ef94a7

update dashboard screenshot

95dc8c8

update system tests

fe988d9

update mde readme and remove comments

c4e5a03

Update Defender XDR

f131c3e

kcreddy marked this pull request as ready for review October 9, 2025 19:24

kcreddy requested a review from a team as a code owner October 9, 2025 19:24

kcreddy added the Integration:m365_defender Microsoft Defender XDR label Oct 9, 2025

kcreddy self-assigned this Oct 9, 2025

andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Oct 9, 2025

efd6 reviewed Oct 9, 2025

View reviewed changes

Address PR comments

c05ba2b

kcreddy requested a review from efd6 October 10, 2025 05:48

efd6 reviewed Oct 10, 2025

View reviewed changes

Address comments

2f1492e

kcreddy requested a review from efd6 October 10, 2025 07:40

kcreddy added the breaking change label Oct 10, 2025

kcreddy changed the title ~~{microsoft_defender_endpoint, m365_defender}.vulnerability: Update implementation~~ {microsoft_defender_endpoint, m365_defender}.vulnerability: New API implementation Oct 10, 2025

Fix test

55b57a1

brijesh-elastic reviewed Oct 11, 2025

View reviewed changes

	"message": "GET "+ state.url.trim_right("/") + ":" + (
	"message": "GET /api/machines/SoftwareVulnerabilitiesExport :" + (

{microsoft_defender_endpoint, m365_defender}.vulnerability: New API implementation #15603

Are you sure you want to change the base?

{microsoft_defender_endpoint, m365_defender}.vulnerability: New API implementation #15603

Uh oh!

Conversation

kcreddy commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Upgrade Flow

Uh oh!

elasticmachine commented Oct 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

elasticmachine commented Oct 10, 2025

💚 Build Succeeded

History

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brijesh-elastic commented Oct 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kcreddy commented Oct 8, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Oct 10, 2025 •

edited

Loading