[AI-5055] DDS: OpenVPN Integration V1.0.0 #19811

ankitarajput-crest · 2025-03-12T11:53:31Z

What does this PR do?

PR for a new integration OpenVPN 1.0.0

Additional Notes

-- OOTB detection rules JSON would be shared separately with the required teams as a part of separate repository .
-- Since during the standard attribute remapping we are not preserving the source attributes as per suggested best practices, it would result in filters using these standard attributes populating the values of other integrations as well as per current datadog behavior.

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
Changelog entries must be created for modifications to shipped code
Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
If you need to backport this PR to another branch, you can add the backport/ label to the PR and it will automatically open a backport PR once this one is merged

Review from iliakur is dismissed. Related teams and files:

agent-integrations
- openvpn/assets/dashboards/openvpn_overview.json

gunterd · 2025-04-15T19:28:11Z

openvpn/assets/logs/openvpn.yaml

+              rule3 %{regex("(.*?)(?=:)")}:(\s+)?\'%{notSpace} %{notSpace} %{regex("Web login authentication failed"):log_type}: \{\\'status\\': %{integer:status}, \\'user\\': \\'%{regex("(.*)(?=\\\\')"):user}\\', \\'reason\\': "%{regex("(.*)(?=\")"):reason}", \\'auth method\\': \\'%{regex("(.*?)(?=\\\\')"):auth_method}\\'}'
+
+
+              rule4 %{regex("(.*?)(?=:)")}:(\s+)?\'%{notSpace} %{notSpace} %{regex("Web login authentication failed"):log_type}: \{\\'status\\': %{integer:status}, \\'user\\': \\'%{regex("(.*)(?=\\\\')"):user}\\', \\'reason\\': "%{regex("(.*)(?=\")"):reason}"}'


These rules have some significant overlap and could lead to fully parsing the logs multiple times. Can you reduce this redundancy?

Also, some of these seem to be JSON or keyvalue based, for example the status and user fields in rule3. Could those be extracted and remapped into the expected log fields rather than using complex parsers and regex?

We are checking on this and will update soon.

We have created two Grok parsers to handle the parsing of web authentication logs—one for the headers and another for key-value format. This separation helps eliminate overlap and reduces redundant parsing.

For auth method field, we observed that keys containing spaces were not correctly extracted by the data::keyvalue parser. To address this, we created static parsing rules.

Additionally, fields such as user and status were escaped as 'user', which prevented them from being parsed correctly by the data::keyvalue parser. To handle this, we added static rules specifically to extract these escaped fields.

gunterd · 2025-04-15T19:31:01Z

openvpn/assets/logs/openvpn.yaml

+            matchRules: >-
+              rule1 \[%{regex("(.*?)(?=])")}\](\s+)?%{regex("AUTH
+              SUCCESS"):log_type}\s+\{%{data::keyvalue(": ",", ")}, 'auth
+              method': '%{regex("(.*?)(?=\\')"):auth_method}',


Similar here - it looks like 'auth method' is a key in a JSON-like structure. Could this be parsed out into an object and then extract the value for the key you're interested in?

Can you please help us understand this comment better?

ankitarajput-crest · 2025-04-17T06:40:02Z

@gunterd We were not able to access this PR yesterday. Let me have a look at the comments and get back to you.

gunterd · 2025-04-28T18:00:11Z

openvpn/assets/logs/openvpn.yaml

+          grok:
+            supportRules: ""
+            matchRules: >-
+              rule1 \[%{regex("(.*?)(?=])")}\](\s+)?%{regex("AUTH


To reduce some of the regex burden here, can you replace \[%{regex("(.*?)(?=])")}\] with (\[\-\])?

We have used [%{regex("(.*?)(?=])")}] as a safekeeping to ensure that grok parser doesn't fail if different value comes in between [].
Based on our testing, we have only observed [-] hence we have updated regex to ([-]) for better efficiency.

gunterd · 2025-04-28T18:56:18Z

openvpn/assets/logs/openvpn.yaml

+              authentication failed: {'status': 1, 'user': 'openvpn', 'reason':
+              'local auth failed: password verification failed'}\""
+            - "[-] [WEB] OUT: '2025-03-11T12:59:46+0000 [stdout#info] Web login
+              authentication failed: {\\'status\\': 2, \\'user\\': \\'abc\\',


Do these logs come in with the backslashes escaped?

Yes, we are getting those logs with backslashes escaped.

cit-pr-commenter · 2025-04-30T08:04:00Z

PR Security Update

All commits in this PR up to and including b9cc37b have been reviewed and marked safe by SDLC security. For any questions, please reach out to #ci-for-external-contributors-collab on Slack.

* Add: OpenVPN integration * Update: labeler for openvpn * Updated tests yaml file * updated test yaml file * Updated changelog number * Updated minor changes * Updated as per PR comments * Update: address review comments * Update: address review comments * Update: minor changes and move openvpn logo * Update: remove filter from Datadog Cloud SIEM group * Update: CODEOWNERS * Update: CODEOWNER add tag * Update: address review comments * Update: logs sample * Update: Address review comments --------- Co-authored-by: manan-crest <[email protected]> 0b4ffe8

jnhunsberger

Need an update on the copy of the dashboard. Comment added.

jnhunsberger · 2025-05-16T01:34:13Z

openvpn/assets/dashboards/openvpn_overview.json

+            "id": 6980405278649262,
+            "definition": {
+                "type": "note",
+                "content": "OpenVPN is a free, open-source protocol that creates secure connections between devices over the internet. It's used to create virtual private networks (VPNs).\n\n\nThe OpenVPN Overview dashboard provides an overall insights of the logs generated by OpenVPN.\n\n\nFor more information, see the [OpenVPN Integration Documentation](https://docs.datadoghq.com/integrations/openvpn/).\n\n**Tips**\n- Use the timeframe selector in the top right of the dashboard to change the default timeframe.\n- Clone this dashboard to rearrange, modify and add widgets and visualizations. ",


Please update copy here.
Current: The OpenVPN Overview dashboard provides an overall insights of the logs generated by OpenVPN.
Proposed: The OpenVPN Overview dashboard provides insights into the logs generated by OpenVPN.

@jnhunsberger Have made the suggested changes in below raised PR.
PR Link: <link>

Add: OpenVPN integration

10fbdb6

temporal-github-worker-1 bot added agent/review-requested docs/review-requested ecosystems/review-requested product/review-requested labels Mar 12, 2025

datadog-agent-integrations-bot bot added documentation release qa/skip-qa Automatically skip this PR for the next QA labels Mar 12, 2025

Update: labeler for openvpn

d45fd74

datadog-agent-integrations-bot bot added dev/testing dev/tooling labels Mar 12, 2025

ankitarajput-crest added 3 commits March 12, 2025 18:31

Updated tests yaml file

d27cc00

updated test yaml file

77b19d2

Updated changelog number

ea9b254

ankitarajput-crest marked this pull request as ready for review March 12, 2025 14:26

ankitarajput-crest requested review from a team as code owners March 12, 2025 14:26

ankitarajput-crest requested a review from a team March 12, 2025 14:26

ankitarajput-crest requested review from a team as code owners March 12, 2025 14:26

datadog-agent-integrations-bot bot added team/agent-integrations team/documentation team/logs-backend team/Logs-Core team/logs-integrations-reviewers team/saas-integrations team/siem-logs-reviewers labels Mar 12, 2025

buraizu added the editorial review Waiting on a more in-depth review from a docs team editor label Mar 12, 2025

temporal-github-worker-1 bot removed agent/approved docs/approved labels Apr 1, 2025

iliakur previously approved these changes Apr 2, 2025

View reviewed changes

temporal-github-worker-1 bot added agent/approved and removed agent/review-requested labels Apr 2, 2025

Update: remove filter from Datadog Cloud SIEM group

8517f32

temporal-github-worker-1 bot added agent/review-requested and removed agent/approved labels Apr 7, 2025

manan-crest added 2 commits April 11, 2025 16:03

Update: CODEOWNERS

84d00e9

Update: CODEOWNER add tag

4d779b1

gunterd reviewed Apr 15, 2025

View reviewed changes

manan-crest added 2 commits April 21, 2025 18:35

Update: address review comments

bb04924

Update: logs sample

6bb04a2

ankitarajput-crest requested a review from gunterd April 28, 2025 11:15

gunterd reviewed Apr 28, 2025

View reviewed changes

Update: Address review comments

b9cc37b

gunterd approved these changes Apr 29, 2025

View reviewed changes

iliakur approved these changes May 2, 2025

View reviewed changes

temporal-github-worker-1 bot added agent/approved and removed agent/review-requested labels May 2, 2025

iliakur added this pull request to the merge queue May 5, 2025

Merged via the queue into DataDog:master with commit 0b4ffe8 May 5, 2025
47 of 48 checks passed

jnhunsberger reviewed May 16, 2025

View reviewed changes

shubhamvekariya-crest mentioned this pull request May 16, 2025

DDS: OpenVPN Update #20314

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[AI-5055] DDS: OpenVPN Integration V1.0.0 #19811

[AI-5055] DDS: OpenVPN Integration V1.0.0 #19811

Uh oh!

ankitarajput-crest commented Mar 12, 2025

Uh oh!

gunterd Apr 15, 2025

Uh oh!

ankitarajput-crest Apr 18, 2025 •

edited

Loading

Uh oh!

manan-crest Apr 21, 2025

Uh oh!

gunterd Apr 15, 2025

Uh oh!

ankitarajput-crest Apr 18, 2025

Uh oh!

ankitarajput-crest commented Apr 17, 2025

Uh oh!

gunterd Apr 28, 2025

Uh oh!

manan-crest Apr 29, 2025

Uh oh!

gunterd Apr 28, 2025

Uh oh!

manan-crest Apr 29, 2025

Uh oh!

cit-pr-commenter bot commented Apr 30, 2025

Uh oh!

Uh oh!

jnhunsberger left a comment

Uh oh!

jnhunsberger May 16, 2025

Uh oh!

ankitarajput-crest May 16, 2025

Uh oh!

Uh oh!

		rule3 %{regex("(.?)(?=:)")}:(\s+)?\'%{notSpace} %{notSpace} %{regex("Web login authentication failed"):log_type}: \{\\'status\\': %{integer:status}, \\'user\\': \\'%{regex("(.)(?=\\\\')"):user}\\', \\'reason\\': "%{regex("(.)(?=\")"):reason}", \\'auth method\\': \\'%{regex("(.?)(?=\\\\')"):auth_method}\\'}'


		rule4 %{regex("(.?)(?=:)")}:(\s+)?\'%{notSpace} %{notSpace} %{regex("Web login authentication failed"):log_type}: \{\\'status\\': %{integer:status}, \\'user\\': \\'%{regex("(.)(?=\\\\')"):user}\\', \\'reason\\': "%{regex("(.*)(?=\")"):reason}"}'

[AI-5055] DDS: OpenVPN Integration V1.0.0 #19811

[AI-5055] DDS: OpenVPN Integration V1.0.0 #19811

Uh oh!

Conversation

ankitarajput-crest commented Mar 12, 2025

What does this PR do?

Additional Notes

Review checklist (to be filled by reviewers)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankitarajput-crest Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankitarajput-crest commented Apr 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cit-pr-commenter bot commented Apr 30, 2025

PR Security Update

Uh oh!

Uh oh!

jnhunsberger left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ankitarajput-crest Apr 18, 2025 •

edited

Loading