Update Max signals value to supported limits #4556
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shashank-elastic
requested review from
traut,
Mikaayenson,
approksiu and
eric-forte-elastic
Rule: Tuning - GuidelinesThese guidelines serve as a reminder set of considerations when tuning an existing rule. Documentation and Context
Rule Metadata Checks
Testing and Validation
|
shashank-elastic
force-pushed
the
issue-4539
branch
from
62c810f to
b34df64
Compare
botelastic
bot
added
Domain: Endpoint
OS: Linux
OS: Windows
shashank-elastic
changed the title
|
⛔️ Test failed Results- ❌ Potential Network Scan Detected (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Memory Threat - Detected - Elastic Defend (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ✅ Potential Cookies Theft via Browser Debugging (eql) - ❌ Memory Threat - Prevented- Elastic Defend (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Malware - Detected - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Ransomware - Detected - Elastic Defend (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Behavior - Detected - Elastic Defend (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Ransomware - Prevented - Elastic Defend (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Potential Internal Linux SSH Brute Force Detected (eql) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Exploit - Detected - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Exploit - Prevented - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Potential Suspicious File Edit (eql) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Rapid7 Threat Command CVEs Correlation (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Malware - Prevented - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ✅ NTDS or SAM Database File Copied (eql) - ❌ Permission Theft - Prevented - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Credential Dumping - Detected - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Creation of Hidden Shared Object File (eql) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Adversary Behavior - Detected - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Potential Network Sweep Detected (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Process Injection - Detected - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ SUID/SGID Bit Set (eql) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Ransomware - Detected - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ File made Immutable by Chattr (eql) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Process Injection - Prevented - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Endpoint Security (Elastic Defend) (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ My First Rule (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Timestomping using Touch Command (eql) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Creation of Hidden Files and Directories via CommandLine (eql) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Potential SYN-Based Port Scan Detected (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Credential Manipulation - Detected - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ✅ Microsoft IIS Connection Strings Decryption (eql) - ❌ Permission Theft - Detected - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Credential Manipulation - Prevented - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Credential Dumping - Prevented - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Ransomware - Prevented - Elastic Endgame (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ External Alerts (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Behavior - Prevented - Elastic Defend (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ✅ IIS HTTP Logging Disabled (eql) - ❌ Malicious File - Detected - Elastic Defend (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Malicious File - Prevented - Elastic Defend (kuery) - coverage_issue: no_rta - stack_validation_failed: no_rta - ❌ Potential External Linux SSH Brute Force Detected (eql) - coverage_issue: no_rta - stack_validation_failed: no_rta |
|
⛔️ Test failed Results
|
|
⛔️ Test failed Results
|
|
⛔️ Test failed Results
|
|
⛔️ Test failed Results
|
|
⛔️ Test failed Results
|
| 'projects.' | ||
| max_signal_standard_setup = 'For information about troubleshooting maximum alerts warning '\ | ||
| 'please refer to this [guide]'\ | ||
| '(https://www.elastic.co/guide/en/security/current/alerts-ui-monitor.html#troubleshoot-max-alerts).' # noqa: E501 | ||
| for rule in self.all_rules: | ||
| if rule.contents.data.max_signals and rule.contents.data.max_signals > 1000: |
There was a problem hiding this comment.
Very much a Nit. We should still be using .get here right? Prior comment was a reference to comparing rule.contents.data.get(max_signals) to something like rule.contents.get(data).get(max_signals)
w0rk3r
previously requested changes
Mar 26, 2025
| To make sure this rule can generate as many alerts as it's configured in its own **Max alerts per run** setting, increase the `xpack.alerting.rules.run.alerts.max` system setting accordingly. | ||
|
|
||
| **NOTE:** Changing `xpack.alerting.rules.run.alerts.max` is not possible in Serverless projects. | ||
| For information about troubleshooting maximum alerts warning please refer to this [guide](https://www.elastic.co/guide/en/security/current/alerts-ui-monitor.html#troubleshoot-max-alerts). |
There was a problem hiding this comment.
Suggested change
| For information about troubleshooting maximum alerts warning please refer to this [guide](https://www.elastic.co/guide/en/security/current/alerts-ui-monitor.html#troubleshoot-max-alerts). | |
| For information on troubleshooting the maximum alerts warning, please refer to this [guide](https://www.elastic.co/guide/en/security/current/alerts-ui-monitor.html#troubleshoot-max-alerts). |
Small adjust
There was a problem hiding this comment.
@approksiu thoughts on this, as we have derived this from the issue template.
There was a problem hiding this comment.
Looks good! I am fine with the change
There was a problem hiding this comment.
This would need to be changed across all rules in this PR
| To make sure this rule can generate as many alerts as it's configured in its own **Max alerts per run** setting, increase the `xpack.alerting.rules.run.alerts.max` system setting accordingly. | ||
|
|
||
| **NOTE:** Changing `xpack.alerting.rules.run.alerts.max` is not possible in Serverless projects. | ||
| For information about troubleshooting maximum alerts warning please refer to this [guide](https://www.elastic.co/guide/en/security/current/alerts-ui-monitor.html#troubleshoot-max-alerts). |
There was a problem hiding this comment.
I'm not sure if this is needed for this one
There was a problem hiding this comment.
Maintaining the style of the previous change to also have consistent setup notes on max_signals = 1000
There was a problem hiding this comment.
I think there is less likelihood that this rule will hit the limit. It's ok to remove this note.
|
⛔️ Test failed Results
|
eric-forte-elastic
approved these changes
Mar 26, 2025
DefSecSentinel
approved these changes
Mar 27, 2025
|
⛔️ Test failed Results
|
|
Given that this change request is blocking the release and the wording change is minor, we will accept the change and continue with the release as it would otherwise be blocked. |
|
⛔️ Test failed Results
|
|
This commit request has been applied in 6f0d2f9. Dismissing request changes to reflect this. |
eric-forte-elastic
dismissed
w0rk3r’s stale review
This commit request has been applied in 6f0d2f9. Dismissing request changes to reflect this.
r0ot
added a commit
to VigilantSec/detection-rules
that referenced
this pull request
Apr 16, 2025
* Temporaily Disable Changed FIles Workflow (elastic#4538) * Temporaily Disable Changed FIles Workflow * bump version * Add new ML detection rules for Privileged Access Detection (elastic#4516) Add detection-rules for privileged access detection integration * Revert "Add new ML detection rules for Privileged Access Detection (elastic#4516)" (elastic#4548) This reverts commit 2ff8d1b. * Min stack rules from 4516 (elastic#4549) * Update defense_evasion_posh_assembly_load.toml (elastic#4543) Co-authored-by: Jonhnathan <[email protected]> * Change description and name of problemchild ML detection-rules (elastic#4545) Changed description and name of problemchild ML detection-rules * Prep for Release 9.0 (elastic#4550) * [ci] Add new docs-builder automation. (elastic#4507) * Add new docs automation * Add path-pattern filters for documentation folders * Update .github/workflows/docs-build.yml Co-authored-by: Jan Calanog <[email protected]> --------- Co-authored-by: Mika Ayenson, PhD <[email protected]> Co-authored-by: Terrance DeJesus <[email protected]> Co-authored-by: Jan Calanog <[email protected]> Co-authored-by: Sergey Polzunov <[email protected]> * [FR] Bump changed-files Version to Patched Version (elastic#4542) * Bump changed-files Version to Patched Version * patch bump * reenable workflow * Use full length commit hash * Bump 44 to 46 --------- Co-authored-by: shashank-elastic <[email protected]> * [New Rule] Adding Coverage for DynamoDB Exfiltration Behaviors (elastic#4535) * new rules for AWS DynamoDB data exfiltration * bumping patch version * adjusting investigation guide * updating patch version * updating patch version * updating patch version --------- Co-authored-by: Colson Wilhoit <[email protected]> * fix: removing outdated code in Kibana client auth (elastic#4495) * Simplify kibana session management * Drop removed options from `kibana_args` set * Style fix * Patch version bump * Bumping kibana lib version * Relax CLI requirement, making `api_key` optional, to allow `help` to run * Create new detection rule set documentation to be included in the new docs. (elastic#4508) * move docs folder to docs-dev * Add new docs folder * update docset.yml to reflect latest usage * Add rules_building_block folder * revert changes to docs-dev/experimental-machine-learning/url-spoof.md * bump patch versions * revert bump --------- Co-authored-by: Mika Ayenson, PhD <[email protected]> Co-authored-by: Terrance DeJesus <[email protected]> * fixing double header in investigation notes (elastic#4490) * [Bug] Update Custom Rules Markdown Location (elastic#4565) * Update to custom-rules markdown location * bump version * Update link reference * Prep main for 9.1 (elastic#4555) * Prep for Release 9.1 * Update Patch Version * Update Patch version * Update Patch version * [Rule Tuning] Added OWA (outlook for web) new AppID (elastic#4568) * Added OWA (outlook for web) new AppID **Title:** Add new Outlook for Web AppID to abnormal Microsoft 365 ClientAppID rule **Description:** This pull request updates the `initial_access_microsoft_365_abnormal_clientappid` rule to include the newly introduced Outlook for Web AppID: - **New AppID**: `9199bf20-a13f-4107-85dc-02114787ef48` ### Context Outlook for Web (OWA) is migrating to a new authentication platform using MSAL and a Single Page Application (SPA) auth model. As part of this backend change, Microsoft is replacing the existing OWA AppID with a new one. This change is being rolled out during the first half of calendar year 2024, with full deployment expected by Q4 2024. - **Old OWA AppID**: `00000002-0000-0ff1-ce00-000000000000` - **New OWA AppID**: `9199bf20-a13f-4107-85dc-02114787ef48` Although no action is required for tenant administrators, this new AppID may show up in logs and should be accounted for in detections relying on known legitimate ClientAppIDs. ### Why this change? The rule `initial_access_microsoft_365_abnormal_clientappid` flags potentially suspicious or unauthorized client applications accessing Microsoft 365 services. To prevent false positives caused by this official change from Microsoft, this PR adds the new OWA AppID to the allowlist. ### References - Microsoft 365 Message Center notice (ref: MC715025) - [MSAL documentation](https://learn.microsoft.com/en-us/azure/active-directory/develop/msal-overview) * Update initial_access_microsoft_365_abnormal_clientappid.toml Updated updated_date * Update Max signals value to supported limits (elastic#4556) * Update ATT&CK coverage URL(s) in docs-dev/ATT&CK-coverage.md (elastic#4571) * deprecating 'Azure Virtual Network Device Modified or Deleted' (elastic#4559) * tuning 'Azure Conditional Access Policy Modified' (elastic#4558) * [Rule Tuning] Tuning Illicit Grant Consent Detections in Azure and M365 (elastic#4557) * tuning Azure rule for illicit grant activity; creating new rule for M365 * Update rules/integrations/o365/initial_access_microsoft_365_illicit_consent_grant_via_registered_application.toml * Update rules/integrations/azure/initial_access_entra_illicit_consent_grant_via_registered_application.toml * adjusted tags * Update rules/integrations/azure/initial_access_entra_illicit_consent_grant_via_registered_application.toml * Update defense_evasion_microsoft_defender_tampering.toml (elastic#4573) Co-authored-by: Jonhnathan <[email protected]> * [Rule Tuning] O365 Exchange Suspicious Mailbox Right Delegation (elastic#4589) * Update docset.yml (elastic#4590) Remove diagnostic hint * [New] Unusual Network Connection to Suspicious Web Service (elastic#4569) * [New] Unusual Network Connection to Suspicious Web Service * Update rule threat order --------- Co-authored-by: Eric Forte <[email protected]> Co-authored-by: eric-forte-elastic <[email protected]> Co-authored-by: Mika Ayenson, PhD <[email protected]> * [New] Unusual Network Connection to Suspicious Top Level Domain (elastic#4563) * [Rule Tuning] Add Host Metadata to ES|QL Aggregation Rules (elastic#4592) Co-authored-by: Mika Ayenson, PhD <[email protected]> * Add investigation guides (elastic#4600) * [Rule Tuning] Suspicious Execution via Scheduled Task (elastic#4599) * Lock versions for releases: 8.14,8.15,8.16,8.17,8.18,9.0 (elastic#4601) * [FN Tuning] Shared Object Created or Changed by Previously Unknown Pr… (elastic#4529) * [FN Tuning] Shared Object Created or Changed by Previously Unknown Process * Update process exclusions in TOML file --------- Co-authored-by: Colson Wilhoit <[email protected]> Co-authored-by: shashank-elastic <[email protected]> Co-authored-by: Terrance DeJesus <[email protected]> Co-authored-by: Eric Forte <[email protected]> * [D4C Conversion] Converting Compatible D4C Rules to DR (elastic#4532) * [D4C Conversion] Converting Compatible D4C Rules to DR * added host.os.type * Rename * Update rules/linux/execution_container_management_binary_launched_inside_container.toml Co-authored-by: Isai <[email protected]> * Update rules/linux/privilege_escalation_debugfs_launched_inside_container.toml Co-authored-by: Isai <[email protected]> * Update rules/linux/privilege_escalation_debugfs_launched_inside_container.toml Co-authored-by: Isai <[email protected]> * Update rules/linux/privilege_escalation_mount_launched_inside_container.toml Co-authored-by: Isai <[email protected]> * Update rules/linux/privilege_escalation_mount_launched_inside_container.toml Co-authored-by: Isai <[email protected]> --------- Co-authored-by: Isai <[email protected]> Co-authored-by: shashank-elastic <[email protected]> Co-authored-by: Terrance DeJesus <[email protected]> Co-authored-by: Mika Ayenson, PhD <[email protected]> * Remove Task List reference (elastic#4605) * [FR] Update Detection Rules MITRE Workflow to SHA Pin (elastic#4581) * Update to pinned hash * version bump * [FR] Add Kibana Action Connector Error to Exception List Workaround (elastic#4583) * Add error catch for workaround * Switch to set for efficiency * Patch version bump --------- Co-authored-by: Mika Ayenson, PhD <[email protected]> * [Rule Tuning] SSH Authorized Keys File Deletion (elastic#4591) Co-authored-by: Mika Ayenson, PhD <[email protected]> * [Rule Tuning] Suspicious WMI Event Subscription Created (elastic#4618) * [Rule Tuning] Suspicious Execution via Scheduled Task * [Rule Tuning] Suspicious WMI Event Subscription Created * [Rule Tuning] Adjusting `Microsoft Entra ID Rare Authentication Requirement for Principal User` (elastic#4562) * tuning 'Microsoft Entra ID Rare Authentication Requirement for Principal User' * updated MITRE ATT&CK mappings * updated index target * updated patch version * updating patch version * bumping patch version * updating patch version * [Rule Tuning] Tuning `Azure Service Principal Credentials Added` (elastic#4570) * tuning 'Azure Service Principal Credentials Added' * updated patch version * added investigation guide * updating patch version * updating patch version * [FR] Add Support for Local Dates Flag (elastic#4582) * Add support for local dates flag * Use two variables * Add support for import-rules-to-repo * Revert arg formatting * Update comment * Pass Rule Path as Path Object * Update to rule loader function * Streamline metadata function * Also support dictionaries * Bump patch version * Reduce complexity * Add if path exists check * Fix version bump * Feature exclude tactic name (elastic#4593) * Added new cli flag to exclude tactic name in rule file name * added a shortcut for the flag and adjusted CLI readme * Add no tactic flag also to import to prevent warnings * Added info about unit test * version bump * Added no_tactic_filename as config option + fixed linting * pyproject version bump --------- Co-authored-by: Mika Ayenson, PhD <[email protected]> Co-authored-by: Eric Forte <[email protected]> --------- Co-authored-by: Eric Forte <[email protected]> Co-authored-by: Kirti Sodhi <[email protected]> Co-authored-by: Samirbous <[email protected]> Co-authored-by: Jonhnathan <[email protected]> Co-authored-by: shashank-elastic <[email protected]> Co-authored-by: Martijn Laarman <[email protected]> Co-authored-by: Mika Ayenson, PhD <[email protected]> Co-authored-by: Terrance DeJesus <[email protected]> Co-authored-by: Jan Calanog <[email protected]> Co-authored-by: Sergey Polzunov <[email protected]> Co-authored-by: Colson Wilhoit <[email protected]> Co-authored-by: M. Visser <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: eric-forte-elastic <[email protected]> Co-authored-by: Ruben Groenewoud <[email protected]> Co-authored-by: Isai <[email protected]> Co-authored-by: Frederik Berg <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request
Issue link(s): #4539
Summary - What I changed
Note - For Self ( Author)
How To Test
Check for missing setup note for rules with max_signals == 1000
Check for rules with max_signals exceeding 1000 count
Checklist
bug,enhancement,schema,maintenance,Rule: New,Rule: Deprecation,Rule: Tuning,Hunt: New, orHunt: Tuningso guidelines can be generatedmeta:rapid-mergelabel if planning to merge within 24 hoursContributor checklist