[PyTorch Debug] Fix issue with negative underflow. #2107
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Pawel Gadzinski <[email protected]>
Signed-off-by: Pawel Gadzinski <[email protected]>
|
/te-ci pytorch |
Signed-off-by: Pawel Gadzinski <[email protected]>
|
/te-ci pytorch |
There was a problem hiding this comment.
Pull Request Overview
This PR fixes an issue with negative underflow percentage calculations in PyTorch FP8 debugging functionality. The problem was that the code only counted 0 values but missed -0 values (represented as 128 in FP8 format). Additionally, it corrects a missing subtraction in the underflow percentage computation.
Key changes:
- Updated underflow detection to include both
0and-0values usingtorch.isinwith[0, 128] - Fixed missing
- (x == 0).sum()in percentage calculation - Updated tests to use random tensors and adjusted tolerance values
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| transformer_engine/debug/features/utils/stats_computation.py | Updated underflow detection logic to handle both positive and negative zero values |
| tests/pytorch/debug/test_log.py | Modified test to use random tensors and updated MSE tolerance |
| tests/pytorch/debug/test_api_features.py | Fixed test to use dequantized tensor for underflow calculation |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
ptrendx
reviewed
Aug 25, 2025
Signed-off-by: Pawel Gadzinski <[email protected]>
for more information, see https://pre-commit.ci
timmoon10
reviewed
Aug 29, 2025
Signed-off-by: Pawel Gadzinski <[email protected]>
Signed-off-by: Pawel Gadzinski <[email protected]>
Signed-off-by: Pawel Gadzinski <[email protected]>
pggPL
force-pushed
the
fix_underflows
branch
from
a590f94 to
7464c48
Compare
for more information, see https://pre-commit.ci
Signed-off-by: Pawel Gadzinski <[email protected]>
for more information, see https://pre-commit.ci
|
/te-ci pytorch |
vthumbe1503
pushed a commit
to vthumbe1503/TransformerEngine
that referenced
this pull request
Sep 19, 2025
for more information, see https://pre-commit.ci [PyTorch Debug] Fix issue with negative underflow% stat. (NVIDIA#2107) * fix underflows log issue Signed-off-by: Pawel Gadzinski <[email protected]> * fix Signed-off-by: Pawel Gadzinski <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Pawel Gadzinski <[email protected]> * fix Signed-off-by: Pawel Gadzinski <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Pawel Gadzinski <[email protected]> * fix Signed-off-by: Pawel Gadzinski <[email protected]> * fix Signed-off-by: Pawel Gadzinski <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Pawel Gadzinski <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Pawel Gadzinski <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
phu0ngng
pushed a commit
to phu0ngng/TransformerEngine
that referenced
this pull request
Sep 22, 2025
* fix underflows log issue Signed-off-by: Pawel Gadzinski <[email protected]> * fix Signed-off-by: Pawel Gadzinski <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Pawel Gadzinski <[email protected]> * fix Signed-off-by: Pawel Gadzinski <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Pawel Gadzinski <[email protected]> * fix Signed-off-by: Pawel Gadzinski <[email protected]> * fix Signed-off-by: Pawel Gadzinski <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Pawel Gadzinski <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Pawel Gadzinski <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The issue with negative percentage of underflow was observed. The reason is that we count only
0infp8_tensor._datatensor, but we need to take into account also-0, represented by128(10000000in binary, only sign bit is 1).This PR also fixes issue with computation of percentage of underflows on one device - I forgot to add
- (x == 0).sum().Type of change
Checklist: