Skip to content

Optimizing the performance of think length limit using custom operators #4279

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 24 commits into
base: develop
Choose a base branch
from
Open

Optimizing the performance of think length limit using custom operators #4279

wants to merge 24 commits into from
+480 −567

Conversation

yuanlehome
Copy link
Collaborator

@yuanlehome yuanlehome commented Sep 25, 2025 •
edited
Loading

Changed:

  1. 修复模型注册机制，原本category是没有生效的，增加is_reasoning_model判断函数（暂未用到）
  2. 删除非nv gpu硬件的思考长度裁剪逻辑代码（原本也没有生效和验证过，后续需要通过自定义算子的形式支持）
  3. 修改common_engine.py里的cfg->fd_config，便于阅读
  4. 支持nv gpu下的思考长度裁剪功能，通过环境变量FD_LIMIT_THINKING_CONTENT_TRUNCATE_STR来指定用于截断思考的插入字符串，</think> for ernie4_5_vl, \n</think>\n\n for ernie_x1，分别实现了两个自定义算子
  5. 其他一些代码优化

Comming soon:

  1. 支持MTP下的思考长度裁剪

@paddle-bot Paddle-bot
Copy link

paddle-bot bot commented Sep 25, 2025

Thanks for your contribution!

@yuanlehome yuanlehome marked this pull request as draft September 26, 2025 05:47
@yuanlehome yuanlehome force-pushed the upgrade_limit_think_length branch from 3bad98a to 73384a6 Compare October 13, 2025 03:19
@yuanlehome yuanlehome marked this pull request as ready for review October 13, 2025 08:07
@yuanlehome yuanlehome force-pushed the upgrade_limit_think_length branch from 62dd5da to 6f1f082 Compare October 14, 2025 10:24
K11OntheBoat
K11OntheBoat previously approved these changes Oct 16, 2025
View reviewed changes
Copy link
Collaborator

@K11OntheBoat K11OntheBoat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
得check一下PD分离下，max_think_len=1的极端情况
可以参考这个PR: #4433
这个PR的一些改动修复了极端情况的Bug.

custom_ops/gpu_ops/limit_thinking_content_length_v2.cu Outdated
__global__ void limit_thinking_content_length_kernel_v2(
int64_t *next_tokens,
const int *max_think_lens,
const int64_t *step_idx, // step_idx 不再需要被修改，改为 const
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个注释删了

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

xiaoxiaohehe001
xiaoxiaohehe001 previously approved these changes Oct 16, 2025
View reviewed changes
Copy link
Collaborator

@xiaoxiaohehe001 xiaoxiaohehe001 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaoxiaohehe001 xiaoxiaohehe001 xiaoxiaohehe001 left review comments

@K11OntheBoat K11OntheBoat K11OntheBoat left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yuanlehome @xiaoxiaohehe001 @K11OntheBoat