Skip to content

"net_msg_max limit reached" consumes CPU #3825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Gerold103 opened this issue Nov 21, 2018 · 4 comments
Closed

"net_msg_max limit reached" consumes CPU #3825

Gerold103 opened this issue Nov 21, 2018 · 4 comments
Labels
bug Something isn't working duplicate This issue or pull request already exists iproto performance

Comments

@Gerold103
Copy link
Collaborator

Gerold103 commented Nov 21, 2018

Tarantool version:
>= 1.9
OS version:
any
Bug description:
Under load on each new request "net_msg_max limit reached" message is logged. It consumes CPU reducing RPS in several times, and disk wasting gigabytes to just store copies of this message.
Steps to reproduce:
Just reach net_msg_max.

@Gerold103 Gerold103 added bug Something isn't working performance iproto labels Nov 21, 2018
@Mons
Copy link
Contributor

Mons commented Nov 21, 2018

I think it would be sufficient to limit output to max 1 message per second

@Gerold103
Copy link
Collaborator Author

Yes. But I think the main problem of this issue is how to make it reusable for any other similar log messages. For example, do not print too long wal write on each occurrence - we can have 50k fibers, first made too long wal write and all others will have it too = 50k messages.

@Gerold103
Copy link
Collaborator Author

Number of such unique log messages is not too big, so a possible solution is add a static variable in a function, making such frequent logging in which we will store last log time and number of occurrences. If more than one message appeared in a small time interval, we will log it once with saying how many times it occurred. It will work for all cases.

@locker
Copy link
Member

locker commented Nov 22, 2018

See #2218

@locker locker closed this as completed Nov 23, 2018
@locker locker added the duplicate This issue or pull request already exists label Nov 23, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working duplicate This issue or pull request already exists iproto performance
Projects
None yet
Development

No branches or pull requests

3 participants