-
-
Notifications
You must be signed in to change notification settings - Fork 6.9k
Don't call .format()
on lazy-translated strings during Field.__init__
#3354
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Really all that needs to be changed here is that these particular lines message = self.error_messages['max_value'].format(max_value=self.max_value) need to be removed? Doing this and running the tests does not yield any test failures, but then again I am not sure where else |
ping @tomchristie or @kevin-brown for the above. If this is the right approach and I have already made the changes and could push them out. Let me know when you get a chance 😄 |
@nryoung we may lack a test for that particular string. It looks like the omitting the format will result in the max_value not being correctly filled whenever an error occurs. |
This does seem to be the case. It seems like we could use the |
I have been doing some research on this. For Django versions >= 1.8
|
encode#3354 - Update tests to reflect new error messages provided by Django field parent classes
encode#3354 - Update tests to reflect new error messages provided by Django field parent classes
to remove problem from encode/django-rest-framework#3354. Signed-off-by: Tomasz Gargas <[email protected]>
to branch with fixes to encode/django-rest-framework#3354. Signed-off-by: Tomasz Gargas <[email protected]>
As a fix for issue encode#3354, commit 607e4ed made the evaluation of some validation error messages lazy. To achieve that, Django's django.utils.functional.lazy() function was used. However, that function is extremely heavy and slow, and slows down string validation significantly (lazy() is evaluated each time for each validator for each field which has one). We noticed this in our production system. Avoid lazy and use a lambda to defer the evaluation instead. This uses the fact that a Django validator is just a simple callable. Using the benchmark attached to the PR (snipped to tottime>100ms): Before, model serializer: 9225123 function calls (9200068 primitive calls) in 8.337 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 25000 1.299 0.000 3.534 0.000 functional.py:125(__prepare_class__) 2415001 0.954 0.000 0.954 0.000 {built-in method builtins.hasattr} 1350000 0.901 0.000 0.901 0.000 functional.py:145(__promise__) 1550001 0.521 0.000 0.521 0.000 {built-in method builtins.setattr} 25000 0.494 0.000 0.735 0.000 {built-in method builtins.__build_class__} 30000 0.298 0.000 0.385 0.000 fields.py:297(__init__) 25000 0.289 0.000 5.895 0.000 fields.py:740(__init__) 25000 0.264 0.000 0.802 0.000 field_mapping.py:66(get_field_kwargs) 25000 0.241 0.000 0.241 0.000 functional.py:100(__proxy__) 670003/670001 0.203 0.000 0.211 0.000 {built-in method builtins.getattr} 5000 0.189 0.000 7.722 0.002 serializers.py:990(get_fields) 25000 0.186 0.000 0.400 0.000 functools.py:186(total_ordering) 25000 0.158 0.000 0.299 0.000 functional.py:234(wrapper) 5000 0.129 0.000 0.136 0.000 serializers.py:1066(get_field_names) 25000 0.104 0.000 1.002 0.000 serializers.py:1195(build_standard_field) After, model serializer: 3400123 function calls (3400068 primitive calls) in 2.824 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 30000 0.259 0.000 0.341 0.000 fields.py:294(__init__) 25000 0.242 0.000 0.674 0.000 field_mapping.py:66(get_field_kwargs) 5000 0.176 0.000 2.209 0.000 serializers.py:990(get_fields) 25000 0.133 0.000 0.514 0.000 fields.py:737(__init__) 295003/295001 0.106 0.000 0.115 0.000 {built-in method builtins.getattr} Before, regular serializer: 8060003 function calls (7960003 primitive calls) in 7.123 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 25000 1.569 0.000 3.897 0.000 functional.py:125(__prepare_class__) 1350000 1.013 0.000 1.013 0.000 functional.py:145(__promise__) 2365000 0.925 0.000 0.925 0.000 {built-in method builtins.hasattr} 1550000 0.512 0.000 0.512 0.000 {built-in method builtins.setattr} 25000 0.378 0.000 0.550 0.000 {built-in method builtins.__build_class__} 25000 0.307 0.000 5.946 0.000 fields.py:740(__init__) 30000 0.277 0.000 0.360 0.000 fields.py:297(__init__) 80000/5000 0.202 0.000 6.526 0.001 copy.py:132(deepcopy) 25000 0.172 0.000 0.172 0.000 functional.py:100(__proxy__) 540000 0.152 0.000 0.152 0.000 {built-in method builtins.getattr} 25000 0.119 0.000 6.199 0.000 fields.py:604(__deepcopy__) After, regular serializer: 2010003 function calls (1935003 primitive calls) in 1.456 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 30000 0.194 0.000 0.263 0.000 fields.py:294(__init__) 80000/5000 0.151 0.000 0.923 0.000 copy.py:132(deepcopy) 25000 0.128 0.000 0.470 0.000 fields.py:737(__init__)
As a fix for issue encode#3354, commit 607e4ed made the evaluation of some validation error messages lazy. To achieve that, Django's django.utils.functional.lazy() function was used. However, that function is extremely heavy and slow, and slows down string validation significantly (lazy() is evaluated each time for each validator for each field which has one). We noticed this in our production system. Use a custom lazy_format() object instead which does the formatting lazily with less overhead. Using the benchmark attached to the PR (snipped to tottime>100ms): Before, model serializer: 9225123 function calls (9200068 primitive calls) in 8.337 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 25000 1.299 0.000 3.534 0.000 functional.py:125(__prepare_class__) 2415001 0.954 0.000 0.954 0.000 {built-in method builtins.hasattr} 1350000 0.901 0.000 0.901 0.000 functional.py:145(__promise__) 1550001 0.521 0.000 0.521 0.000 {built-in method builtins.setattr} 25000 0.494 0.000 0.735 0.000 {built-in method builtins.__build_class__} 30000 0.298 0.000 0.385 0.000 fields.py:297(__init__) 25000 0.289 0.000 5.895 0.000 fields.py:740(__init__) 25000 0.264 0.000 0.802 0.000 field_mapping.py:66(get_field_kwargs) 25000 0.241 0.000 0.241 0.000 functional.py:100(__proxy__) 670003/670001 0.203 0.000 0.211 0.000 {built-in method builtins.getattr} 5000 0.189 0.000 7.722 0.002 serializers.py:990(get_fields) 25000 0.186 0.000 0.400 0.000 functools.py:186(total_ordering) 25000 0.158 0.000 0.299 0.000 functional.py:234(wrapper) 5000 0.129 0.000 0.136 0.000 serializers.py:1066(get_field_names) 25000 0.104 0.000 1.002 0.000 serializers.py:1195(build_standard_field) After, model serializer: 3265096 function calls (3240059 primitive calls) in 2.645 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 30000 0.237 0.000 0.315 0.000 fields.py:295(__init__) 25000 0.218 0.000 0.639 0.000 field_mapping.py:66(get_field_kwargs) 25000 0.214 0.000 0.665 0.000 fields.py:743(__init__) 5000 0.156 0.000 2.086 0.000 serializers.py:988(get_fields) 25000 0.107 0.000 0.210 0.000 functional.py:234(wrapper) Before, regular serializer: 8060003 function calls (7960003 primitive calls) in 7.123 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 25000 1.569 0.000 3.897 0.000 functional.py:125(__prepare_class__) 1350000 1.013 0.000 1.013 0.000 functional.py:145(__promise__) 2365000 0.925 0.000 0.925 0.000 {built-in method builtins.hasattr} 1550000 0.512 0.000 0.512 0.000 {built-in method builtins.setattr} 25000 0.378 0.000 0.550 0.000 {built-in method builtins.__build_class__} 25000 0.307 0.000 5.946 0.000 fields.py:740(__init__) 30000 0.277 0.000 0.360 0.000 fields.py:297(__init__) 80000/5000 0.202 0.000 6.526 0.001 copy.py:132(deepcopy) 25000 0.172 0.000 0.172 0.000 functional.py:100(__proxy__) 540000 0.152 0.000 0.152 0.000 {built-in method builtins.getattr} 25000 0.119 0.000 6.199 0.000 fields.py:604(__deepcopy__) After, regular serializer: 2150003 function calls (2050003 primitive calls) in 1.609 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 30000 0.224 0.000 0.293 0.000 fields.py:295(__init__) 25000 0.181 0.000 0.607 0.000 fields.py:743(__init__) 80000/5000 0.151 0.000 1.074 0.000 copy.py:132(deepcopy) 25000 0.102 0.000 0.819 0.000 fields.py:607(__deepcopy__)
As a fix for issue encode#3354, commit 607e4ed made the evaluation of some validation error messages lazy. To achieve that, Django's django.utils.functional.lazy() function was used. However, that function is extremely heavy and slow, and slows down string validation significantly (lazy() is evaluated each time for each validator for each field which has one). We noticed this in our production system. Use a custom lazy_format() object instead which does the formatting lazily with less overhead. Using the benchmark attached to the PR (snipped to tottime>100ms): Before, model serializer: 9225123 function calls (9200068 primitive calls) in 8.337 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 25000 1.299 0.000 3.534 0.000 functional.py:125(__prepare_class__) 2415001 0.954 0.000 0.954 0.000 {built-in method builtins.hasattr} 1350000 0.901 0.000 0.901 0.000 functional.py:145(__promise__) 1550001 0.521 0.000 0.521 0.000 {built-in method builtins.setattr} 25000 0.494 0.000 0.735 0.000 {built-in method builtins.__build_class__} 30000 0.298 0.000 0.385 0.000 fields.py:297(__init__) 25000 0.289 0.000 5.895 0.000 fields.py:740(__init__) 25000 0.264 0.000 0.802 0.000 field_mapping.py:66(get_field_kwargs) 25000 0.241 0.000 0.241 0.000 functional.py:100(__proxy__) 670003/670001 0.203 0.000 0.211 0.000 {built-in method builtins.getattr} 5000 0.189 0.000 7.722 0.002 serializers.py:990(get_fields) 25000 0.186 0.000 0.400 0.000 functools.py:186(total_ordering) 25000 0.158 0.000 0.299 0.000 functional.py:234(wrapper) 5000 0.129 0.000 0.136 0.000 serializers.py:1066(get_field_names) 25000 0.104 0.000 1.002 0.000 serializers.py:1195(build_standard_field) After, model serializer: 3265096 function calls (3240059 primitive calls) in 2.645 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 30000 0.237 0.000 0.315 0.000 fields.py:295(__init__) 25000 0.218 0.000 0.639 0.000 field_mapping.py:66(get_field_kwargs) 25000 0.214 0.000 0.665 0.000 fields.py:743(__init__) 5000 0.156 0.000 2.086 0.000 serializers.py:988(get_fields) 25000 0.107 0.000 0.210 0.000 functional.py:234(wrapper) Before, regular serializer: 8060003 function calls (7960003 primitive calls) in 7.123 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 25000 1.569 0.000 3.897 0.000 functional.py:125(__prepare_class__) 1350000 1.013 0.000 1.013 0.000 functional.py:145(__promise__) 2365000 0.925 0.000 0.925 0.000 {built-in method builtins.hasattr} 1550000 0.512 0.000 0.512 0.000 {built-in method builtins.setattr} 25000 0.378 0.000 0.550 0.000 {built-in method builtins.__build_class__} 25000 0.307 0.000 5.946 0.000 fields.py:740(__init__) 30000 0.277 0.000 0.360 0.000 fields.py:297(__init__) 80000/5000 0.202 0.000 6.526 0.001 copy.py:132(deepcopy) 25000 0.172 0.000 0.172 0.000 functional.py:100(__proxy__) 540000 0.152 0.000 0.152 0.000 {built-in method builtins.getattr} 25000 0.119 0.000 6.199 0.000 fields.py:604(__deepcopy__) After, regular serializer: 2150003 function calls (2050003 primitive calls) in 1.609 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 30000 0.224 0.000 0.293 0.000 fields.py:295(__init__) 25000 0.181 0.000 0.607 0.000 fields.py:743(__init__) 80000/5000 0.151 0.000 1.074 0.000 copy.py:132(deepcopy) 25000 0.102 0.000 0.819 0.000 fields.py:607(__deepcopy__)
As a fix for issue encode#3354, commit 607e4ed made the evaluation of some validation error messages lazy. To achieve that, Django's django.utils.functional.lazy() function was used. However, that function is extremely heavy and slow, and slows down string validation significantly (lazy() is evaluated each time for each validator for each field which has one). We noticed this in our production system. Use a custom lazy_format() object instead which does the formatting lazily with less overhead. Using the benchmark attached to the PR (snipped to tottime>100ms): Before, model serializer: 9225123 function calls (9200068 primitive calls) in 8.337 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 25000 1.299 0.000 3.534 0.000 functional.py:125(__prepare_class__) 2415001 0.954 0.000 0.954 0.000 {built-in method builtins.hasattr} 1350000 0.901 0.000 0.901 0.000 functional.py:145(__promise__) 1550001 0.521 0.000 0.521 0.000 {built-in method builtins.setattr} 25000 0.494 0.000 0.735 0.000 {built-in method builtins.__build_class__} 30000 0.298 0.000 0.385 0.000 fields.py:297(__init__) 25000 0.289 0.000 5.895 0.000 fields.py:740(__init__) 25000 0.264 0.000 0.802 0.000 field_mapping.py:66(get_field_kwargs) 25000 0.241 0.000 0.241 0.000 functional.py:100(__proxy__) 670003/670001 0.203 0.000 0.211 0.000 {built-in method builtins.getattr} 5000 0.189 0.000 7.722 0.002 serializers.py:990(get_fields) 25000 0.186 0.000 0.400 0.000 functools.py:186(total_ordering) 25000 0.158 0.000 0.299 0.000 functional.py:234(wrapper) 5000 0.129 0.000 0.136 0.000 serializers.py:1066(get_field_names) 25000 0.104 0.000 1.002 0.000 serializers.py:1195(build_standard_field) After, model serializer: 3265096 function calls (3240059 primitive calls) in 2.645 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 30000 0.237 0.000 0.315 0.000 fields.py:295(__init__) 25000 0.218 0.000 0.639 0.000 field_mapping.py:66(get_field_kwargs) 25000 0.214 0.000 0.665 0.000 fields.py:743(__init__) 5000 0.156 0.000 2.086 0.000 serializers.py:988(get_fields) 25000 0.107 0.000 0.210 0.000 functional.py:234(wrapper) Before, regular serializer: 8060003 function calls (7960003 primitive calls) in 7.123 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 25000 1.569 0.000 3.897 0.000 functional.py:125(__prepare_class__) 1350000 1.013 0.000 1.013 0.000 functional.py:145(__promise__) 2365000 0.925 0.000 0.925 0.000 {built-in method builtins.hasattr} 1550000 0.512 0.000 0.512 0.000 {built-in method builtins.setattr} 25000 0.378 0.000 0.550 0.000 {built-in method builtins.__build_class__} 25000 0.307 0.000 5.946 0.000 fields.py:740(__init__) 30000 0.277 0.000 0.360 0.000 fields.py:297(__init__) 80000/5000 0.202 0.000 6.526 0.001 copy.py:132(deepcopy) 25000 0.172 0.000 0.172 0.000 functional.py:100(__proxy__) 540000 0.152 0.000 0.152 0.000 {built-in method builtins.getattr} 25000 0.119 0.000 6.199 0.000 fields.py:604(__deepcopy__) After, regular serializer: 2150003 function calls (2050003 primitive calls) in 1.609 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 30000 0.224 0.000 0.293 0.000 fields.py:295(__init__) 25000 0.181 0.000 0.607 0.000 fields.py:743(__init__) 80000/5000 0.151 0.000 1.074 0.000 copy.py:132(deepcopy) 25000 0.102 0.000 0.819 0.000 fields.py:607(__deepcopy__)
This from IRC...
Which is:
And...
The text was updated successfully, but these errors were encountered: