Skip to content

Lint Errors and spl_kmem_caches regression #236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 29, 2020

Conversation

sdimitro
Copy link
Contributor

@sdimitro sdimitro commented Jul 29, 2020

First Commit: Disable pylint false-positives for pytest

Context:
Issue: pytest-dev/pytest#7473
PR that was just merged: pytest-dev/pytest#7476

Opened issue to get rid of this once pytest is updated:
#235

Second Commit: Update spl_kmem_caches command to be aware of new percpu counter

The following commit from ZoL introduced this new counter to be used instead of what sdb was looking at before
openzfs/zfs@ec1fea4

Note that since our test crash dump was generated with old bits, we'd need a new crash dump to test that new condition. Until that happens I decided that the following manual testing should suffice.

= Manual testing

Before:

sdb> spl_kmem_caches | filter 'obj.skc_linux_cache != 0x0' | spl_kmem_caches
name                     entry_size active_objs active_memory                         source total_memory util
------------------------ ---------- ----------- ------------- ------------------------------ ------------ ----
zio_link_cache                   48           0          0.0B           zio_link_cache[SLUB]       59.8KB    0
zio_data_buf_8192              8192           0          0.0B        zio_data_buf_8192[SLUB]      320.0KB    0
zio_data_buf_7168              8192           0          0.0B        zio_data_buf_7168[SLUB]      288.0KB    0
zio_data_buf_6144              8192           0          0.0B        zio_data_buf_6144[SLUB]      288.0KB    0
zio_data_buf_5120              8192           0          0.0B        zio_data_buf_5120[SLUB]      352.0KB    0
zio_data_buf_512                512           0          0.0B         zio_data_buf_512[SLUB]        1.3MB    0
zio_data_buf_4096              4096           0          0.0B        zio_data_buf_4096[SLUB]      192.0KB    0
zio_data_buf_3584              3584           0          0.0B        zio_data_buf_3584[SLUB]      220.5KB    0
...<cropped>...

After:

sdb> spl_kmem_caches | filter 'obj.skc_linux_cache != 0x0' | spl_kmem_caches
name                     entry_size active_objs active_memory                         source total_memory util
------------------------ ---------- ----------- ------------- ------------------------------ ------------ ----
dnode_t                        1240       17220        20.4MB                  dnode_t[SLUB]       20.5MB   99
zio_buf_16384                 16384        1263        19.7MB            zio_buf_16384[SLUB]       20.0MB   98
zfs_znode_cache                1096       15927        16.6MB          zfs_znode_cache[SLUB]       16.8MB   99
arc_buf_hdr_t_full              456       32307        14.0MB       arc_buf_hdr_t_full[SLUB]       14.1MB   99
dmu_buf_impl_t                  512       18062         8.8MB           dmu_buf_impl_t[SLUB]        9.1MB   97
abd_t                           248       32308         7.6MB                    abd_t[SLUB]        7.7MB   98
sa_cache                        256       15919         3.9MB                 sa_cache[SLUB]        3.9MB   99
zio_buf_512                     512        2654         1.3MB              zio_buf_512[SLUB]        1.3MB   98
zio_data_buf_1024              1024        1051         1.0MB        zio_data_buf_1024[SLUB]        1.0MB   98
zio_data_buf_512                512        1604       802.0KB         zio_data_buf_512[SLUB]        1.4MB   57
zfs_btree_leaf_cache           4096          66       264.0KB     zfs_btree_leaf_cache[SLUB]      512.0KB   51
zio_data_buf_5120              8192          31       248.0KB        zio_data_buf_5120[SLUB]      352.0KB   70
zio_data_buf_12288            12288          19       228.0KB       zio_data_buf_12288[SLUB]      336.0KB   67
...<cropped>...

Prakash pointed out a bug that was missed by my manual testing above where SPL-based caches would use the percpu counter. This was missed because I was only looking at SLUB-based caches in the above out. After re-iterating now both work as before.

sdb> spl_kmem_caches
name                     entry_size active_objs active_memory                         source total_memory util
------------------------ ---------- ----------- ------------- ------------------------------ ------------ ----
dnode_t                        1240       18266        21.6MB                  dnode_t[SLUB]       21.8MB   99
zio_buf_16384                 16384        1311        20.5MB            zio_buf_16384[SLUB]       20.8MB   98
zfs_znode_cache                1096       16888        17.7MB          zfs_znode_cache[SLUB]       17.8MB   99
arc_buf_hdr_t_full              456       33713        14.7MB       arc_buf_hdr_t_full[SLUB]       14.7MB   99
zio_buf_131072               135680          88        11.4MB           zio_buf_131072[SPL ]       18.6MB   61
dmu_buf_impl_t                  512       19150         9.4MB           dmu_buf_impl_t[SLUB]        9.5MB   98
abd_t                           248       33714         8.0MB                    abd_t[SLUB]        8.1MB   98
zio_data_buf_131072          135680          40         5.2MB      zio_data_buf_131072[SPL ]       10.4MB   50
zio_data_buf_114688          119296          40         4.6MB      zio_data_buf_114688[SPL ]        5.5MB   83
sa_cache                        256       16880         4.1MB                 sa_cache[SLUB]        4.1MB   99
zio_data_buf_98304           102912          40         3.9MB       zio_data_buf_98304[SPL ]        4.7MB   83
zio_data_buf_57344            61952          64         3.8MB       zio_data_buf_57344[SPL ]        4.3MB   88
zio_data_buf_65536            70144          56         3.7MB       zio_data_buf_65536[SPL ]        4.3MB   87
zio_data_buf_49152            53760          72         3.7MB       zio_data_buf_49152[SPL ]        4.1MB   90
...<cropped>...

Context:
Issue: pytest-dev/pytest#7473
PR that was just merged: pytest-dev/pytest#7476

Opened issue to get rid of this once `pytest` is updated:
delphix#235
@codecov-commenter
Copy link

codecov-commenter commented Jul 29, 2020

Codecov Report

Merging #236 into master will increase coverage by 0.03%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #236      +/-   ##
==========================================
+ Coverage   87.35%   87.38%   +0.03%     
==========================================
  Files          60       60              
  Lines        2467     2473       +6     
==========================================
+ Hits         2155     2161       +6     
  Misses        312      312              
Impacted Files Coverage Δ
sdb/commands/spl/internal/kmem_helpers.py 98.30% <100.00%> (+0.09%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5c1c96...dc1daf0. Read the comment docs.

Copy link
Contributor

@ahrens ahrens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice handling of both old and new ways of getting the allocation count!

# Fall back to the old-mechanism if that percpu_counter member
# doesn't exist (an AttributeError will be thrown).
#
return int(cache.skc_obj_alloc.value_())
return int(cache.skc_obj_alloc.value_())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is equivalent to what we run in the except AttributeError case, so rather than duplicating the two, would it be better to use:

except AttributeError:
    pass

and let the code reach this line in that case?

Or perhaps we should more simply remove this line, since I think it's dead code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually thanks for pointing this out... this pointed out a bug in my code. We don't want to look at the percpu counter if this is not a linux_slub backed cache. I changed the code and retesting now.

return int(cache.skc_obj_total.value_())


def obj_alloc(cache: drgn.Object) -> int:
assert sdb.type_canonical_name(cache.type_) == 'struct spl_kmem_cache *'
try:
return int(drgn_percpu.percpu_counter_sum(cache.skc_linux_alloc))
except AttributeError:
Copy link
Contributor

@prakashsurya prakashsurya Jul 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

out of curiosity, is there no way to ask if the structure contains a given member? e.g. something like:

if 'skc_linux_alloc' in cache:
    return int(drgn_percpu.percpu_counter_sum(cache.skc_linux_alloc))
else:
    return int(cache.skc_obj_alloc.value_())

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I did look at drgn.Type API for that and there wasn't such a thing. The way to do this would be something like if 'skc_linux_alloc in [ x.name for x in type.members]`. I thought that the code for the loop was probably not worth it. Does this make sense? Do you think a variant of this alternative would be preferable?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, got it. I'm good with what you have.

The following commit from ZoL introduced this counter:
openzfs/zfs@ec1fea4
@sdimitro
Copy link
Contributor Author

Updated the PR applying @prakashsurya 's suggestion and fixing a bug uncovered by it.

@sdimitro sdimitro merged commit 2ff3976 into delphix:master Jul 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants