Skip to content

PERF: to speed up rendering of styler #34863

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 9, 2020
Merged

PERF: to speed up rendering of styler #34863

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
89b2d70
PERF: speed up rendering of styler (#19917)
jihwans Jun 18, 2020
e4d4370
PERF: speed up rendering of styler (#19917)
jihwans Jun 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions asv_bench/asv.conf.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
"xlwt": [],
"odfpy": [],
"pytest": [],
"jinja2": [],
// If using Windows with python 2.7 and want to build using the
// mingw toolchain (rather than MSVC), uncomment the following line.
// "libpython": [],
Expand Down
34 changes: 34 additions & 0 deletions asv_bench/benchmarks/io/style.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
import numpy as np

from pandas import DataFrame


class RenderApply:

params = [[12, 24, 36], [12, 120]]
param_names = ["cols", "rows"]

def setup(self, cols, rows):
self.df = DataFrame(
np.random.randn(rows, cols),
columns=[f"float_{i+1}" for i in range(cols)],
index=[f"row_{i+1}" for i in range(rows)],
)
self._style_apply()

def time_render(self, cols, rows):
self.st.render()

def peakmem_apply(self, cols, rows):
self._style_apply()

def peakmem_render(self, cols, rows):
self.st.render()

def _style_apply(self):
def _apply_func(s):
return [
"background-color: lightcyan" if s.name == "row_1" else "" for v in s
]

self.st = self.df.style.apply(_apply_func, axis=1)
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.1.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -806,6 +806,7 @@ Performance improvements
- Performance improvement in :class:`pandas.core.groupby.RollingGroupby` (:issue:`34052`)
- Performance improvement in arithmetic operations (sub, add, mul, div) for MultiIndex (:issue:`34297`)
- Performance improvement in `DataFrame[bool_indexer]` when `bool_indexer` is a list (:issue:`33924`)
- Significant performance improvement of :meth:`io.formats.style.Styler.render` with styles added with various ways such as :meth:`io.formats.style.Styler.apply`, :meth:`io.formats.style.Styler.applymap` or :meth:`io.formats.style.Styler.bar` (:issue:`19917`)

.. ---------------------------------------------------------------------------

Expand Down
14 changes: 9 additions & 5 deletions pandas/io/formats/style.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -561,11 +561,15 @@ def _update_ctx(self, attrs: DataFrame) -> None:
Whitespace shouldn't matter and the final trailing ';' shouldn't
matter.
"""
for row_label, v in attrs.iterrows():
for col_label, col in v.items():
i = self.index.get_indexer([row_label])[0]
j = self.columns.get_indexer([col_label])[0]
for pair in col.rstrip(";").split(";"):
rows = [(row_label, v) for row_label, v in attrs.iterrows()]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should use itertuples here (its actually much faster)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jeff - that's a good thing to know and I tried it but could not figure out doing the same thing with itertuples.
To get the col_label within the inner loop, I need to use ._fields(), getattr(), list slicing, etc to separate index, ... basically many extra steps. I am not sure how much we can save here.

However, it seems that .get_indexer is the one that caused much delay. So real solution should be something that will eliminate get_indexer entirely or some acceleration effort done on get_indexer.

I can think of one way to avoid get_indexer -- simply taking index & columns as list and use it to get integer index # of given label. However, I was not sure if I could do that safely because I am not sure all the labels given in attrs always matches that of self.index and self.columns. probably not.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh i see, you are doing get indexer once for the rows, you can do the same once for the columns. you can throw this in a dict {label -> int}. This will vastly speed up things.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you mean, each row will have same columns ..?
how about 4x4 table that has some attr assignments on column A, B on row 1, 2 but on column C, D only on row 3, 4 ..?
It would be great if the original author of this function could answer this -- or, are you may be?
I admit that I did the patch much relying on guesses based on common sense ( or my version of common sense :) )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it is certain that we really do not have to use get_indexer method, probably something this should work, outside the loops:
rowmap = { label: i for i, label in enumerate(self.index) }
colmap = { label: i for i, label in enumerate(self.columns) }

Copy link
Contributor Author

@jihwans jihwans Jun 25, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the exact code that worked for my app:

    def _update_ctx(self, attrs: DataFrame) -> None:
        coli = {k: i for i, k in enumerate(self.columns)}
        rowi = {k: i for i, k in enumerate(self.index)}
        for jj in range(len(attrs.columns)):
            cn = attrs.columns[jj]
            j = coli[cn]
            for rn, c in attrs[[cn]].itertuples():
                if not c: continue
                c = c.rstrip(";")
                if not c: continue
                i = rowi[rn]
                for pair in c.split(";"):
                    self.ctx[(i, j)].append(pair)

However, it outperform the current patch only slightly with benchmark.
for 1200 case, it took 3.02s where it took 3.13 with the current patch.
Plus we're not sure if this is okay with all occasions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you avoid the append? I think if this was a comprehension (or at least the last append) would be much better

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see how how it can be better.
Let's say we have a defaultdict out of comprehension with new pairs to be added to existing ctx. What should happen afterward is basically same things as this one, I would assume.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok i actually like your code above a little better, its very idiomatic and easy to understand. push it up and ping on green.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really don't feel safe with this code. It might break someone's code.
What if the attrs's column is specified different way other than plain column label?

row_idx = self.index.get_indexer([x[0] for x in rows])
for ii, row in enumerate(rows):
i = row_idx[ii]
cols = [(col_label, col) for col_label, col in row[1].items() if col]
col_idx = self.columns.get_indexer([x[0] for x in cols])
for jj, itm in enumerate(cols):
j = col_idx[jj]
for pair in itm[1].rstrip(";").split(";"):
self.ctx[(i, j)].append(pair)

def _copy(self, deepcopy: bool = False) -> "Styler":
Expand Down
13 changes: 3 additions & 10 deletions pandas/tests/io/formats/test_style.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -405,9 +405,10 @@ def f(x):

result = self.df.style.where(f, style1)._compute().ctx
expected = {
(r, c): [style1 if f(self.df.loc[row, col]) else ""]
(r, c): [style1]
for r, row in enumerate(self.df.index)
for c, col in enumerate(self.df.columns)
if f(self.df.loc[row, col])
}
assert result == expected

Expand Down Expand Up @@ -966,7 +967,6 @@ def test_bar_align_mid_nans(self):
"transparent 25.0%, #d65f5f 25.0%, "
"#d65f5f 50.0%, transparent 50.0%)",
],
(1, 0): [""],
(0, 1): [
"width: 10em",
" height: 80%",
Expand Down Expand Up @@ -994,7 +994,6 @@ def test_bar_align_zero_nans(self):
"transparent 50.0%, #d65f5f 50.0%, "
"#d65f5f 75.0%, transparent 75.0%)",
],
(1, 0): [""],
(0, 1): [
"width: 10em",
" height: 80%",
Expand Down Expand Up @@ -1091,7 +1090,7 @@ def test_format_with_bad_na_rep(self):
def test_highlight_null(self, null_color="red"):
df = pd.DataFrame({"A": [0, np.nan]})
result = df.style.highlight_null()._compute().ctx
expected = {(0, 0): [""], (1, 0): ["background-color: red"]}
expected = {(1, 0): ["background-color: red"]}
assert result == expected

def test_highlight_null_subset(self):
Expand All @@ -1104,9 +1103,7 @@ def test_highlight_null_subset(self):
.ctx
)
expected = {
(0, 0): [""],
(1, 0): ["background-color: red"],
(0, 1): [""],
(1, 1): ["background-color: green"],
}
assert result == expected
Expand Down Expand Up @@ -1219,17 +1216,13 @@ def test_highlight_max(self):
expected = {
(1, 0): ["background-color: yellow"],
(1, 1): ["background-color: yellow"],
(0, 1): [""],
(0, 0): [""],
}
assert result == expected

result = getattr(df.style, attr)(axis=1)._compute().ctx
expected = {
(0, 1): ["background-color: yellow"],
(1, 1): ["background-color: yellow"],
(0, 0): [""],
(1, 0): [""],
}
assert result == expected

Expand Down