Layer::Diff should use new LayerStateStack or DisplayListMatrixClipTracker objects

[flar]

The Layer::Diff method maintains tree state to perform its work. The other 2 recursion methods Layer::Preroll and Layer::Paint were recently modified to use a new LayerStateStack mechanism to manage their state as they recurse through a layer tree. The Diff method tracks state similarly, but doesn't necessarily use all of the mechanisms that the LayerStateStack object provides. Minimally, though, it tracks transform and clip which can now be maintained via the DisplayListMatrixClipTracker object which provides better support for 4x4 matrices and difference clipping.

Diff should minimally use at least DisplayListMatrixClipTracker for better (and future-proofed) clip/matrix tracking, and look at the full LayerStateStack to see if it provides needed functionality with minimal overhead so that all 3 recursion methods have essentially consistent, if not identical, state management code in them.

[flar]

@knopp thoughts?

[knopp]

I agree that all 3 should be consistent. I need to take a good look at the new classes to see how they're meant to be used.

Couple of things:

• Preroll and Paint already aren't consistent, because paint applies integralTransform and preroll doesn't. This complicates diff context (because it has to account for the difference). Can we fix this in layers?
• There is more state tracked in stack in the diff context, we still need to put it somewhere. But perhaps it can be separate from clip and transform.

[flar]

I noticed that too. We should fix the difference in the integralTransform in a separate PR for bisecting purposes if anything goes wrong.

With regard to the extra state. Preroll and Paint don't use it for all state that they track, just clip, transform, and mutators (opacity, filters)

[flar]

I think Diff probably wants to apply all of the bounds mutators in the "state stack" to a given layer's bounds? That feature is not present in the current LSS.

Moving to DisplayListMatrixClipTracker might be a simple short term change that would enable us to embrace full SkM44's in TransformLayer, and then moving over to LSS for everything else might be a lower priority longer term goal. But if it's not too bad, then having LSS manage it all for consistency would be the best end state.

[flar]

It would also be interesting to merge Diff/Preroll since both of them compute bounds, but in just slightly different ways. (Preroll computes local bounds, Diff computes destination bounds).

[knopp]

DiffContext result is used in preroll to cull layers. Also diff doesn't need to walk the entire tree (it skips retained layers).

[knopp]

I'll try applying integralTransform to preroll tomorrow to see how it works and whether i can remove the workaround from diff context. That should make it easier to move it to DisplayListMatrixClipTracker.

[flar]

DiffContext result is used in preroll to cull layers. Also diff doesn't need to walk the entire tree (it skips retained layers).

I don't think this is true. Preroll doesn't look at diff context info at all AFAIK. It produces local bounds for culling layers during Paint, but it doesn't take info from Diff.

[knopp]

https://github.com/flutter/engine/blob/d85707af098739a002489dce858d436d3246e69b/flow/compositor_context.cc#L128-L134

  std::optional<SkRect> clip_rect =
      frame_damage
          ? frame_damage->ComputeClipRect(layer_tree, !ignore_raster_cache)
          : std::nullopt;


  bool root_needs_readback = layer_tree.Preroll(
      *this, ignore_raster_cache, clip_rect ? *clip_rect : kGiantRect);

First clip_rect is being computed (using the DiffContext), then it is passed to LayerTree::Preroll as cull_rect.

[flar]

First clip_rect is being computed, then it is passed to LayerTree::Preroll as cull_rect.

It does maintain an item called "cull_rect", but it only uses it for determining whether it matters to generate a cache entry, not for culling painting or sub-trees in Preroll.

[flar]

If you look for calls to the LSS::content_culled method, it only happens from Paint methods or from RasterCacheItem methods. So, we do "cull" RasterCacheItems, in the sense that we mark them as not useful, but I wasn't considering that culling because it doesn't abort a sub-tree. So, ... terminology ...

[knopp]

I think originally it used to affect needs_painting(), that was later changed to query actual clip from canvas. So maybe the cull rect in preroll it's less useful now.

[flar]

Also, bdfLayer sets its bounds to the (local_)cull_rect.