[Android] Flutter 3.19 beta - Widget freezes on resize

[timbotimbo]

Is there an existing issue for this?

• I have searched the existing issues

Unity version

• I am using Unity 2022.3 LTS

Common issues

• I have checked the common issues

Flutter version

3.19.0-0.4.pre

Description

@jamesncl
Given that you seem to be more experienced in android development, I hope you have some insight.

I was testing the flutter 3.19 0.4-pre beta, to see if the fix for #12 was included.

However I found a new bug that affects the widget on resizing.
Run your example, and trigger the keyboard to open, or the screen to rotate.

The widget is now frozen, with this error spamming the console:
I/ImageReaderSurfaceProducer(16480): Skipped frame because resize is in flight.

The cause seems to be this PR flutter/engine#47946.

• In short: Flutter used to rebuild the Virtual Display when it was resized, but now it calls a resize function on Android >= 12 instead.

 @TargetApi(31)
  private void resize31(
      View embeddedView, int width, int height, final Runnable onNewSizeFrameAvailable) {
    renderTarget.resize(width, height);
    // On Android versions 31+ resizing of a Virtual Display's Presentation is natively supported.
    virtualDisplay.resize(width, height, densityDpi);
    embeddedView.postDelayed(onNewSizeFrameAvailable, 0);
  }

Which at first glance seems great as it might solve some random bugs of unity becoming invisible on keyboard/screen rotations.
However a resize now simply breaks the widget as either Unity or this widget don't seem to handle this resize properly.

Given that the PR calls a resize function, maybe there is some kind of event or callback that we can listen to?

Video

These videos use the reproduction project I used for the other issue, but the difference doesn't really matter here.

3.16.5	3.19.0-0.4-pre
3.16.mp4	3.19.mp4

Minimum reproducible example (MRE)

• Enable autorotation and all screen orientations in the player settings of the example project.

• Run in Flutter and rotate the screen.
  This works fine in Flutter 3.13, 3.16 etc., but not in the new 3.19 beta versions.

• Alternatively, put a textfield over the unity widget and tap it to open the keyboard.
  I added a branch with a textfield to my reproduction repo which doesn't need unity to be installed.

What platforms are you seeing the problem on?

Android

Devices

Samsung Galaxy S20FE, Android 13. Likely any other device on android >= 12.

Anything else?

Flutter should release a new version (3.19 or 3.20?) this month.
If this makes it in, this widget is broken on both 3.16 and the next version.

[timbotimbo]

I just tested the flutter master branch at tag 3.20.0-0.0-pre, and it doesn't trigger this error.
But it doesn't resize to the proper size like it did before.

Will need to look deeper into this to figure out what is happening.

[jamesncl]

Thanks for the detailed report!

My first thought is that visually, this reminds me of a problem I ran into where Unity rendering would occasionally appear to freeze - Unity itself was still active, because you could still pass and receive messages between Flutter and Unity, and game time continued to pass, but the rendering itself stopped, resulting in that stretched look in your screen recording.

The only workaround I could find was [a slightly hacky pause-and-resume of Unity], which I had to do whenever the Android activity paused and resumed see ResumeUnityOnActivityResume.kt and also when Unity is detached from a PlatformView and attached to a new one (see UnityViewStack.kt).

So I wonder if we now need to do the same on response to this new onResize event. Unfortunately I don't currently have any Android 12 devices to test this with, so I'll order one today and do some experimenting hopefully later this week.

[timbotimbo]

This should also work on an android emulator if by any chance you have a mac with apple silicon cpu.
Those emulators run on arm natively and have worked for me so far.

I think it is a different error because of the console spam, and because a later change seems to partially fix this.

The master branch fixes the freezing with a commit about preventing frames being out of sync.
You can see the cube still spin at the very right edge here, it just resized to the wrong size.

Screen_Recording_20240205_160505.mp4

I will try to get a full issue in the flutter repo soon.

[timbotimbo]

More context flutter/flutter#142952
The issue in the first comment is a flutter bug and can be ignored. However the video in the comment above this is the 'correct' new behaviour on the master branch.

On actions like a screen rotation or a keyboard open the VirtualDisplay gets resized instead of recreated. And Unity doesn't seem to respond to that properly, it resizes to completely different dimensions instead.

I've tried several Unity versions from Unity 2019.4.40 up to 2022.3.19.
I've also tried various Player setttings related to screens and resolution like resizable window, but they don't make a difference.

The onConfigurationChanged function does seem to get triggered more often now.
I've been toying with triggering something on a resize there, but I haven't gotten anything useful yet. (I'm not an android dev, so it is kind of a deep dive here)

[jamesncl]

So, just to confirm, are you now testing this using Flutter 3.19.0 stable? Also, can I check what you have set for the android:configChanges property of the main activity in your app's AndroidManifest.xml? Just to make sure we are testing the same configuration.

One thing that might be useful, just in case you weren't aware, is the Activity which is bundled with the default Unity export for Android - you can find it in the Unity Editor folder. For example on Mac (assuming you have Anroid build tools installed) it might be at

/Applications/Unity/Hub/Editor/2022.3.13f1/PlaybackEngines/AndroidPlayer/Source/com/unity3d/player/UnityPlayerActivity.java

This is the main activity that Unity is 'supposed' to run in when it's being used as a library (which we're not using, because we're having to make Unity work inside a custom FlutterActivity - that is what the FakeUnityPlayerActivity is for).

So my first thought is looking at UnityPlayerActivity.java is there is this:

    // This ensures the layout will be correct.
    @Override public void onConfigurationChanged(Configuration newConfig)
    {
        super.onConfigurationChanged(newConfig);
        mUnityPlayer.configurationChanged(newConfig);
    }

Although I didn't think that implementing this in FakeUnityPlayerActivity would be of any purpose, because I thought that onConfigurationChanged was NOT called on orientation change when orientation is included in the android:configChanges property in AndroidManifest.xml (which I think is required - because otherwise the main Activity gets destroyed and recreated on orientation change, which then completely kills Unity). Does that match your understanding? Because if I'm not understanding things correctly, maybe we need to override onConfigurationChanged in FakeUnityPlayerActivity to call configurationChanged on mUnityPlayer. Sorry I haven't tried this myself yet, but to be honest I'm struggling a bit with a chronic health condition which is limiting my ability to work, so I really appreciate your input.

[timbotimbo]

(I deleted some comments above that ended up being useless)

Update:
All issues I mentioned above turned out to be on the Flutter side.

• 3.19 freeze/errors
  Flutter bug that was fixed on master just before the 3.20 versions.
  I don't expect a hotfix release for 3.19
• 3.20 beta resize issue
  This was just fixed on the Flutter master branch.

Looks like we need to wait ~3 months for the next Flutter (3.23?) or use a future 3.20 beta release.

[jamesncl]

This is going to become a problem from 1st May for iOS, when the new required reasons for APIs kicks in - this affects quite a lot of common flutter plugins (and Unity framework). Most have now included privacy manifests in the latest versions, but many (eg share_plus) now require Flutter 3.16 or 3.19.

It's unclear if you can just continue to use older versions of these libs and provide an over-arching privacy manifest in your app, to cover the API usages in the libs /pods your app uses, because the docs talk about "SDKs that require a privacy manifest and signature" which suggests to me that the lib itself has to provide the privacy manifest?

Would be interested to hear your thoughts on this. I'm wondering if I'm going to be forced into using a beta version of flutter 3.20 to solve this issue with flutter_embed_unity AND be able to pass iOS app review after May 1st, which I'm very reluctant to do!

[timbotimbo]

The flutter framework itself only has the privacy manifest and code signing in 3.19+.
For the manifest you could get away with manually adding one, but the signing can't be helped.
Knowing apple i'm honestly expecting everything below 3.19 to fail on ios publishing.

Unity does seem to have the manifests in order in the latest releases.

Regarding this android issue, the platform view works and can resize on the current flutter master.

However, they introduced a new issue on master where it breaks on any app resume.
I got them to fix it, but that fix got reverted yesterday....

So the only stable approach on android so far is hybrid composition, although that has some downsides.

[jamesncl]

Thank you for raising the app resume issue, another really well-researched bug report!

Can I ask what do you mean by 'hybrid composition'? I've considered branching my projects to have an iOS version using Flutter 3.19 (to pass app store review) and Android version using Flutter 3.13 (to avoid this issue), but as a last resort

[timbotimbo]

As long as the flutter deprecations and package versions aren't too different, I think separate branches are the cleanest option.

With hybrid composition I mean a different platformview mode. The other unity plugin allows you to switch from androidView to initExpensiveAndroidView.
That mode has none of these glitches, but plenty of other minor ones.
See juicycleff/flutter-unity-view-widget#929 (comment)

[jamesncl]

Ah right, thank you - yes I had discounted AndroidViewSurface as an option due to the issues you reported on

[timbotimbo]

Both the resize and resume issues seem to be fixed in the new 3.22.0 release.