Expose low level IME interactions

[justinmc]

Use case

Currently, it's very hard for Flutter developers to make changes to Flutter's text input experience without losing access to Flutter's entire text input stack. Specifically, if I want to [example]

This problem arises because in each of Flutter's platform embedders, Flutter abstracts differences in the platforms' text input APIs and communicates with the framework using common platform channel methods (text_input.dart). For Flutter app developers, there is no access to the platform's unique nuances if they aren't covered in the common platform channel methods. It's not possible to access them by writing a plugin without rewriting all the rest of Flutter's text input code as well.

Proposal

Instead of abstracting platform differences in the engine, the platform channel calls could mirror the native platform APIs. Any abstraction could be done in the framework, on top of these verbatim platform channel methods. Then, Flutter app developers could potentially access these calls themselves while still using Flutter's existing text editing stack.

Open questions

• Would it be possible to do this with FFI, or even FFIgen?
• Flutter suffers from a distributed system problem in that the framework and engine both maintain a copy of the current text input state, both sides can make changes to this, and it must be synced between them asynchronously. Does this new approach make this problem worse or better?

Creating this issue in collaboration with @matthew-carroll from a discussion in #150068.

[matthew-carroll]

That's close, but there are some details here that I would express differently to make clear the problem and the proposed approach.

It's impossible

it's very hard for Flutter developers to make changes to Flutter's text input experience

As far as I know, it's not "hard" to do that. It's "impossible". Flutter installs itself as the singular IME delegate. Using a custom delegate means removing Flutter from all IME signals. That effectively means that Flutter is no longer usable for such a situation.

The current approach violates Flutter's principles

Flutter states explicitly that it's not a "lowest common denominator solution"

https://docs.flutter.dev/resources/faq#can-i-access-platform-services-and-apis-like-sensors-and-local-storage

I believe the "lowest common denominator" stance used to exist in a top-level principles/values document, but I can't find it right now. Nonetheless, many Flutter developers will remember this principle being stated many times over the years.

Flutter's IME integration is the epitome of a "lowest common denominator" solution. It's a generalization across all platform IME operations, and represents only a partial set of capabilities on a per-platform basis.

The problems

Given the above details, we arrive at the following problems:

• Existing functionality is only partial - there are any number of IME capabilities that are currently unavailable to Flutter developers.
• There's no dependable stance from the Flutter organization that would inform developers when/if any particular IME capability will be added.
• There's no consistent and repeatable approach to integrating more/new platform IME APIs into the Flutter IME API surface.

Historical examples

In addition to future facing missing pieces, it's worth noting the historical impact of this approach.

• No delta support: For most of Flutter's history there was no delta support at all. But for a happenstance interaction between myself, @justinmc, and @Renzo-Olivares in relation to Super Editor, it's unclear when/if deltas would have been implemented. In the absence of deltas, Flutter was held out of serious consideration for any document editing UI.
• Wasn't clear where Scribble APIs belonged: Scribble APIs were added for iOS sometime after iOS itself gained support for them. When that happened, it wasn't clear where to put them. These were APIs that were defined by a single platform, and they certainly weren't as generic as something like "inserting text". Due to the lowest common denominator approach to IME, the addition of Scribble was more complicated and confusing that it should have been. Speaking of Scribble, @justinmc is now looking into similar functionality on Android and it's once again unclear what the story is to design that API surface.
• iOS context menu: Contributions are underway at the time of this writing to integrate new iOS context menu functionality. This effort is again plagued by questions about platform-specificity vs generic Flutter APIs. At the moment, this iOS-only implementation has platform-agnostic APIs. What will happen when Android needs something similar, but different?

Future looking examples

At some point I'll audit the platform IME APIs to find out just how many capabilities Flutter is missing. But just from the most recent WWDC, we know that a number of AI-related behaviors or coming to text editing. And it's probably not super simple stuff. The AI mechanism appears to have access to full document scopes and can edit multiple paragraphs of content. How will any of that make its way into Flutter under the current Flutter approach?

The only tractable approach

There is only one approach to IME integration that upholds Flutter's own principles, and solves the problems mentioned above.

Flutter must state a mission to replicate all platform IME APIs, from every support platform, in the Flutter API surface. This mission can be ongoing - it can be aspirational - but the mission needs to be declared.

Once the mission is declared, areas of the Flutter API can be carved out on a per-platform basis. This makes it crystal clear where new iOS, Android, Mac, Windows, Linux APIs are expected to live. Moreover, the names of those APIs will be expected to match the platform as close as possible, taking nearly all guesswork out of the process. It's simple porting.

The declared mission also clarifies to Flutter users what they can expect. They will know that at some point every IME API for every platform will be supported. They'll also be able to contribute those ports themselves without risking repeated arguments with Flutter team developers about whether exposing this API is an acceptable tradeoff.

Edit (June 30, 2024):

I found the lowest common denominator in the project guidelines:

Avoid the lowest common denominator
It is common for SDKs that target multiple platforms (or meta-platforms that themselves run on multiple platforms, like the Web) to provide APIs that work on all their target platforms. Unfortunately, this usually means that features that are unique to one platform or another are unavailable.

For Flutter, we want to avoid this by explicitly aiming to be the best way to develop for each platform individually. Our ability to be used cross- platform is secondary to our ability to be used on each platform. For example, TextInputAction has values that only make sense on some platforms. Similarly, our platform channel mechanism is designed to allow separate extensions to be created on each platform.

https://github.com/flutter/flutter/blob/master/docs/contributing/Style-guide-for-Flutter-repo.md#avoid-the-lowest-common-denominator

[knopp]

I love the idea of API per platform. I think that would be lot better than trying to shoehorn all platform features into single class.

Would it be possible to do this with FFI, or even FFIgen?

This would possibly involve reimplementing platform channels in FFI. The sad reality is that having UI separate thread makes interacting with platform APIs through FFI very awkward. Unless we can get rid of UI thread (unlikely, but would very much like to see that one day), we need to make the hop to platform thread somehow. Platform channels do that for you.

Distributed system problem

Yes, this really is annoying, and it's not the only instance where Flutter needs to replicate state (ie keyboard events or accessibility). The biggest reason for this is separate UI thread, and the reason for having one still eludes me to be honest. I can't see how it improves performance since the platform thread is very much idle (apart from a tiny amount of time during event processing and compositor commit). The objective of keeping the platform thread idle (why?) if costing us a huge amount of complexity all over Flutter.

Let's take macOS for example. There are number of synchronous text input calls for which we need to compute answer asynchronously. So we have three options:

• Sync the state with platform thread from UI thread. We do that right now and it's a constant source of problems. Some things like custom complex bindings in DefaultKeyBinding.dict are unfixable and will be broken forever.

• Use undocumented asynchronous API that WebKit is using. This can get problematic when deploying to AppStore, since Apple scans for a subset of undocumented API. There is precedent of Chromium using some undocumented API even in the AppStore safe build, so it's aways bit of a guessing game to figure out which APIs will be the problematic ones.

• Pause platform thread until you get reply from UI thread. This is like playing with knives a bit. I do this in bunch of places in super_native_extensions, it's doable if done carefully, there is even precedent for it in iOS embedder for keyboard event handling. Especially if the dart method is not async it should be fairly safe since there is nothing in synchronous dart code that can block on platform thread. (asynchronous dart code can block on platform thread, but I think here we would only need synchronous calls). I'm not sure how well platform channels handle situations where the isolate gets shut down before dart side replies, this might in theory leave the platform thread blocked forever if not handled well. It's should be fixable. For super_native_extensions i wrote my own version of platform channels and one reason for it to ensure predictable behavior when the port refuses message or isolate gets shut down. And also to be able to receive messages when dispatch queue is blocked. We would probably need to keep pumping the CFRunLoop instead.

• Get rid of UI thread, run Dart code on platform thread. This is more or less pipe dream. I don't think it would meaningfully regress performance (since platform thread is mostly idle and doing too much work on UI thread will drop frames anyway), except for when doing thread merging, but the solution there is to not do thread merging (adopt the macOS approach for platform views everywhere). Still, this would be huge breaking change, very unlikely to happen. One can dream thoguh.

Overall I think that the third approach (holding the platform thread until the reply is available) is worth considering on both macOS and iOS, but it does need to be done carefully.

[jonahwilliams]

FYI @knopp I'm trying to collect sets of issues that would be improved by merged platform/ui thread

[knopp]

FYI @knopp I'm trying to collect sets of issues that would be improved by merged platform/ui thread

I was thinking about opening an issue for this, is there maybe one already?

[jonahwilliams]

There is not, go for it 😄

[knopp]

Put something together in #150525

[dcharkes]

Drive by comment:

• Would it be possible to do this with FFI, or even FFIgen?
• Flutter suffers from a distributed system problem in that the framework and engine both maintain a copy of the current text input state, both sides can make changes to this, and it must be synced between them asynchronously. Does this new approach make this problem worse or better?

When using FFI, we can avoid the copy of the current text by keeping only a single copy, in native memory. All operations of Dart on the text would be mutating the native memory directly. You'd only use synchronous FFI calls and only direct memory access via pointer[int]. (This is the pattern that we usually use when using FFI, I don't have enough background knowledge about text input in Flutter to know if that's feasible.)

Edit: Ah yes, the other pattern is keeping a single copy in Dart and accessing it via NativeCallable.isolateLocal (provided you're guaranteed that you're on the right isolate, which is true in the platform isolate in Flutter).