Support Texture widget

[cloudwebrtc]

We have ported the WebRTC plugin for iOS/Android, but there is no Texture widget in flutter-desktop-embedding, Texture is needed to support video rendering and video player development.

I can help with that. I assume c++ based for all desktops ?

…

On Fri, 13 Jul 2018, 09:12 湖北捷智云技术有限公司, ***@***.***> wrote: We have ported the WebRTC plugin <https://github.com/cloudwebrtc/flutter-webrtc> for iOS/Android, but there is no Texture widget in flutter-desktop-embedding, Texture is needed to support video rendering and video player development. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#107>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATuCwhiMbSeeT4752YrP7Alxv3E60xkUks5uGEhAgaJpZM4VOaKo> .

[cloudwebrtc]

Yes, use C++ to write the plugins for all desktops.

[stuartmorgan-g]

From a quick skim of the existing implementations it looks like Texture implementation requires access to engine APIs that aren't currently exposed via the embedding API.

@chinmaygarde Is that accurate?

Not having the Texture implementation is a bit of s show stopper for me too. I want to use the Texture to push 2D and 3D graphics to the GUI.

Polite hint to @chinmaygarde. If you can comment on this ?

[chinmaygarde]

Apologies for the late reply.

Support for external textures to embedders is definitely something on the roadmap. Currently, external textures are handled very differently on iOS (CoreVideo) and Android (GL_OES_EGL_image_external) and so exposing the same to embedders is not straightforward. It would be great if we could collaborate on figuring out a few things first. For example, should the embedder API try to create an abstraction over textures on the various platforms or support platform specific APIs? In the specific case of the WebRTC plugin linked, I suspect the easiest way for the platform to express its textures would be in the form of CoreVideo pixel buffers. OTOH, on Linux, I suspect EGL images would be better. Should the embedder API support both platform specific APIs? Or, should the engine only deal with plain old GL_TEXTURE_2D objects and have the embedder render into the same. That would require sharing of the OpenGL context used by Flutter with the embedder or creating another context within the same sharegroup.

@cloudwebrtc: I think exposing the current support for CoreVideo and EGL_Image_External as-is would be the easiest way to support your use case.

@gedw99: How are you going to generate and update the textures with the graphics you mentioned? Are you going to push new textures (from a pool) to Flutter each frame or do you want to update the same texture? Are there threading restrictions to the way you are going to be generating your textures? Can you provide a reduced test case for how you want to generate your textures? That would be extremely helpful to try to pin down an API that works for the broadest audience. Thanks!

Hey @chinmaygarde

Answers to the questions

How are you going to generate and update the textures with the graphics you mentioned?

• generated using opengl buffers

Are you going to push new textures (from a pool) to Flutter each frame or do you want to update the same texture?

• yes new textures

Are there threading restrictions to the way you are going to be generating your textures?

• I will be computin the textures from many threads back to a singleton.

Can you provide a reduced test case for how you want to generate your textures?

• Not yet. Its most a mish mash of test use cases.

Some Background:
I am building things like Inkscape for 2D editing or Blender for 3d editing if you need a well known analogy. At the end of the day they output a raster image to the Texture.
But whats interesting is that everything i am building is interactive in that there is a heavy use of the Flutter gestures to determine the users intent. The User interactions hints are rendering at the Dart / Flutter level - for example the cross hairs when moving an object in 3D or the orbit tool when doing a rotate or scale transformation of a 3D object. Or even when doing basic hit testing in 3D. The mutation is sent back to the backend and it produces a new texture.
Hopefully the background helps. I learnt a fair bit form looking at the Cloud Webrtc code too, although very different between IOS and Android. But they have a sink and a channel and the use cases i just describes have the same.

[chinmaygarde]

Engine patch with example embedder implementation has been posted flutter/engine#7087.

[cloudwebrtc]

@chinmaygarde I used your branch to create a webrtc plugin example that seems to work fine, thank you.
@stuartmorgan @gedw99
I modified some code:
1, FLETextureDelegate needs to be placed in the FLEViewController.h public declaration in order to provide it to the external framework to link it.
https://github.com/cloudwebrtc/flutter-desktop-embedding/blob/flutter-webrtc/library/macos/FLEViewController.h#L21
2. I added a FlutterEventChannel object to adapt the listen method of the dart layer EventChannel.
https://github.com/cloudwebrtc/flutter-desktop-embedding/blob/flutter-webrtc/plugins/flutter_webrtc/macos/FlutterEventChannel.h

/**
 * An event sink callback.
 *
 * @param event The event.
 */
typedef void (^FlutterEventSink)(id _Nullable event);

extern NSObject const* FlutterEndOfEventStream;

@protocol FlutterStreamHandler

- (FLEMethodError* _Nullable)onListenWithArguments:(id _Nullable)arguments
                                       eventSink:(FlutterEventSink)events;

- (FLEMethodError* _Nullable)onCancelWithArguments:(id _Nullable)arguments;

@end

@interface FlutterEventChannel : NSObject

+ (instancetype)eventChannelWithName:(NSString*)name
                     binaryMessenger:(NSObject<FLEBinaryMessenger>*)messenger;

- (void)setStreamHandler:(NSObject<FlutterStreamHandler>* _Nullable)handler;
@end

3. added a FlutterTextureRegistry class, adapted to the iOS-like Texture interface.
   https://github.com/cloudwebrtc/flutter-desktop-embedding/blob/flutter-webrtc/plugins/flutter_webrtc/macos/FlutterTextureRegistry.h

@protocol FlutterTexture<NSObject>
- (CVPixelBufferRef _Nullable)copyPixelBuffer;
@end

@protocol FlutterTextureRegistry<NSObject>
- (int64_t)registerTexture:(NSObject<FlutterTexture> *)texture;
- (void)textureFrameAvailable:(int64_t)textureId;
- (void)unregisterTexture:(int64_t)textureId;
@end

NS_ASSUME_NONNULL_BEGIN

@interface FLETextureRegister : NSObject<FlutterTextureRegistry, FLETextureDelegate>
@property(nullable, weak) FLEViewController *controller;
-(instancetype) initWithController:(FLEViewController *)controller;
@end

NS_ASSUME_NONNULL_END

Should I add FlutterEventChannel and FlutterTextureRegistry to FlutterEmbedderMac.framework?

Thanks again, the Linux/Win plugin should be easier to port, just copy the FlutterOpenGLTexture object and provide the GL_RGBA8 buffer.

For testing please refer to:flutter-webrtc/flutter-webrtc#37 (comment)

screenshots

[stuartmorgan-g]

Should I add FlutterEventChannel and FlutterTextureRegistry to FlutterEmbedderMac.framework?

I'd be happy to accept a patch that adds FLEEventChannel (the naming convention I've been using for the macOS code in this project is to use FLE as a prefix, replacing Flutter when the iOS class starts with Flutter) and related classes to the framework!

It'll need to wait to land until the engine change lands though, and then the texture APIs will need to be added to the macOS framework (which will likely have some changes from what's in that branch), but feel free to start the PR whenever you like and it go through review in the meantime.

[cloudwebrtc]

Okay, I created a PR for FELEventChannel. #174

[chinmaygarde]

Engine support landed in flutter/engine#7087