🐛 useSkiaFrameProcessor shows black screen with or without frame.render() but useFrameProcessor does not

[ElianMalessy]

What's happening?

When i use the "useSkiaFrameProcessor" even though I have frame.render(), what is rendered is a black screen, instead of the camera feed. When I have the normal "useFrameProcessor" I do not have this issue, and it shows the camera feed. I can draw onto the screen with skia with no problem, and if I do frame.render(paint), that color takes up the whole screen.

Other info:
"react-native-reanimated": "~3.10.1"
"react-native-worklets-core": "^1.3.3",

This is my babel.config.js:

module.exports = function (api) {
  api.cache(true);
  return {
    presets: [
      ["babel-preset-expo", { jsxImportSource: "nativewind" }],
      "nativewind/babel",
    ],
    
    plugins: [
      ['react-native-worklets-core/plugin'],
      ['react-native-reanimated/plugin']
    ],
  };
};

Reproduceable Code

const device = useCameraDevice(position);
const pixelFormat = Platform.OS === 'ios' ? 'rgb' : 'yuv';
const format = useMemo(
    () => (device != null ? getBestFormat(device, 720, 1000) : undefined),
    [device],
);

const frameProcessor = useSkiaFrameProcessor(
    frame => {
      'worklet';
      frame.render();
    },
    [plugin, paint],
);


return (
    <View style={styles.container} onTouchEnd={flipCamera}>
      <StatusBar barStyle="light-content" />
      {!hasPermission && <Text style={styles.text}>No Camera Permission.</Text>}
      {hasPermission && device != null && (
        <Camera
          style={StyleSheet.absoluteFill}
          device={device}
          isActive={true}
          frameProcessor={frameProcessor}
          format={format}
          pixelFormat={pixelFormat}
        />
      )}
    </View>
);

Relevant log output

06-29 03:22:32.254  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.255  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:32.270  2169  3141 I CameraView: invokeOnAverageFpsChanged(30.211480362537763)
06-29 03:22:32.317  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.317  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:32.353  2169  2169 D EGL_emulation: app_time_stats: avg=33.12ms min=10.70ms max=67.84ms count=31
06-29 03:22:32.363  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.363  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.421  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.431  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.431  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.431  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.431  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.432  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.432  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.432  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.433  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.433  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.433  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.433  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.433  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.435  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.435  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.435  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.435  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.435  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.436  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.436  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.436  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.436  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.437  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.437  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.439  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.439  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.440  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.440  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.440  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.495  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.495  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:32.513  2169  2396 D EGL_emulation: app_time_stats: avg=30.25ms min=1.78ms max=81.17ms count=30
06-29 03:22:32.757  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.758  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:32.933  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.933  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:33.271  2169  3141 I CameraView: invokeOnAverageFpsChanged(30.333670374115265)

Camera Device

{
  "formats": [],
  "sensorOrientation": "landscape-left",
  "hardwareLevel": "limited",
  "maxZoom": 1,
  "minZoom": 1,
  "maxExposure": 6,
  "supportsLowLightBoost": false,
  "neutralZoom": 1,
  "physicalDevices": [
    "ultra-wide-angle-camera"
  ],
  "supportsFocus": true,
  "supportsRawCapture": false,
  "isMultiCam": false,
  "minFocusDistance": 999.999985098839,
  "minExposure": -6,
  "name": "10 (BACK) androidx.camera.camera2",
  "hasFlash": false,
  "hasTorch": false,
  "position": "back",
  "id": "10"
}

Device

Pixel 8 emulator API 35

VisionCamera Version

4.3.2

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

• I am using Expo
• I have enabled Frame Processors (react-native-worklets-core)
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar issues in this repository and found none.

[maintenance-hans]

Guten Tag, Hans here.

Note

New features, bugfixes, updates and other improvements are all handled mostly by @mrousavy in his free time.
To support @mrousavy, please consider 💖 sponsoring him on GitHub 💖.
Sponsored issues will be prioritized.

[ElianMalessy]

This is the result I get from frame.__skImage.encodeToBase64(), clearly just a completely black image:

iVBORw0KGgoAAAANSUhEUgAABQAAAALQCAIAAABAH0oBAAAAA3NCSVQICAjb4U/gAAAKjklEQVR4nO3BMQEAAADCoPVP7W8GoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIA3NUYAAd1QgUwAAAAASUVORK5CYII=

[mrousavy]

Did you try pixelFormat="rgb"?

[ElianMalessy]

When I do that, I get "Frame Processor Error: Exception in HostFunction: java.lang.Error: Frame has invalid PixelFormat! Only YUV_420_888 is supported. Did you set pixelFormat="yuv"?, js engine: VisionCamera"

[mrousavy]

This error is not thrown in VisionCamera or Skia.

I'm assuming you are using the resize plugin? I don't see you mentioning that in your issue though.

[ElianMalessy]

Sorry about that, I was using the resize plugin. I've removed it an am no longer getting an error, but my screen is still black.

[WillyGV]

Hello, can you try to do this?
https://github.com/mrousavy/react-native-vision-camera/issues/2951#issuecomment-2229539912
but without mirror mode.
to know if the error is for the same reason.

[steq28]

same here

[RayanMoarkech]

I have the same issue here too. Nothing worked. I tried to change the frame.render() position, tried all the attributes of Camera... Once I change to useFrameProcessor it works perfectly but of course without the Skia features.

[RayanMoarkech]

It is working for me now with the new versions release

[usamaabutt]

Hi @RayanMoarkech with what version release?

[dztran]

Hello all,
I am new to Vision camera and have been going through the features the last 4wks. I ran into this issue also with the latest version of all the packages. It works fine with useFrameProcessor but black screen with useSkiaFrameProcessor.

I know this issue is old, but anyone has any idea or clue to look for.
Appreciated

ping @RayanMoarkech to see if you have any pointer in the version that work for you.