Skip to content

Gstreamer does not use HW encoding #224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
radeksalomon opened this issue Feb 12, 2022 · 8 comments
Closed

Gstreamer does not use HW encoding #224

radeksalomon opened this issue Feb 12, 2022 · 8 comments
Labels
bug Something isn't working p: gstreamer video player Issues related to the gstreamer video player plugin.

Comments

@radeksalomon
Copy link

radeksalomon commented Feb 12, 2022

Regarding the feature/gstreamer-video-player

Arch: rpi-debian-bullseye last updates
hw: rpi3 b+ and CM4
Full KMS dtoverlay=vc4-kms-v3d

No difference for flutter debug and release mode.

Gstreamer is not using hw encoding so every core is around 40 %. Please see the screenshot and attached log. It falls to sw encoding immediately after start.

With FKMS we got same results. Gstreamer is not using hw decoding.

We tried also Gstreamer directly and it was able to play the video using hw decoding (CPU was around 5 %).

Screenshot 2022-02-12 at 14 09 50

[flutter-pi] WARNING: display didn't provide valid physical dimensions.
             The device-pixel ratio will default to 1.0, which may not be the fitting device-pixel ratio for your display.
===================================
display mode:
  resolution: 1920 x 1080
  refresh rate: 60Hz
  physical size: 0mm x 0mm
  flutter device pixel ratio: 1.000000
===================================
EGL information:
  version: 1.4
  vendor: "Mesa Project"
  client extensions: "EGL_EXT_device_base EGL_EXT_device_enumeration EGL_EXT_device_query EGL_EXT_platform_base EGL_KHR_client_get_all_proc_addresses EGL_EXT_client_extensions EGL_KHR_debug EGL_EXT_platform_device EGL_EXT_platform_wayland EGL_KHR_platform_wayland EGL_EXT_platform_x11 EGL_KHR_platform_x11 EGL_MESA_platform_gbm EGL_KHR_platform_gbm EGL_MESA_platform_surfaceless"
  display extensions: "EGL_ANDROID_blob_cache EGL_EXT_buffer_age EGL_EXT_image_dma_buf_import EGL_EXT_image_dma_buf_import_modifiers EGL_KHR_cl_event2 EGL_KHR_config_attribs EGL_KHR_create_context EGL_KHR_create_context_no_error EGL_KHR_fence_sync EGL_KHR_get_all_proc_addresses EGL_KHR_gl_colorspace EGL_KHR_gl_renderbuffer_image EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_3D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_image EGL_KHR_image_base EGL_KHR_image_pixmap EGL_KHR_no_config_context EGL_KHR_reusable_sync EGL_KHR_surfaceless_context EGL_EXT_pixel_format_float EGL_KHR_wait_sync EGL_MESA_configless_context EGL_MESA_drm_image EGL_MESA_image_dma_buf_export EGL_MESA_query_driver EGL_WL_bind_wayland_display "
===================================
OpenGL ES information:
  version: "OpenGL ES 3.1 Mesa 20.3.5"
  shading language version: "OpenGL ES GLSL ES 3.10"
  vendor: "Broadcom"
  renderer: "V3D 4.2"
  extensions: "GL_EXT_blend_minmax GL_EXT_multi_draw_arrays GL_EXT_texture_compression_s3tc GL_EXT_texture_compression_dxt1 GL_EXT_texture_format_BGRA8888 GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth24 GL_OES_element_index_uint GL_OES_fbo_render_mipmap GL_OES_mapbuffer GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_stencil8 GL_OES_texture_3D GL_OES_texture_float GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_half_float GL_EXT_draw_instanced GL_EXT_texture_sRGB_decode GL_OES_EGL_image GL_OES_depth_texture GL_OES_packed_depth_stencil GL_EXT_texture_type_2_10_10_10_REV GL_OES_get_program_binary GL_APPLE_texture_max_level GL_EXT_discard_framebuffer GL_EXT_read_format_bgra GL_EXT_frag_depth GL_NV_fbo_color_attachments GL_OES_EGL_image_external GL_OES_EGL_sync GL_OES_vertex_array_object GL_ANGLE_pack_reverse_row_order GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_EXT_occlusion_query_boolean GL_EXT_texture_rg GL_EXT_unpack_subimage GL_NV_draw_buffers GL_NV_read_buffer GL_NV_read_depth GL_NV_read_depth_stencil GL_NV_read_stencil GL_EXT_draw_buffers GL_EXT_map_buffer_range GL_KHR_debug GL_KHR_texture_compression_astc_ldr GL_NV_pixel_buffer_object GL_OES_depth_texture_cube_map GL_OES_required_internalformat GL_OES_surfaceless_context GL_EXT_color_buffer_float GL_EXT_sRGB_write_control GL_EXT_separate_shader_objects GL_EXT_shader_implicit_conversions GL_EXT_shader_integer_mix GL_EXT_base_instance GL_EXT_compressed_ETC1_RGB8_sub_texture GL_EXT_draw_elements_base_vertex GL_EXT_primitive_bounding_box GL_EXT_shader_io_blocks GL_EXT_texture_border_clamp GL_KHR_context_flush_control GL_NV_image_formats GL_OES_draw_elements_base_vertex GL_OES_primitive_bounding_box GL_OES_shader_io_blocks GL_OES_texture_border_clamp GL_OES_texture_stencil8 GL_OES_texture_storage_multisample_2d_array GL_EXT_buffer_storage GL_EXT_float_blend GL_EXT_geometry_point_size GL_EXT_geometry_shader GL_KHR_no_error GL_KHR_texture_compression_astc_sliced_3d GL_OES_EGL_image_external_essl3 GL_OES_geometry_point_size GL_OES_geometry_shader GL_OES_shader_image_atomic GL_EXT_texture_compression_s3tc_srgb GL_MESA_shader_integer_functions GL_KHR_parallel_shader_compile GL_EXT_EGL_image_storage GL_MESA_framebuffer_flip_y GL_EXT_texture_query_lod "
===================================
[gstreamer video player plugin] on_initialize
[gstreamer video player plugin] respond success on_create
[gstreamer video player plugin] on_receive_evch
[compositor] Could not move cursor to front. Mouse cursor may be invisible. drmdev_plane_get_max_zpos_value: Invalid argument
[compositor] GPU does not supported the desired HW plane order.
             Some UI layers may be invisible.
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] gstreamer error: code: 7, domain: gst-stream-error-quark, msg: No valid frames decoded before end of stream (debug info: ../gst-libs/gst/video/gstvideodecoder.c(1246): gst_video_decoder_sink_event_default (): /GstPipeline:pipeline0/GstURIDecodeBin:src/GstDecodeBin:decodebin0/v4l2h264dec:v4l2h264dec0:
no valid frames found)
[gstreamer video player] Hardware decoder failed. Falling back to software decoding...
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player] Could not query buffering state. (gst_element_query)
[gstreamer video player plugin] Got video info: stream? no, w x h:  1920 x  1056, duration: 0:02:54.048000000
[compositor] Preferred framebuffer depth for hardware cursor is not supported by flutter-pi.

(flutter-pi:2726): GStreamer-CRITICAL **: 16:02:12.918: gst_event_new_seek: assertion 'rate != 0.0' failed

(flutter-pi:2726): GStreamer-CRITICAL **: 16:02:12.918: gst_element_send_event: assertion 'event != NULL' failed
[gstreamer video player] Could not set the new playback speed / playback position (speed: 0.000000, pos: 0:00:00.002608475).  ```

@radeksalomon
Copy link
Author

radeksalomon commented Feb 12, 2022

After some digging and testing it seems it is related to the file format and codec (mayby plugin in gstreamer). For example try to run this trailer: https://trailers.apple.com/movies/universal/jurassic-world-dominion/jurassic-world-dominion-trailer-1-stereo_h1080p.mov.
We will test it and to be sure, we do not solve issues, which are not really issue, for now, what is you recommendation for the file encoding to decode it in the player? We try to understand the plugin sources, but as I understood to gstreamer, there are the pipes and different pipes define, how to decode different formats.

@ardera
Copy link
Owner

ardera commented Feb 14, 2022

I know of this issue actually (That's what I meant with the hw decoder failing in this comment) The h264 decoder just fails out of nowhere. I'm not sure if it's a bug inside the gstreamer plugin or the kernel driver. But it's something about the h264 stream itself, I have 2 example video files with exact same format, but one works and one doesn't. It could be it's the compression level of the h264 stream, but I haven't verified that

@ardera ardera added bug Something isn't working p: gstreamer video player Issues related to the gstreamer video player plugin. labels Feb 28, 2022
@ardera
Copy link
Owner

ardera commented Mar 6, 2022

okay it's is a gstreamer problem that happens when the resolution of the video changes mid-decode (gstreamer doesn't handle that correctly): raspberrypi/firmware#1673

workaround is:

@radeksalomon
Copy link
Author

Nice. Good to know. We will try and will report back. As we develop it on arch linux, arch uses upstream for rpi firmware. So when it is ready, we test it.

@ardera
Copy link
Owner

ardera commented Mar 18, 2022

Just to clarify, the issue is not in the rpi firmware, but gstreamer. Fix for raspbian has landed in bullseye some days ago. Which version of gstreamer do you have on arch? @radeksalomon

@radeksalomon
Copy link
Author

@ardera Thanks. We will make a try. Archlinux has 1.20.1 released 5 days ago. I see in the release notes:

  • matroskamux: Handle multiview-mode/flags/pixel-aspect-ratio caps
    fields correctly when checking caps equality on input caps changes

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/4bbb293eacbb795b45152e97796092ebbdee9d28/subprojects/gst-devtools/NEWS

So maybe it should work. I let you know, when we try it.

@ardera
Copy link
Owner

ardera commented Mar 20, 2022

I don't think 1.20.1 contains the fix. There's an open issue about backporting the fix to 1.20: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1094

So I think you're gonna have to build gstreamer from source to get hardware acceleration, for now at least

@radeksalomon
Copy link
Author

Thx. I will follow this issue too. But we can build it from source. We have dedicated builders also for our own packages. I let you know...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working p: gstreamer video player Issues related to the gstreamer video player plugin.
Projects
None yet
Development

No branches or pull requests

2 participants