raspberry pi 3: startx crash on openbox and lxde after latest upgrade

[enricoT]

openbox: symbol lookup error: /opt/vc/lib/libEGL.so.1: undefined symbol: glDiscardFramebufferEXT

To open openbox or lxde downgrade was necessary:
[2016-07-03 14:04] [PACMAN] Running 'pacman -U linux-raspberrypi-4.4.14-1-armv7h.pkg.tar.xz raspberrypi-firmware-bootloader-20160627-1-any.pkg.tar.xz raspberrypi-firmware-bootloader-x-20160627-1-any.pkg.tar.xz raspberrypi-firmware-tools-20160627-1-any.pkg.tar.xz raspberrypi-firmware-examples-20160627-1-any.pkg.tar.xz raspberrypi-firmware-20160627-1-any.pkg.tar.xz'
[2016-07-03 14:04] [ALPM] transaction started
[2016-07-03 14:04] [ALPM] downgraded linux-raspberrypi (4.4.14-2 -> 4.4.14-1)
[2016-07-03 14:04] [ALPM-SCRIPTLET] >>> Updating module dependencies. Please wait ...
[2016-07-03 14:04] [ALPM] downgraded raspberrypi-firmware-bootloader (20160702-1 -> 20160627-1)
[2016-07-03 14:04] [ALPM] downgraded raspberrypi-firmware-bootloader-x (20160702-1 -> 20160627-1)
[2016-07-03 14:04] [ALPM] downgraded raspberrypi-firmware-tools (20160702-1 -> 20160627-1)
[2016-07-03 14:06] [ALPM] downgraded raspberrypi-firmware-examples (20160702-1 -> 20160627-1)
[2016-07-03 14:06] [ALPM] downgraded raspberrypi-firmware (20160702-1 -> 20160627-1)

[popcornmix]

If you are wanting to use the experimental arm side GL driver, then make sure it is on the ld library patch before /opt/vc/lib. See: anholt/mesa#24

[clivem]

I think the problem is that by putting the SONAME on those GL libs, you are now going to run into problems where userland progs that have been compiled with mesa version and linked against symbols that don't exist in the RPi version.

[popcornmix]

I have confirmed that kodi linked with previous /opt/vc/lib libraries will still run after the update.
The behaviour of apps using the experimental arm side GL driver is something I know less about (@spl237 did some testing and seemed happy).

However our position is that /opt/vc/lib should be before /usr/lib/* in the ld library path if you want to use the firmware side GL driver and it should be after if you want to use the arm side GL driver.

raspi-config will support this behaviour.

[clivem]

Clue: #110

[tesfabpel]

It also affects me on Raspberry Pi 2

[popcornmix]

@tesfabpel No me too reports please.
Explain exactly what your problem is, including the distribution you are using and if you are using the experimental arm side GL driver.

[clivem]

This has nothing to do with the "experimental arm side GL driver", AFAICT.

Surely anything linked against EGL but not GLESv2 is going to have issues with undefined symbols, if it is forced to use the RPi supplied libEGL?

[popcornmix]

@clivem we are trying to discover if the only issue is the ld library path ordering.
Assuming it is then that will be fixed with update to raspbian, and other distributions may need to change their library path ordering.
If there are other issues then we need more info in the bug reports.

[tesfabpel]

@popcornmix ArchLinux ARM.
How can I check if I am using the experimental arm side GL driver? (I have installed xf86-video-fbdev)

[clivem]

we are trying to discover if the only issue is the ld library path ordering.

Well, either I'm doing a bad job at explaining or you are not listening. Have fun!

[Ruffio]

@clivem, What kind of comment/attitude is that? Please respect others free contribution

[popcornmix]

@tesfabpel What are the exact symptoms you see? Posy any error messages you see.

In raspbian the desktop manager and typical apps do not use EGL.
The software mesa driver is too slow to be usable.
The experimental driver may be used, but that excludes the use of firmware driver, and so ld library path ordering can handle that.
If under Arch the desktop manager is linking with the software mesa EGL driver, but not actually using it then that may be an awkward obstacle.

[tesfabpel]

I use LXDM as the display manager... In its logs there is a line saying the same as above (/opt/vc/lib/libEGL.so.1: undefined symbol: glDiscardFramebufferEXT)

I tried spawning a blank X server with this command: X :1
then I tried running glxgears and it worked...

Then I commented the line in /etc/ld.so.conf.d/00-raspberry-firmware.conf (/opt/vc/lib), then ran ldconfig -v and rebooted and now everything works again...

[popcornmix]

and rebooted and now everything works again...

Almost everything. You won't be able to run apps using firmware side 3D or camera or video decode.

[popcornmix]

@tesfabpel Does renaming 00-raspberry-firmware.conf to zz-raspberry-firmware.conf (without commenting out its contents) and then running ldconfig also work?

(still not a perfect solution, but may allow camera/video to work, but not firmware side 3d).

[clivem]

@popcornmix I'm not at home at the moment, but the kludge I implemented this morning, following a Fedora build that included that userland SONAME patch, when anything I had linked to libEGL stopped working, was to add -lGLESv2 to the CMakefile.txt EGL target_libraries and rebuild userland. Cant just add GLESv2, because then you have a recursive build dependency, GLESv2 has EGL as a target lib.

[popcornmix]

@clivem do you mean:

diff --git a/interface/khronos/CMakeLists.txt b/interface/khronos/CMakeLists.txt
index d8b1280..6b80ef5 100644
--- a/interface/khronos/CMakeLists.txt
+++ b/interface/khronos/CMakeLists.txt
@@ -71,7 +71,7 @@ add_library(GLESv2_static STATIC ${GLES_SOURCE})
 add_library(khrn_static STATIC
    ${EGL_SOURCE} ${GLES_SOURCE} ${VG_SOURCE} ${WFC_SOURCE} ${CLIENT_SOURCE})

-target_link_libraries(EGL khrn_client vchiq_arm vcos bcm_host -lm)
+target_link_libraries(EGL khrn_client vchiq_arm vcos bcm_host -lm -lGLESv2)
 target_link_libraries(GLESv2 EGL khrn_client vcos)
 target_link_libraries(WFC EGL)
 target_link_libraries(OpenVG EGL)

That's not something that works in my cross compile environment (presumably as there is no GLESv2 installed when this runs).

[tesfabpel]

@popcornmix I don't think it changes anything at all...
In my case ldconfig -v reported this:

/usr/lib/libfakeroot:
        ...
/opt/vc/lib:
        libOpenVG.so.1 -> libOpenVG.so.1
        libvcsm.so -> libvcsm.so
        libdebug_sym.so -> libdebug_sym.so
        libEGL.so.1 -> libEGL.so.1
        libmmal_util.so -> libmmal_util.so
        libbcm_host.so -> libbcm_host.so
        libmmal_vc_client.so -> libmmal_vc_client.so
        libmmal.so -> libmmal.so
        libmmal_components.so -> libmmal_components.so
        libcontainers.so -> libcontainers.so
        libvchiq_arm.so -> libvchiq_arm.so
        libmmal_core.so -> libmmal_core.so
        libdtovl.so -> libdtovl.so
        libopenmaxil.so -> libopenmaxil.so
        libWFC.so -> libWFC.so
        libvcos.so -> libvcos.so
        libGLESv2.so.2 -> libGLESv2.so.2
/usr/lib:
        ...

That's because /etc/ld.so.conf contains this:

#
# /etc/ld.so.conf
#

include /etc/ld.so.conf.d/*.conf

# End of file

And in the .d directory there are only these two files:
00-raspberrypi-firmware.conf
fakeroot.conf

[popcornmix]

On raspbian the mesa libs come from /usr/lib/arm-linux-gnueabihf which is set from /etc/ld.so.conf.d/arm-linux-gnueabihf.conf.
So you can change the order of the two by modifying the names of the files in /etc/ld.so.conf.d.
It looks like things are different with Arch.

[clivem]

target_link_libraries(EGL khrn_client vchiq_arm vcos bcm_host -lm -lGLESv2)

Yep, that's it. (For me it worked, compiling on a Pi with the Fedora system mesa-libGLES-devel package installed, so /usr/lib/libGLESv2.so was there to satisfy ld at link time.) I wasn't saying this was the "right" way to sort this, but just a quick band-aid, for me to get on with doing productive things this morning. Re-compiling every app to explicitly link against libGLESv2 additionally, (that had previously been compiled against system/mesa EGL), isn't the answer. If the Rpi libEGL needs libGLESv2 to avoid unsatisifed symbols with apps that have been compiled against system EGL without needing a specific -lGLESv2, it needs sorting there, IMHO.

$ ldd /usr/lib/vc/libEGL.so.1
        linux-vdso.so.1 (0x7ee7f000)
        libbcm_host.so => /usr/lib/vc/libbcm_host.so (0x76f0d000)
        libm.so.6 => /lib/libm.so.6 (0x76e8b000)
        libGLESv2.so.2 => /usr/lib/vc/libGLESv2.so.2 (0x76e63000)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        libvchiq_arm.so => /usr/lib/vc/libvchiq_arm.so (0x76e4c000)
        libvcos.so => /usr/lib/vc/libvcos.so (0x76e30000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x76e08000)
        libdl.so.2 => /lib/libdl.so.2 (0x76df3000)
        librt.so.1 => /lib/librt.so.1 (0x76ddc000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x76db0000)
        libc.so.6 => /lib/libc.so.6 (0x76c59000)
        /lib/ld-linux-armhf.so.3 (0x54b8d000)

(My RPi vc stuff lives in /usr/lib/vc, not under /opt/vc like you do it with Raspbian. Not that this matters.)

[fhaun]

I've done an raspberrypi-firmware-20160703-1* update now. System is ALARM (Arch Linux on Raspbery 3)

/etc/ld.so.conf.d/00-raspberrypi-firmware.conf

# Disabled by default due to ABI incompatibility with Mesa
#/opt/vc/lib/

So the

symbol lookup error: /opt/vc/lib/libEGL.so.1: undefined symbol: glDiscardFramebufferEXT

is gone.

Problem is, kodi wants libs in /opt/vc/lib/

/usr/lib/kodi/kodi.bin: error while loading shared libraries: libbcm_host.so: cannot open shared object file:: No such file or directory

My workaround is:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/vc/lib
kodi-standalone

[fhaun]

With the firmware update "raspberrypi-firmware-20160704-1" kodi is crashing.

http://pastebin.com/26Drtag4

[popcornmix]

If you are having problems with the update then:
sudo rpi-update 01814172b4e6479cc2670fd254dfb738424dbecf
should get you back to the previous working state.

We are working on an alternative solution to this. It may be pushed out tomorrow.

[clivem]

We are working on an alternative solution to this. It may be pushed out tomorrow.

Do you want me to be the crash test dummy, before you push it to firmware? ;)

[popcornmix]

I think the new scheme should be safe for the next update.

The real issue to avoid the wrong libEGL.so / libGLESv2.so being picked up in the different use cases (firmware GL driver/experimental arm GL driver/mesa software GL driver).

We've decided ld library path ordering can handle the first two cases but not the third (where both firmware GL driver and mesa software GL driver may both be wanted in same session). We've got to go for the more disruptive (but probably more correct) scheme.

The libs in /opt/vc/lib will be renamed to avoid the collision. e.g. /opt/vc/lib/libbrcmEGL.so and /opt/vc/lib/libbrcmGLESv2.so. This will obviously break any users of the firmware GL driver, so for the next raspbian release we'll include both the old and new named libs.

We'll announce the change that is happening and encourage applications that use the firmware GL driver to rebuild with the new names.

In a subsequent raspbian release we'll remove the old named firmware GL libs.

[clivem]

where both firmware GL driver and mesa software GL driver may both be wanted in same session

I'm probably being dim, but why on earth would anyone want to use (software) mesa when there is a drop-in replacement (hardware) accelerated GL driver available? What am I missing?

I mean, anyone who has used an nvidia card in Linux for years, with their binary drivers, ends up using their replacement GL lib at runtime, (typically by having it first in ld path before any other copy), and doesn't need to fallback to using mesa. I don't get why, if there is a hardware accelerated libEGL available, you'd want to use mesa version in same session.....

[popcornmix]

The firmware driver doesn't integrate with X, and is GLES only (most X apps are desktop GL).

Currently the hardware mesa driver is experimental and stops the use of gpu facilities like camera and video decode. There will be new schemes for these but they will take time.

So all 3 drivers have their uses. Maybe there will come a day when everyone uses the one true driver, but it's not today.

[clivem]

Thanks for the explanation. That makes more sense now. (And you're probably figuring out about now, that 95% of my Pi usage is headless, and that I don't use camera or video decode.)

[popcornmix]

I've pushed a firmware update. Does this resolve the issue?

[clivem]

@popcornmix This is just reverting the SONAME patch, not what you proposed above with libbrcmEGL and libbrcmGLESv2 naming, right?

[popcornmix]

No it is both.

[clivem]

@popcornmix Can you please check the update into userland repo?

[popcornmix]

Ah yes. Done.

[clivem]

Ta. I'll spin a re-build of my Fedora packages.....

[fhaun]

On ALARM (Arch Linux) everything seems fine now. Thanks!

[popcornmix]

@enricoT is it okay for you now?

[tesfabpel]

@popcornmix Seems fine to me... Thanks! :)

[ali1234]

We'll announce the change that is happening and encourage applications that use the firmware GL driver to rebuild with the new names.

Where was this announced? Nobody seems to know about it at all.