Unaccounted for WebXR state when not enough detail present in the environment

[milesgreen]

Description

When entering WebXR, there appears to be an unaccounted for state, prior to 'session-started'. It appears this state happens when the camera is not receiving enough information to even begin tracking the environment. For example, if the user is too close to a surface, the scene is too dark, or the user is looking at a very blank wall. The session-started status only seems to begin once the WebXR engine is able to start tracking an environment with enough detail. It also appears that ModelViewer does not render the floating 3D model during this state.

This leaves us with a potentially anomalous state, when the WebXR session has actually started (the camera feed is on) but ModelViewer has not yet registered state as session-started and the 3D model is not displayed. Potentially leading to a confusing state for the user.

It would be useful to be able to handle this state though, as it could be used to provide additional guidance to users during the WebXR experience - 'unable to scan environment', 'needs more light', 'back up a bit please', etc.

Live Demo

In the below demo, both JavaScript event listeners and CSS attribute selectors are being used to visually update status changes. You'll notice that when WebXR session starts, and the camera is intentionally too close to a blank surface, we see the camera feed, yet the CSS attribute selectors still reflect status as not-presenting, no JS ar-status event has fired, and the 3D model is not rendered. Only when the camera is panned to reveal a little bit of detail - enough to start trying to track the environment - does the JS event and CSS update to a status of session-started and the model then appears. By panning for more detail, we see the status change as expected from session-started to object-placed.

https://puzzle-chip-pluto.glitch.me/

Version

• model-viewer: v1.6.0

Browser Affected

• Chrome, version: 90.0.4430.91
• Edge
• Firefox
• IE
• Safari

OS

• Android, v 11
• iOS
• Linux
• MacOS
• Windows

AR

• WebXR
• SceneViewer
• QuickLook

Device

Pixel 4 XL

WebXRStatus-480p.mov

[milesgreen]

@elalish - Maybe the above issue is addressed in PR #2279 ?

[elalish]

Maybe? It has some problems I'm still working on fixing, but if you'd be willing to build and test for me, that would be helpful.

[milesgreen]

Sure thing. I'll take a look. Just build from the latest master branch? On it.

[milesgreen]

Tested with latest build.

I'm still seeing the above issue.

I see what I think is the new Instant Placement behaviour (which is nice btw).

But I still see the anomalous state when not enough detail is in the scene.

New Glitch with the manual build of model-viewer.min.js as at 2021.05.03

https://yellow-brass-helmet.glitch.me/

WebXRStatus2-480p.mov

[elalish]

Yeah, I think this isn't just at the beginning of a session; if you cover the camera for a few seconds, the render vanishes because the pose is null because it lost tracking. I think adding another enum for tracking-lost would make sense. Sound good?

[milesgreen]

Ah, yes. Right, you can fall back into that state too. I can imagine users being able to get into this state if, after an object has been placed, the user moves in too close to a surface to inspect a 3D object and looses tracking. This is more noticeable for vertical placement, as walls tend to have less detail.

Definitely would be helpful to be able to catch this state as an additional state enum and be able to react to it.

Just thinking through the sequence of events though... it's useful to have a single event that will only fire once and at the very start of the WebXR session - in order to set up DOM elements and so on. I had been treating session-started as this event. But in actuality, it's not firing until WebXR is tracking.

Given this new tracking-lost state, is there a chance the user could end up bouncing between the two states of session-started and tracking-lost multiple times in a session? Thus making it harder to set up any DOM changes when entering WebXR.

Possibly session-started should be triggered once, when the WebXR session really first starts and we see the camera feed (regardless of tracking), and introduce a new tracking and tracking-lost event to handle the tracking events. object-placed, not-presenting, failed events continue to behave as is.

Would this be too much of a breaking change?

[elalish]

I like this idea, thanks for thinking it through. I think session-started not happening right at session start is a bug rather than a breaking change. As I was implementing the tracking event I realized it's a little tricky because you always start out not tracking, but that's expected; it takes a few frames to understand your space. I'm going to put in a delay so that it only sends a not-tracking event if it's been long enough that there seems to be a problem.

[milesgreen]

Nice one, Emmett. Thanks.

[milesgreen]

@elalish - I tested a build of your ar-tracking update, and it works really nicely:

https://cedar-plaid-drain.glitch.me/

session-started event now happens as soon as going into WebXR ✅
The ar-tracking events work great - through both CSS rules and JS events ✅

Thanks!

WebXRStatus-ARStatus-480p.mov

[elalish]

Thanks for testing it! That's good to hear.