Skip to content

[camerax] Implement lockCaptureOrientation & unlockCaptureOrientation #5285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 75 commits into from
Jan 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
dec3d69
Merge remote-tracking branch 'upstream/main' into camx_occ
camsim99 May 1, 2023
0e0333b
Merge remote-tracking branch 'upstream/main'
camsim99 May 2, 2023
bd7ac99
Merge remote-tracking branch 'upstream/main'
camsim99 May 3, 2023
5c3363b
Merge remote-tracking branch 'upstream/main'
camsim99 May 10, 2023
fed9621
Undo changes
camsim99 May 10, 2023
5aabe34
Merge remote-tracking branch 'upstream/main'
camsim99 May 12, 2023
2b9a352
Merge remote-tracking branch 'upstream/main'
camsim99 May 25, 2023
a1173da
Merge remote-tracking branch 'upstream/main'
camsim99 May 30, 2023
cbc3d6b
Merge remote-tracking branch 'upstream/main'
camsim99 May 30, 2023
cae5a4c
Merge remote-tracking branch 'upstream/main'
camsim99 Jun 1, 2023
72283db
Merge remote-tracking branch 'upstream/main'
camsim99 Jun 5, 2023
166a77c
Merge remote-tracking branch 'upstream/main'
camsim99 Jun 5, 2023
399780e
Merge remote-tracking branch 'upstream/main'
camsim99 Jun 14, 2023
8d5d0e7
Merge remote-tracking branch 'upstream/main'
camsim99 Jun 26, 2023
084d960
Merge remote-tracking branch 'upstream/main'
camsim99 Jul 12, 2023
d2a59ac
Merge remote-tracking branch 'upstream/main'
camsim99 Jul 17, 2023
a1422bf
Merge remote-tracking branch 'upstream/main'
camsim99 Jul 17, 2023
bdd87a6
Merge remote-tracking branch 'upstream/main'
camsim99 Jul 18, 2023
137a28b
Merge remote-tracking branch 'upstream/main'
camsim99 Jul 19, 2023
bc0db5a
Merge remote-tracking branch 'upstream/main'
camsim99 Jul 20, 2023
d04b466
Merge remote-tracking branch 'upstream/main'
camsim99 Jul 21, 2023
a9cfe87
Merge remote-tracking branch 'upstream/main'
camsim99 Jul 24, 2023
a32def1
Merge remote-tracking branch 'upstream/main'
camsim99 Aug 2, 2023
4785148
Merge remote-tracking branch 'upstream/main'
camsim99 Aug 14, 2023
7a8fc69
Merge remote-tracking branch 'upstream/main'
camsim99 Aug 15, 2023
b02e15f
Merge remote-tracking branch 'upstream/main'
camsim99 Aug 15, 2023
c6e5868
Merge remote-tracking branch 'upstream/main'
camsim99 Aug 17, 2023
0c0065a
Merge remote-tracking branch 'upstream/main'
camsim99 Aug 28, 2023
9dfe259
Merge remote-tracking branch 'upstream/main'
camsim99 Aug 29, 2023
bfcc0df
Merge remote-tracking branch 'upstream/main'
camsim99 Aug 30, 2023
b80cc86
Merge remote-tracking branch 'upstream/main'
camsim99 Sep 5, 2023
4663838
Merge remote-tracking branch 'upstream/main'
camsim99 Sep 13, 2023
33fe515
Merge remote-tracking branch 'upstream/main'
camsim99 Sep 21, 2023
0c34fb6
Merge remote-tracking branch 'upstream/main'
camsim99 Sep 26, 2023
b4b6347
Merge remote-tracking branch 'upstream/main'
camsim99 Sep 28, 2023
ea4387d
Merge remote-tracking branch 'upstream/main'
camsim99 Oct 9, 2023
5c22eb8
Merge remote-tracking branch 'upstream/main'
camsim99 Oct 10, 2023
49b4743
Merge remote-tracking branch 'upstream/main'
camsim99 Oct 24, 2023
87d1382
Merge remote-tracking branch 'upstream/main'
camsim99 Oct 25, 2023
4ba04e0
Merge remote-tracking branch 'upstream/main'
camsim99 Oct 25, 2023
dc8e8f3
Merge remote-tracking branch 'upstream/main'
camsim99 Oct 26, 2023
3affd3a
Merge remote-tracking branch 'upstream/main'
camsim99 Oct 27, 2023
cf4b868
Merge remote-tracking branch 'upstream/main'
camsim99 Oct 30, 2023
5c1470e
Rotation things :)
camsim99 Oct 31, 2023
13fd29e
Implement unlockCaptureOrientation
camsim99 Oct 31, 2023
6096c37
Refactor DeviceOrientationManager
camsim99 Nov 7, 2023
fd99476
Add new files
camsim99 Nov 7, 2023
03efa2f
Correct old dart tests
camsim99 Nov 7, 2023
b37dbbf
Add new wrapper tests
camsim99 Nov 8, 2023
1307099
Start attepmting plugin dart test code
camsim99 Nov 8, 2023
b4ad9c3
Fix and add java tests
camsim99 Nov 8, 2023
39ab573
Debugging detached objects for testing
camsim99 Nov 9, 2023
85754de
Merge remote-tracking branch 'upstream/main' into camx_orientation
camsim99 Nov 27, 2023
2ecdcb0
Make corrections from merge
camsim99 Nov 27, 2023
e7fa2c2
Fix plugin dart tests
camsim99 Nov 27, 2023
6b6080e
Merge remote-tracking branch 'upstream/main' into camx_orientation
camsim99 Nov 27, 2023
64e07de
Bump version
camsim99 Nov 28, 2023
0e614aa
Formatting
camsim99 Nov 28, 2023
85f8190
Merge remote-tracking branch 'upstream/main' into camx_orientation
camsim99 Dec 4, 2023
00879d2
1: debugging orientation and doing some wrapping
camsim99 Dec 6, 2023
95daaf6
add potential fix
camsim99 Dec 6, 2023
b5acca5
Add correct solution
camsim99 Dec 8, 2023
a410de1
Add theoretical fix
camsim99 Dec 9, 2023
f1d21e2
Fix tests
camsim99 Dec 11, 2023
2be21da
Add branching for potential Camx error
camsim99 Dec 11, 2023
6f81dbd
Merge remote-tracking branch 'upstream/main' into camx_orientation
camsim99 Dec 11, 2023
b28ad12
Address Gray review
camsim99 Dec 11, 2023
0e01e43
Self review
camsim99 Dec 11, 2023
99c42f6
Merge + correct nits
camsim99 Dec 27, 2023
00c6ace
Address review
camsim99 Dec 27, 2023
d2ecd9a
Fix mocks
camsim99 Dec 27, 2023
6e35acd
correct Java test
camsim99 Dec 27, 2023
cc80fc2
Correct names
camsim99 Dec 28, 2023
06b38cf
Nits
camsim99 Jan 2, 2024
34d074c
Fix todo format
camsim99 Jan 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions packages/camera/camera_android_camerax/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
## 0.5.0+25

* Implements `lockCaptureOrientation` and `unlockCaptureOrientation`.

## 0.5.0+24

* Updates example app to use non-deprecated video_player method.
Expand Down
4 changes: 0 additions & 4 deletions packages/camera/camera_android_camerax/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,6 @@ dependencies:
and thus, the plugin will fall back to 480p if configured with a
`ResolutionPreset`.

### Locking/Unlocking capture orientation \[[Issue #125915][125915]\]

`lockCaptureOrientation` & `unLockCaptureOrientation` are unimplemented.

### Exposure mode, point, & offset configuration \[[Issue #120468][120468]\]

`setExposureMode`, `setExposurePoint`, & `setExposureOffset` are unimplemented.
Expand Down
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to reviewer:
Refactored DeviceOrientationManager.java functionality into its own class (previously a part of SystemServices host/Flutter API implementations) for clarity. I will mark* changes related to this throughout this PR to make it clear that these are simply a refactor and no changes in logic occurred.

*Marked with comment: "Part of DeviceOrientation refactor!"

Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ public final class CameraAndroidCameraxPlugin implements FlutterPlugin, Activity
private ImageCaptureHostApiImpl imageCaptureHostApiImpl;
private CameraControlHostApiImpl cameraControlHostApiImpl;
public @Nullable SystemServicesHostApiImpl systemServicesHostApiImpl;
public @Nullable DeviceOrientationManagerHostApiImpl deviceOrientationManagerHostApiImpl;

@VisibleForTesting
public @Nullable ProcessCameraProviderHostApiImpl processCameraProviderHostApiImpl;
Expand Down Expand Up @@ -71,6 +72,10 @@ public void setUp(
systemServicesHostApiImpl =
new SystemServicesHostApiImpl(binaryMessenger, instanceManager, context);
GeneratedCameraXLibrary.SystemServicesHostApi.setup(binaryMessenger, systemServicesHostApiImpl);
deviceOrientationManagerHostApiImpl =
new DeviceOrientationManagerHostApiImpl(binaryMessenger, instanceManager);
GeneratedCameraXLibrary.DeviceOrientationManagerHostApi.setup(
binaryMessenger, deviceOrientationManagerHostApiImpl);
GeneratedCameraXLibrary.PreviewHostApi.setup(
binaryMessenger, new PreviewHostApiImpl(binaryMessenger, instanceManager, textureRegistry));
imageCaptureHostApiImpl =
Expand Down Expand Up @@ -145,6 +150,7 @@ public void onAttachedToActivity(@NonNull ActivityPluginBinding activityPluginBi
systemServicesHostApiImpl.setActivity(activity);
systemServicesHostApiImpl.setPermissionsRegistry(
activityPluginBinding::addRequestPermissionsResultListener);
deviceOrientationManagerHostApiImpl.setActivity(activity);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be unsetting the activity in onDetachedFromActivity (for both this and systemServicesHostApiImpl)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so after reading the docs so thanks for pointing this out! I don't want this change to get lost in this PR since it seems like a legitimate bug, so I filed flutter/flutter#139937 and will follow up with a PR.

}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
import android.view.Surface;
import android.view.WindowManager;
import androidx.annotation.NonNull;
import androidx.annotation.Nullable;
import androidx.annotation.VisibleForTesting;
import io.flutter.embedding.engine.systemchannels.PlatformChannel;
import io.flutter.embedding.engine.systemchannels.PlatformChannel.DeviceOrientation;
Expand Down Expand Up @@ -85,120 +84,6 @@ public void stop() {
broadcastReceiver = null;
}

/**
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to reviewer: I removed all unused functionality.

* Returns the device's photo orientation in degrees based on the sensor orientation and the last
* known UI orientation.
*
* <p>Returns one of 0, 90, 180 or 270.
*
* @return The device's photo orientation in degrees.
*/
public int getPhotoOrientation() {
return this.getPhotoOrientation(this.lastOrientation);
}

/**
* Returns the device's photo orientation in degrees based on the sensor orientation and the
* supplied {@link PlatformChannel.DeviceOrientation} value.
*
* <p>Returns one of 0, 90, 180 or 270.
*
* @param orientation The {@link PlatformChannel.DeviceOrientation} value that is to be converted
* into degrees.
* @return The device's photo orientation in degrees.
*/
public int getPhotoOrientation(@Nullable PlatformChannel.DeviceOrientation orientation) {
int angle = 0;
// Fallback to device orientation when the orientation value is null.
if (orientation == null) {
orientation = getUIOrientation();
}

switch (orientation) {
case PORTRAIT_UP:
angle = 90;
break;
case PORTRAIT_DOWN:
angle = 270;
break;
case LANDSCAPE_LEFT:
angle = isFrontFacing ? 180 : 0;
break;
case LANDSCAPE_RIGHT:
angle = isFrontFacing ? 0 : 180;
break;
}

// Sensor orientation is 90 for most devices, or 270 for some devices (eg. Nexus 5X).
// This has to be taken into account so the JPEG is rotated properly.
// For devices with orientation of 90, this simply returns the mapping from ORIENTATIONS.
// For devices with orientation of 270, the JPEG is rotated 180 degrees instead.
return (angle + sensorOrientation + 270) % 360;
}

/**
* Returns the device's video orientation in clockwise degrees based on the sensor orientation and
* the last known UI orientation.
*
* <p>Returns one of 0, 90, 180 or 270.
*
* @return The device's video orientation in clockwise degrees.
*/
public int getVideoOrientation() {
return this.getVideoOrientation(this.lastOrientation);
}

/**
* Returns the device's video orientation in clockwise degrees based on the sensor orientation and
* the supplied {@link PlatformChannel.DeviceOrientation} value.
*
* <p>Returns one of 0, 90, 180 or 270.
*
* <p>More details can be found in the official Android documentation:
* https://developer.android.com/reference/android/media/MediaRecorder#setOrientationHint(int)
*
* <p>See also:
* https://developer.android.com/training/camera2/camera-preview-large-screens#orientation_calculation
*
* @param orientation The {@link PlatformChannel.DeviceOrientation} value that is to be converted
* into degrees.
* @return The device's video orientation in clockwise degrees.
*/
public int getVideoOrientation(@Nullable PlatformChannel.DeviceOrientation orientation) {
int angle = 0;

// Fallback to device orientation when the orientation value is null.
if (orientation == null) {
orientation = getUIOrientation();
}

switch (orientation) {
case PORTRAIT_UP:
angle = 0;
break;
case PORTRAIT_DOWN:
angle = 180;
break;
case LANDSCAPE_LEFT:
angle = 270;
break;
case LANDSCAPE_RIGHT:
angle = 90;
break;
}

if (isFrontFacing) {
angle *= -1;
}

return (angle + sensorOrientation + 360) % 360;
}

/** @return the last received UI orientation. */
public @Nullable PlatformChannel.DeviceOrientation getLastUIOrientation() {
return this.lastOrientation;
}

/**
* Handles orientation changes based on change events triggered by the OrientationIntentFilter.
*
Expand Down Expand Up @@ -241,7 +126,7 @@ static void handleOrientationChange(
@SuppressWarnings("deprecation")
@VisibleForTesting
PlatformChannel.DeviceOrientation getUIOrientation() {
final int rotation = getDisplay().getRotation();
final int rotation = getDefaultRotation();
final int orientation = activity.getResources().getConfiguration().orientation;

switch (orientation) {
Expand All @@ -265,57 +150,18 @@ PlatformChannel.DeviceOrientation getUIOrientation() {
}

/**
* Calculates the sensor orientation based on the supplied angle.
*
* <p>This method is visible for testing purposes only and should never be used outside this
* class.
*
* @param angle Orientation angle.
* @return The sensor orientation based on the supplied angle.
*/
@VisibleForTesting
PlatformChannel.DeviceOrientation calculateSensorOrientation(int angle) {
final int tolerance = 45;
angle += tolerance;

// Orientation is 0 in the default orientation mode. This is portrait-mode for phones
// and landscape for tablets. We have to compensate for this by calculating the default
// orientation, and apply an offset accordingly.
int defaultDeviceOrientation = getDeviceDefaultOrientation();
if (defaultDeviceOrientation == Configuration.ORIENTATION_LANDSCAPE) {
angle += 90;
}
// Determine the orientation
angle = angle % 360;
return new PlatformChannel.DeviceOrientation[] {
PlatformChannel.DeviceOrientation.PORTRAIT_UP,
PlatformChannel.DeviceOrientation.LANDSCAPE_LEFT,
PlatformChannel.DeviceOrientation.PORTRAIT_DOWN,
PlatformChannel.DeviceOrientation.LANDSCAPE_RIGHT,
}
[angle / 90];
}

/**
* Gets the default orientation of the device.
* Gets default capture rotation for CameraX {@code UseCase}s.
*
* <p>This method is visible for testing purposes only and should never be used outside this
* class.
* <p>See
* https://developer.android.com/reference/androidx/camera/core/ImageCapture#setTargetRotation(int),
* for instance.
*
* @return The default orientation of the device.
* @return The rotation of the screen from its "natural" orientation; one of {@code
* Surface.ROTATION_0}, {@code Surface.ROTATION_90}, {@code Surface.ROTATION_180}, {@code
* Surface.ROTATION_270}
*/
@VisibleForTesting
int getDeviceDefaultOrientation() {
Configuration config = activity.getResources().getConfiguration();
int rotation = getDisplay().getRotation();
if (((rotation == Surface.ROTATION_0 || rotation == Surface.ROTATION_180)
&& config.orientation == Configuration.ORIENTATION_LANDSCAPE)
|| ((rotation == Surface.ROTATION_90 || rotation == Surface.ROTATION_270)
&& config.orientation == Configuration.ORIENTATION_PORTRAIT)) {
return Configuration.ORIENTATION_LANDSCAPE;
} else {
return Configuration.ORIENTATION_PORTRAIT;
}
int getDefaultRotation() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I believe this method is named getDefaultRotation because it is the rotation that CameraX use cases (at least ImageCapture) default to right? I think we might want to name it something else, though, because without that context from the comment here, it mostly gets used in places where it looks like we are getting a default rotation of the camera proxy, (i.e. proxy.getDefaultRotation()) which at first glance doesn't seem like something that would be a function of the current rotation of the device, but rather a persistent default rotation.

Basically it just feels more like a getCurrentRotation than a getDefaultRotation, to me at least. Not a big deal, though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To address this, I renamed the wrapped method to getDefaultDisplayRotation() and left this since this really a part of the DeviceOrientationManager and would like to keep it consistent with camera_android for now.

return getDisplay().getRotation();
}

/**
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

package io.flutter.plugins.camerax;

import androidx.annotation.NonNull;
import io.flutter.plugin.common.BinaryMessenger;
import io.flutter.plugins.camerax.GeneratedCameraXLibrary.DeviceOrientationManagerFlutterApi;

public class DeviceOrientationManagerFlutterApiImpl extends DeviceOrientationManagerFlutterApi {
public DeviceOrientationManagerFlutterApiImpl(@NonNull BinaryMessenger binaryMessenger) {
super(binaryMessenger);
}

public void sendDeviceOrientationChangedEvent(
@NonNull String orientation, @NonNull Reply<Void> reply) {
super.onDeviceOrientationChanged(orientation, reply);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

package io.flutter.plugins.camerax;

import android.app.Activity;
import androidx.annotation.NonNull;
import androidx.annotation.Nullable;
import androidx.annotation.VisibleForTesting;
import io.flutter.embedding.engine.systemchannels.PlatformChannel.DeviceOrientation;
import io.flutter.plugin.common.BinaryMessenger;
import io.flutter.plugins.camerax.CameraPermissionsManager.PermissionsRegistry;
import io.flutter.plugins.camerax.GeneratedCameraXLibrary.DeviceOrientationManagerHostApi;

public class DeviceOrientationManagerHostApiImpl implements DeviceOrientationManagerHostApi {
private final BinaryMessenger binaryMessenger;
private final InstanceManager instanceManager;

@VisibleForTesting public @NonNull CameraXProxy cameraXProxy = new CameraXProxy();
@VisibleForTesting public @Nullable DeviceOrientationManager deviceOrientationManager;

@VisibleForTesting
public @NonNull DeviceOrientationManagerFlutterApiImpl deviceOrientationManagerFlutterApiImpl;

private Activity activity;
private PermissionsRegistry permissionsRegistry;

public DeviceOrientationManagerHostApiImpl(
@NonNull BinaryMessenger binaryMessenger, @NonNull InstanceManager instanceManager) {
this.binaryMessenger = binaryMessenger;
this.instanceManager = instanceManager;
this.deviceOrientationManagerFlutterApiImpl =
new DeviceOrientationManagerFlutterApiImpl(binaryMessenger);
}

public void setActivity(@NonNull Activity activity) {
this.activity = activity;
}

/**
* Starts listening for device orientation changes using an instance of a {@link
* DeviceOrientationManager}.
*
* <p>Whenever a change in device orientation is detected by the {@code DeviceOrientationManager},
* the {@link SystemServicesFlutterApi} will be used to notify the Dart side.
*/
@Override
public void startListeningForDeviceOrientationChange(
@NonNull Boolean isFrontFacing, @NonNull Long sensorOrientation) {
deviceOrientationManager =
cameraXProxy.createDeviceOrientationManager(
activity,
isFrontFacing,
sensorOrientation.intValue(),
(DeviceOrientation newOrientation) -> {
deviceOrientationManagerFlutterApiImpl.sendDeviceOrientationChangedEvent(
serializeDeviceOrientation(newOrientation), reply -> {});
});
deviceOrientationManager.start();
}

/** Serializes {@code DeviceOrientation} into a String that the Dart side is able to recognize. */
String serializeDeviceOrientation(DeviceOrientation orientation) {
return orientation.toString();
}

/**
* Tells the {@code deviceOrientationManager} to stop listening for orientation updates.
*
* <p>Has no effect if the {@code deviceOrientationManager} was never created to listen for device
* orientation updates.
*/
@Override
public void stopListeningForDeviceOrientationChange() {
if (deviceOrientationManager != null) {
deviceOrientationManager.stop();
}
}

/**
* Gets default capture rotation for CameraX {@code UseCase}s.
*
* <p>The default capture rotation for CameraX is the rotation of default {@code Display} at the
* time that a {@code UseCase} is bound, but the default {@code Display} does not change in this
* plugin, so this value is {@code Display}-agnostic.
*
* <p>See
* https://developer.android.com/reference/androidx/camera/core/ImageCapture#setTargetRotation(int)
* for instance for more information on how this default value is used.
*/
@Override
public @NonNull Long getDefaultDisplayRotation() {
int defaultRotation;
try {
defaultRotation = deviceOrientationManager.getDefaultRotation();
} catch (NullPointerException e) {
throw new IllegalStateException(
"startListeningForDeviceOrientationChange must first be called to subscribe to device orientation changes in order to retrieve default rotation.");
}

return Long.valueOf(defaultRotation);
}
}
Loading