`get_screen_to_ndc_transform` multiple values for cameras when `in_ndc=False`

[jasonyzhang]

If you do not know the root cause of the problem / bug, and wish someone to help you, please
post according to this template:

🐛 Bugs / Unexpected behaviors

Using a perspective camera with in_ndc=False causes cameras to be passed in twice to get_screen_to_ndc_transform. The reason is that cameras is both an ordered argument and usually in the kwargs dict.

pytorch3d/pytorch3d/renderer/cameras.py

Lines 1607 to 1609 in 4046677

	def get_screen_to_ndc_transform(
	cameras, with_xyflip: bool = False, **kwargs
	) -> Transform3d:

NOTE: Please look at the existing list of Issues tagged with the label 'bug`. Only open a new issue if this bug has not already been reported. If an issue already exists, please comment there instead..

Instructions To Reproduce the Issue:

cameras = pytorch3d.renderer.PerspectiveCameras(in_ndc=False)
rend = renderer_textured(meshes, cameras=cameras)

raises a TypeError:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_1472741/3117266896.py in <module>
      1 cameras = pytorch3d.renderer.PerspectiveCameras(in_ndc=False, device=device)
----> 2 rend = renderer_textured(meshes, cameras=cameras)

~/miniconda3/envs/p3d/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/miniconda3/envs/p3d/lib/python3.8/site-packages/pytorch3d-0.5.0-py3.8-linux-x86_64.egg/pytorch3d/renderer/mesh/renderer.py in forward(self, meshes_world, **kwargs)
     57         For this set rasterizer.raster_settings.clip_barycentric_coords=True
     58         """
---> 59         fragments = self.rasterizer(meshes_world, **kwargs)
     60         images = self.shader(fragments, meshes_world, **kwargs)
     61 

~/miniconda3/envs/p3d/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/miniconda3/envs/p3d/lib/python3.8/site-packages/pytorch3d-0.5.0-py3.8-linux-x86_64.egg/pytorch3d/renderer/mesh/rasterizer.py in forward(self, meshes_world, **kwargs)
    145             Fragments: Rasterization outputs as a named tuple.
    146         """
--> 147         meshes_proj = self.transform(meshes_world, **kwargs)
    148         raster_settings = kwargs.get("raster_settings", self.raster_settings)
    149 

~/miniconda3/envs/p3d/lib/python3.8/site-packages/pytorch3d-0.5.0-py3.8-linux-x86_64.egg/pytorch3d/renderer/mesh/rasterizer.py in transform(self, meshes_world, **kwargs)
    127         )
    128         # view to NDC transform
--> 129         to_ndc_transform = cameras.get_ndc_camera_transform(**kwargs)
    130         projection_transform = cameras.get_projection_transform(**kwargs).compose(
    131             to_ndc_transform

~/miniconda3/envs/p3d/lib/python3.8/site-packages/pytorch3d-0.5.0-py3.8-linux-x86_64.egg/pytorch3d/renderer/cameras.py in get_ndc_camera_transform(self, **kwargs)
   1045                 matrix=pr_point_fix.transpose(1, 2).contiguous(), device=self.device
   1046             )
-> 1047             screen_to_ndc_transform = get_screen_to_ndc_transform(
   1048                 self, with_xyflip=False, **kwargs
   1049             )

TypeError: get_screen_to_ndc_transform() got multiple values for argument 'cameras'

[nikhilaravi]

@jasonyzhang thanks for pointing this out. We will discuss how to resolve this!

[nikhilaravi]

We will fix this issue: **kwargs can be removed from the input to get_screen_to_ndc_transform and get_ndc_to_screen_transform and instead, at their callsites we can parse out the image_size and pass it into the function directly.

[gkioxari]

Ooops! We are fixing it! @jasonyzhang note that the cameras in master (soon to be the new release) have changed wrt to prior implementations and there are some braking changes. See this (very lonely!) discussion thread: #783

[jasonyzhang]

Great work! I'm very excited to see all the new improvements to Pytorch3D.

[YaroslavShchekaturov]

Hi there! I've tried you new version of the code with in_ndc= False , the code is below

R_RC = torch.tensor([[1,0,0],[0,1,0],[0,0,1] ])
R_RC = torch.unsqueeze(R_RC,0)
T_RC = torch.unsqueeze(torch.tensor([0,0,0 ] ),0)

cameras =PerspectiveCameras(R=R_RC,T=T_RC,image_size=(500,500),)
camera =PerspectiveCameras(R=R_RC,T=T_RC,image_size=(500,500),)
K = torch.tensor([[1,0,0],[0,1,0],[0,0,1]])

camera = cameras

lights = PointLights(device=device, location=[[0.0, 0.0, -3.0]])

raster_settings = RasterizationSettings(
image_size=(500,500),
blur_radius=0.0,
faces_per_pixel=1,

perspective_correct=False,

)

renderer = MeshRenderer(
rasterizer=MeshRasterizer(
cameras=camera,
raster_settings=raster_settings
),
shader=SoftPhongShader(
device=device,
cameras=camera,
lights=lights
)
)

Render the cow mesh from each viewing angle

target_images = renderer(mesh, cameras=cameras.cuda(), lights=lights)
plt.imshow(target_images[0].cpu())
plt.show()

But I got the following error :

ValueError: Wrong number (2) of cameras for 1 meshes

I got the same error for
cameras =PerspectiveCameras(R=R_RC,T=T_RC,image_size=(500,500),in_ndc = False)

[aja9675]

I am also seeing "ValueError: Wrong number (2) of cameras for 1 meshes" when using PerspectiveCameras(). I built from source, including 8041178.

[bottler]

@YaroslavShchekaturov and maybe @aja9675 too: There are two ways to specify image size when initializing the cameras. For square images, you can use a sequence (list/tuple/tensor) of integers of length N. For non square images, you need a list/tuple of pairs of integers, or a tensor of shape (N,2). N is the number of cameras or 1.

So,

PerspectiveCameras(R=R_RC,T=T_RC,image_size=(500,500),)

means two cameras, both with image size 500x500. (The R and T are broadcasted from batch size 1 to batch size 2.

I think you intend a single camera for a 500x500 image. So you could write

PerspectiveCameras(R=R_RC,T=T_RC,image_size=((500,500),))

or

PerspectiveCameras(R=R_RC,T=T_RC,image_size=(500,))