Rotated Object Detection

[austinmw]

Summary

Hi, first, I really appreciate your newly added support for object detection!

This feature request is probably not a high priority, but I wanted to submit anyway, since rotated object detection is particularly useful for aerial imagery—It strikes a nice balance between object detection and instance segmentation, since it's typically faster than segmentation, but can more accurately handle objects like vehicles (for example, trucks in a parking lot).

MMRotate provides a nice demo and motivating example: https://github.com/open-mmlab/mmrotate/blob/main/demo/MMRotate_Tutorial.ipynb

Rationale

Often in aerial imagery, pixel-level information is unnecessary, but standard object detection can not localize particular objects well enough. For example:

Implementation

Any easy-lift implementation would do initially. It doesn't appear that torchvision has built-in support for this yet.

MMRotate is particularly well developed, but they use their own MMEngine and MMCV as a base instead of PyTorch Lightning, so that might be higher effort to integrate than wanted (though integration between MMDetection, MMRotate, MMSegmentation would be incredibly powerful IMO).

Alternatives

No response

Additional information

No response

[adamjstewart]

@ashnair1 did the work on our new ObjectDetectionTask trainer, may be interested in working on this.

[ashnair1]

Thanks for opening this! Rotated Object Detection is definitely something I would love to see in torchgeo (considering I have to regularly deal with it at work 😄). We briefly talked about it in the ObjectDetection PR here.

As @austinmw correctly noted, torchvision doesn't currently support rotated boxes (and the necessary ops) though there is an open issue tracking this - pytorch/vision#2761. In the short term, the only way to support Rotated Object Detection would be to leverage either MMRotate or detectron2. But both projects are non-trivial to install and integrate with. One of the advantages of torchgeo is its ease of installation and I'm not sure if we should trade that in for this. Ideally, we would like torchvision to support at least the necessary ops. Discussion is picking up in the linked issue.

If that does not pan out, we could go ahead with integration with either of the two projects mentioned. But then we will need to define the scope. Both projects support normal detection and segmentation as well. Do we leverage this and for example replace segmentation-models-pytorch?

Alternatively, we could write up a tutorial wherein we use torchgeo and one of the aforementioned projects to show users how to train a rotated object detector. This would give users an example of how to use it without us having to commit it as a dependency.

[austinmw]

Thanks for your detailed response! All of your points make a lot of sense to me, excited to track how this discussion and progress goes.

In my personal opinion and for my use cases, the functionality I'd like to use is the power of torchgeo datamodules, so even a blog post on using them with an external library like MMRotate would be incredibly helpful.

[adamjstewart]

If we ever get around to implementing this, the FAIR1M dataset (#232) contains rotated bounding boxes and would be a good test case.

[robmarkcole]

I would like to do a status check on this - I dug through the linked issues and they are open.

I wonder if other frameworks could be considered too - for instance Yolov8 which supports rotated boxes (license might be an issue however).