Papers with UnrealCV

A curated list of papers using UnrealCV.

This is a list of papers that use UnrealCV, an open source project to help researchers build virtual worlds using Unreal Engine 4. The papers are categorized by their applications, such as semantic understanding, 3D vision, embodied vision, etc.

The following labels are used to indicate the usage of UnrealCV in each paper:

• 📊Dataset: generate synthetic dataset for training.
• 🔍Diagnosis: control the factor for model diagnosis.
• 🏃Interaction: train/test agent(s) by interacting with the virtual worlds (e.g., RL).

Table of Content

• Semantic Understanding
  • Segmentation
  • Text Detection
  • Person Re-identification
  • Activity Recognition
• 3D Vision
  • Depth Estimation
  • Scene Reconstruction
  • Pose Estimation
• Embodied Vision
  • Visual Navigation
  • Active Object Tracking
  • Multi-agent Cooperation

Contributing

If you find any papers that used UnrealCV but are not included in this list, please feel free to send a PR or open an issue.

When sending PRs, please put the new paper at the correct chronological position as the following format:

- Usage Tags(📊🔍🏃) 
  **Paper Title** 
  *Author(s)* 
  Conference/Journal Year. [[Paper](link)] [[code](link)] [[Website](link)]

Semantic Understanding

Object Segmentation and Detection

• 📊 ScaleNet: Guiding Object Proposal Generation in Supermarkets and Beyond. Siyuan Qiao, Wei Shen, Weichao Qiu, Chenxi Liu, Alan Yuille. ICCV 2017. [paper] [code]

• 📊 Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation. Tremblay, Jonathan, Thang To, and Stan Birchfield. CVPR 2018 Workshop. [paper] [code]

• 📊 A Framework for Self-Training Perceptual Agents in Simulated Photorealistic Environments. Patrick Mania, Michael Beetz. ICRA 2019. [paper] [code]

• 📊 Semantic Part Detection via Matching: Learning to Generalize to Novel Viewpoints from Limited Training Data. Yutong Bai, Qing Liu, Lingxi Xie, Weichao Qiu, Yan Zheng, Alan Yuille. ICCV 2019. [paper]

• 📊 AugPOD: Augmentation-oriented Probabilistic Object Detection. Chuan-Wei Wang*, Chin-An Cheng*, Ching-Ju Cheng*, Hou-Ning Hu, Hung-Kuo Chu, Min Sun. CVPR 2019 Workshop. [paper]

• 📊 OmniSCV: An Omnidirectional Synthetic Image Generator for Computer Vision. Berenguel-Baeta, Bruno, Jesus Bermudez-Cameo, and Jose J. Guerrero. Sensors 2020. [paper]

Text Detection and Recognition

• 📊 UnrealText: Synthesizing Realistic Scene Text Images from the Unreal World. Minghui Liao, Guan Pang, Jing Huang, Tal Hassner, Xiang Bai. CVPR 2020. [paper] [code]

• 📊 Synthetic-to-Real Unsupervised Domain Adaptation for Scene Text Detection in the Wild. Weijia Wu, Ning Lu, Enze Xie, Yuxing Wang, Wenwen Yu, Cheng Yang, Hong Zhou. ACCV 2020. [paper] [code]

Person Re-identification

• 📊 UnrealPerson: An Adaptive Pipeline for Costless Person Re-identification. Tianyu Zhang, Lingxi Xie, Longhui Wei, Zijie Zhuang, Yongfei Zhang, Bo Li, Qi Tian. CVPR 2021. [paper] [code]

• 📊 Robust Person Re-Identification with Wireless Signals Xi, Dong, Wengang Zhou, and Houqiang Li. ICME 2023. [paper]

Activity Recognition

• 🔍 Identity Preserve Transform: Understand What Activity Classification Models Have Learnt. Jialing Lyu, Weichao Qiu, Xinyue Wei, Yi Zhang, Alan Yuille, Zheng-Jun Zha. CVPR 2020 Workshop. [paper]
• 📊 Active shooter detection and robust tracking utilizing supplemental synthetic data. Joshua R. Waite, Jiale Feng, Riley Tavassoli, Laura Harris, Sin Yong Tan, Subhadeep Chakraborty, Soumik Sarkar. Arxiv 2023. [paper]

3D Vision

Depth Estimation

• 🔍 Unrealstereo: Controlling hazardous factors to analyze stereo vision. Yi Zhang, Weichao Qiu, Qi Chen, Xiaolin Hu, Alan Yuille. 3DV 2018. [paper] [code]

• 📊 Coupled Real-Synthetic Domain Adaptation for Real-World Deep Depth Enhancement. Xiao Gu, Yao Guo, Fani Deligianni, Guang-Zhong Yang. IEEE Transactions on Image Processing (TIP) 2020. [paper]

• 📊 🏃 Enhancing optical-flow-based control by learning visual appearance cues for flying robots. G. C. H. E. de Croon, C. De Wagter, T. Seidl. Nature Machine Intelligence 2021. [paper]

• 📊 SMD-Nets: Stereo Mixture Density Networks. Fabio Tosi, Yiyi Liao, Carolin Schmitt, Andreas Geiger. CVPR 2021. [paper][code]

• 📊 G2-MonoDepth: A General Framework of Generalized Depth Inference from Monocular RGB+ X Data. Haotian Wang, Meng Yang, Nanning Zheng. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2023. [paper]

• 📊 Learning Occluded Branch Depth Maps in Forest Environments Using RGB-D Images Geckeler, C., Aucone, E., Schnider, Y., Simeon, A., von Bassewitz, J. P., Zhu, Y., & Mintchev, S. IEEE Robotics and Automation Letters (RA-L) 2024. [[paper](https://ieeexplore.ieee.org/document/10403997]

Scene Reconstruction

• 📊 Submodular Trajectory Optimization for Aerial 3D Scanning. Mike Roberts, Debadeepta Dey, Anh Truong, Sudipta Sinha, Shital Shah, Ashish Kapoor, Pat Hanrahan, Neel Joshi. ICCV 2017. [paper ] [project]
• 📊 GEN-SLAM: Generative Modeling for Monocular Simultaneous Localization and Mapping. Punarjay Chakravarty, Praveen Narayanan, Tom Roussel. ICRA 2019. [paper]
• 📊 QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAM. Lachlan Nicholson, Michael Milford, Niko Sünderhauf. IEEE Robotics and Automation Letters (RA-L) 2019. [paper]
• 🏃 Path Planning for Active V-Slam Based on Reinforcement Learning. Borui Li, Fuchun Sun, Huaping Liu, Bin Fang. International Conference on Cognitive Systems and Signal Processing 2019. [paper]
• 📊 UnrealNavigation: Simulation Software for testing SLAM in Virtual Reality. Anne M. Bettens, Benjamin Morrell, Mauricio Coen, Neil McHenry, Xiaofeng Wu, Peter Gibbens, Gregory Chamitoff. AIAA Scitech 2020 Forum. [paper] [code]
• 📊 An Efficient Sampling-based Method for Online Informative Path Planning in Unknown Environments. Lukas Schmid, Michael Pantic, Raghav Khanna, Lionel Ott, Roland Siegwart, Juan Nieto. IEEE Robotics and Automation Letters (RA-L) 2020. [paper] [code]
• 🏃 Next-Best View Policy for 3D Reconstruction. Peralta, D., Casimiro, J., Nilles, A.M., Aguilar, J.A., Atienza, R., and Cajote, R. ECCV Workshops 2020. [paper] [code]
• 📊 Flight Planning for Survey-Grade 3D Reconstruction of Truss Bridges. Zhexiong Shang, Zhigang Shen. Remote Sens. 2022. [paper]
• 📊 View Planning Using Discrete Optimization for 3D Reconstruction of Row Crops. Athanasios Bacharis, Henry J. Nelson and Nikolaos Papanikolopoulos. IROS 2022. [paper]

Pose Estimation

• 📊 A Unified Framework for Multi-View Multi-Class Object Pose Estimation. Chi Li, Jin Bai, Gregory D. Hager. ECCV 2018. [paper]

• 📊 🏃 CRAVES: Controlling Robotic Arm with a Vision-based, Economic System. Yiming Zuo*, Weichao Qiu*, Lingxi Xie, Fangwei Zhong, Yizhou Wang, Alan Yuille. CVPR 2019. [paper] [code]

• 📊 Learning from Synthetic Animals. Jiteng Mu*, Weichao Qiu*, Gregory Hager, Alan Yuille. CVPR 2020 (Oral). [paper] [code]

• 📊 AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild. Zhe Zhang, Chunyu Wang, Weichao Qiu, Wenhu Qin, Wenjun Zeng. IJCV 2020. [paper] [code]

• 📊 Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering. Pedro F. Proença, Yang Gao. ICRA 2020. [paper] [project]

• 📊 Learning From Synthetic Vehicles. Tae Soo Kim, Bohoon Shim, Michael Peven, Weichao Qiu, Alan Yuille, Gregory D. Hager. WACV 2022. [paper] [dataset]

• 🏃 Proactive Multi-Camera Collaboration for 3D Human Pose Estimation. Hai Ci*, Mickel Liu*, Xuehai Pan*, Fangwei Zhong, Yizhou Wang. ICLR 2023. [paper] [project]

• 📊 Monocular 3D Human Pose Estimation for Sports Broadcasts using Partial Sports Field Registration. Tobias Baumgartner, Stefanie Klatt. CVPR-W 2023. [paper ] [code]

• 📊 Hardware-accelerated Mars Sample Localization via Deep Transfer Learning from Photorealistic Simulations R. Castilla-Arquillo, C. J. P ́erez-del-Pulgar, G. J. Paz-Delgado and L. Gerdes IEEE Robotics and Automation Letters 2022. [paper] [code]

View Synthesis

• 📊 Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, Abhishek Kar. SIGGRAPH 2019. [paper][project]

Event Camera

• 📊 ESIM: an Open Event Camera Simulator. Henri Rebecq, Daniel Gehrig, Davide Scaramuzza. CoRL 2018. [paper][code]

• 📊 Insights into Batch Selection for Event-Camera Motion Estimation. Valerdi, Juan L., Chiara Bartolozzi, and Arren Glover. Sensors 2023. [paper]

Embodied Vision

Visual Navigation

• 🏃 An Improved Method Based on Deep Reinforcement Learning for Target Searching. Xiao Long Wei, Xiang Lin Huang, Tao Lu, Ge Ge Song. International Conference on Robotics and Automation Engineering (ICRAE) 2019. [paper]
• 🏃 Training an Agent to Find and Reach an Object in Different Environments using Visual Reinforcement Learning and Transfer Learning. Evelyn Conceição Santos Batista, Wouter Caarls, Leonardo A. Forero, Marco Aurélio C. Pacheco. ICAART 2021. [paper]
• 📊 🏃 Enhancing optical-flow-based control by learning visual appearance cues for flying robots. G. C. H. E. de Croon, C. De Wagter, T. Seidl. Nature Machine Intelligence 2021. [paper]
• 📊 🏃 Simultaneous localization and mapping architecture for small bodies and space exploration.. Bettens, A., Morrell, B., Coen, M., Wu, X., Gibbens, P., & Chamitoff, G.. Advances in Space Research 2024, 73(1), 1185-1197. [paper]

Active Object Tracking

• 🏃 End-to-end Active Object Tracking via Reinforcement Learning. Wenhan Luo*, Peng Sun*, Fangwei Zhong, Wei Liu, Tong Zhang, Yizhou Wang. ICML 2018. [paper] [project]

• 🏃 End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learning. Wenhan Luo*, Peng Sun*, Fangwei Zhong*, Wei Liu, Tong Zhang, Yizhou Wang. IEEE TPAMI 2019. [paper] [project]

• 🏃 AD-VAT: An Asymmetric Dueling mechanism for learning Visual Active Tracking. Fangwei Zhong, Wenhan Luo, Peng Sun, Tingyun Yan, Yizhou Wang. ICLR 2019. [paper] [code]

• 🏃 🔍 AD-VAT+: An Asymmetric Dueling Mechanism for Learning and Understanding Visual Active Tracking. Fangwei Zhong, Wenhan Luo, Peng Sun, Tingyun Yan, Yizhou Wang. IEEE TPAMI 2019. [paper] [code]

• 🏃 Pose-Assisted Multi-Camera Collaboration for Active Object Tracking. Jing Li*, Jing Xu*, Fangwei Zhong*, Xiangyu Kong, Yu Qiao, Yizhou Wang. AAAI 2020. [paper] [code]

• 🏃 🔍 Towards Distraction-Robust Active Visual Tracking. Fangwei Zhong, Wenhan Luo, Peng Sun, Tingyun Yan, Yizhou Wang. ICML 2021. [paper] [code] [project]

• 🏃 Anti-Distractor Active Object Tracking in 3D Environments. Mao Xi, Yun Zhou, Zheng Chen, Wengang Zhou, Houqiang Li. IEEE TCSVT 2022. [paper]

• 🏃 RSPT: Reconstruct Surroundings and Predict Trajectories for Generalizable Active Object Tracking. Fangwei Zhong*, Xiao Bi*, Yudi Zhang, Wei Zhang, Yizhou Wang. AAAI 2023. [paper] [project]

Multi-Agent Cooperation

• 🏃 Pose-Assisted Multi-Camera Collaboration for Active Object Tracking. Jing Li*, Jing Xu*, Fangwei Zhong*, Xiangyu Kong, Yu Qiao, Yizhou Wang. AAAI 2020. [paper] [code]

• 🏃 Proactive Multi-Camera Collaboration for 3D Human Pose Estimation. Hai Ci*, Mickel Liu*, Xuehai Pan*, Fangwei Zhong, Yizhou Wang. ICLR 2023. [paper] [project]

Agricultural/Plant Science

• 📊 A scalable pipeline to create synthetic datasets from functional–structural plant models for deep learning. Dirk Norbert Baker, Felix Maximilian Bauer, Mona Giraud, Andrea Schnepf, Jens Henrik Göbbert, Hanno Scharr, Ebba Hvannberg, Morris Riedel. in silico Plants, 6(1), diad022. [paper]
• 📊 Adapting Agricultural Virtual Environments in Game Engines to Improve HPC Accessibility. Dirk Norbert Baker, Felix Bauer, Andrea Schnepf, Hanno Scharr, Morris Riedel, Jens Henrik Göbbert, Ebba Hvannberg. [paper]