Jiangmiao Pang

Hi! I am a Research Scientist and Team Lead at Shanghai AI Laboratory, where I lead the OpenRobotLab. We are dedicated to building Embodied AGI systems and empowering academia and industry through open-source initiatives. Before that, I was with MMLab@CUHK and worked with Prof. Dahua Lin.

Currently, we are focusing on multi-modal perception, interaction, and their conjunction with robot learning. We are building various real embodied systems so as expected to create positive impacts on humans. NOTE: We are looking for self-motivated researchers, engineers, interns, and joint-training PhD students in Multimodal Learning, Robot Learning, and Embodied AI. If you are interested or experienced in related fields, please feel free to directly contact me!

I was also part-time affiliated with OpenMMLab as a core contributor, where I led/co-authored several visual perception codebases, mainly including MMDetection , MMTracking , and MMDetection3D . My works have won 1st prizes at COCO Object Detection Challenge 2018 and 2019 (without external data), and 1st runner-up at Waymo 2D Object Tracking Challenge 2020.

Email / GitHub / Google Scholar / Twitter / Zhihu

News

[2024/01] The website is updated occasionally. For the most recent updates, please refer to my Google Scholar. : )
[2024/01] We present PointLLM & EmbodiedScan for Perception, UniHSI for Interaction, and HIMLoco for Control.
[2023/06] OC-SORT is accepted to CVPR 2023. It is recognized as one of the Most Influential CVPR Papers.
[2022/10] DfM and DenseSiam are accepted to ECCV 2022. DfM is selected as Oral Presentation.
[2022/06] Video K-Net is accepted to CVPR 2022 as Oral Presentation.

Selected Publications

--> For the full publication list, please refer to my Google Scholar.

† denotes corresponding author. * denotes equal contribution. My teammates/students/mentees are underline marked.

	GRUtopia: Dream General Robots in a City at Scale Hanqing Wang, Jiahe Chen, Wensi Huang, Qingwei Ben, Tai Wang, Boyu Mi, Tao Huang, Siheng Zhao, Yilun Chen, Sizhe Yang, Peizhou Cao, Wenye Yu, Zichao Ye, Jialun Li, Junfeng Long, Zirui Wang, Huiling Wang, Ying Zhao, Zhongying Tu, Yu Qiao, Dahua Lin, Jiangmiao Pang^† In Submission, 2024 project page / arXiv / code / bibtex
	Learning H-Infinity Locomotion Control Junfeng Long, Wenye Yu, Quanyi Li, Zirui Wang, Dahua Lin, Jiangmiao Pang^† In Submission, 2024 project page / arXiv / code / bibtex
	CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics Jiawei Gao, Ziqin Wang, Zeqi Xiao, Jingbo Wang, Tai Wang, Jinkun Cao, Xiaolin Hu, Si Liu^†, Jifeng Dai^†, Jiangmiao Pang^† In Submission, 2024 arXiv / code / bibtex
	MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations Ruiyuan Lyu, Tai Wang, Jingli Lin, Shuai Yang, Xiaohan Mao, Yilun Chen, Runsen Xu, Haifeng Huang, Chenming Zhu, Dahua Lin, Jiangmiao Pang^† In Submission, 2024 project page / arXiv / code / bibtex
	Grounded 3D-LLM with Referent Tokens Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, Jiangmiao Pang^† In Submission*, 2024 project page / arXiv / code / bibtex
	PointLLM: Empowering Large Language Models to Understand Point Clouds Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang^†, Dahua Lin European Conference on Computer Vision (ECCV), 2024, 2024 Oral Presentation with all Strong Accept recommendations project page / arXiv / code / bibtex
	GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction Xiao Chen, Quanyi Li, Tai Wang, Tianfan Xue^†, Jiangmiao Pang^† Computer Vision and Pattern Recognition (CVPR), 2024 project page / arXiv / bibtex
	EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, Xihui Liu, Cewu Lu, Dahua Lin, Jiangmiao Pang^† Computer Vision and Pattern Recognition (CVPR)*, 2024 project page / arXiv / code / bibtex
	Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response Junfeng Long, Zirui Wang, Quanyi Li, Jiawei Gao, Liu Cao, Jiangmiao Pang^† International Conference on Learning Representations (ICLR), 2024 project page / arXiv / code / bibtex
	Unified Human-Scene Interaction via Prompted Chain-of-Contacts Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, Jiangmiao Pang^† International Conference on Learning Representations (ICLR), 2024 Spotlight Presentation project page / arXiv / code / bibtex
	DORT: Modeling Dynamic Objects in Recurrent for Multi-Camera 3D Object Detection and Tracking Qing Lian, Tai Wang, Dahua Lin, Jiangmiao Pang^† Conference on Robot Learning (CoRL), 2023 arXiv / paper / code / bibtex
	OV-PARTS: Towards Open-Vocabulary Part Segmentation Meng Wei, Xiaoyu Yue, Wenwei Zhang, Shu Kong, Xihui Liu, Jiangmiao Pang^† Neural Information Processing Systems Datasets and Benchmarks Track, 2023 arXiv / code / bibtex
	Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking Jinkun Cao, Jiangmiao Pang, Xinshuo Weng, Rawal Khirodkar, Kris Kitani Computer Vision and Pattern Recognition (CVPR), 2023 Most Influential CVPR Papers arXiv / paper / code / bibtex
	Monocular 3D Object Detection with Depth from Motion Tai Wang, Jiangmiao Pang^†, Dahua Lin European Conference on Computer Vision (ECCV), 2022 Oral Presentation arXiv / code / Zhihu / bibtex
	K-Net: Towards Unified Image Segmentation Wenwei Zhang, Jiangmiao Pang, Kai Chen, Chen Change Loy Neural Information Processing Systems (NeurIPS), 2021 project page / arXiv / code / video / Zhihu / bibtex
	FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection Tai Wang, Xinge Zhu, Jiangmiao Pang, Dahua Lin International Conference on Computer Vision Workshops (ICCVW), 2021 Best Paper Award at ICCV 2021 workshop on 3DODI arXiv / code / slide / Zhihu / bibtex
	Quasi-Dense Similarity Learning for Multiple Object Tracking Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, Fisher Yu Computer Vision and Pattern Recognition (CVPR), 2021 Oral Presentation project page / arXiv / code / video / bibtex
	Libra R-CNN: Towards Balanced Learning for Object Detection Jiangmiao Pang, Kai Chen, Qi Li, Zhihai Xu, Huajun Feng, Jianping Shi, Wanli Ouyang, Dahua Lin Computer Vision and Pattern Recognition (CVPR), 2019 arXiv / code / Zhihu / bibtex
	Hybrid Task Cascade for Instance Segmentation Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Jianping Shi, Wanli Ouyang, Chen Change Loy, Dahua Lin Computer Vision and Pattern Recognition (CVPR), 2019 Winning entry at COCO Object Detection Challenge, ECCV 2018 arXiv / code / Zhihu / bibtex
	R²-CNN: Fast Tiny Object Detection in Large-scale Remote Sensing Images Jiangmiao Pang, Cong Li, Jianping Shi, Zhihai Xu, Huajun Feng IEEE Transactions on Geoscience and Remote Sensing (TGRS), 2019 ESI Highly Cited Paper arXiv / paper / bibtex

Selected Awards

World's Top 2% Scientist, Stanford University
Most Influential CVPR Papers, OC-SORT in CVPR 2023
Best Paper Award of Workshop on 3D Object Detection from Images, ICCV 2021
Doctoral Consortium Award, CVPR 2021
1st runner up at Waymo 2D Object Tracking Challenge, CVPR 2020
Outstanding Reviewer, ICCV 2019
1st prize at COCO Object Detection Challenge (without external data), ICCV 2019
1st prize at COCO Object Detection Challenge, ECCV 2018

Services

Conference Reviewer: CVPR, ICCV, ECCV, RSS, CoRL, NeurIPS, ICLR, ICML, AAAI, IROS, ACCV
Journal Reviewer: TPAMI, IJCV, TRO, TIP, RA-L, TGRS, Neurocomputing, Pattern Recognition Letters, Signal Processing Letters

Design and source code from Jon Barron's website

Jiangmiao Pang

News

Selected Publications

GRUtopia: Dream General Robots in a City at Scale

Learning H-Infinity Locomotion Control

CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics

MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations

Grounded 3D-LLM with Referent Tokens

PointLLM: Empowering Large Language Models to Understand Point Clouds

GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction

EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI

Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response

Unified Human-Scene Interaction via Prompted Chain-of-Contacts

DORT: Modeling Dynamic Objects in Recurrent for Multi-Camera 3D Object Detection and Tracking

OV-PARTS: Towards Open-Vocabulary Part Segmentation

Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking

Monocular 3D Object Detection with Depth from Motion

K-Net: Towards Unified Image Segmentation

FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection

Quasi-Dense Similarity Learning for Multiple Object Tracking

Libra R-CNN: Towards Balanced Learning for Object Detection

Hybrid Task Cascade for Instance Segmentation

R2-CNN: Fast Tiny Object Detection in Large-scale Remote Sensing Images

Selected Awards

Services

R²-CNN: Fast Tiny Object Detection in Large-scale Remote Sensing Images