Weijie Su

Short Bio

Currently, Weijie Su is a Researcher at Shanghai AI Laboratory, headed by Prof. Jifeng Dai. He also works closely with Dr. Xizhou Zhu.

Weijie Su received Ph.D. degree from University of Science and Technology of China (USTC) in June 2024, under the supervision of Prof. Bin Li and Prof. Jifeng Dai. Before that, he received his B.S. degree from USTC in June 2019, majoring in Information and Computing Science (Computational Mathematics).

Research Interests Multimodal Foundation Model; Pre-training / Representation Learning; Visual Recognition; General Capable Agents.

If you are interested in an internship at Shanghai AI Laboratory related to my research field, please send me an email.

News

2023.05 Introducing GITM, a Generally Capable Agent that Fully Masters Minecraft. 2023.02 Our M3I Pre-training and SiameseIM are accepted by CVPR 2023. 2021.06 Presentation slides and video of our Deformable DETR on ICLR 2021 are available now. 2021.01 Deformable DETR is accepted for oral presentation by ICLR 2021. 2020.11 Code of Deformable DETR is now available at this github repository. 2020.05 Presentation slides and video of our VL-BERT on ICLR 2020 are available now. 2019.12 Our paper VL-BERT got accepted by ICLR 2020. 2019.11 Code for our paper VL-BERT has been made public available at github.

Experiences

2024.7 - Present Researcher, Shanghai AI Laboratory.

2024.1 - 2024.6 Research Intern, Shanghai AI Laboratory (Mentor: Prof. Jifeng Dai).

2020.1 - 2024.6 Research Intern, SenseTime Research (Mentor: Prof. Jifeng Dai and Dr. Xizhou Zhu).

2018.7 - 2019.9 Research Intern, Microsoft Research Asia (Mentor: Prof. Jifeng Dai).

2017.7 - 2018.6 Research Intern, GCL Lab at USTC (Mentor: Prof. Ligang Liu).

Honors

Outstanding Graduate, University of Science and Technology of China, 2024.

Future Star Award (未来之星奖), SenseTime, 2024.

National Scholarship (国家奖学金), Ministry of Education of the People's Republic of China, 2020.

Outstanding Graduate, University of Science and Technology of China, 2019.

Bronze Medal, The 30th Chinese Mathematical Olympiad (CMO), 2014。

Publications

(^† Equal Contribution)

Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning [Code] [Blog] Chenyu Yang^†, Xizhou Zhu^†, Jinguo Zhu^†, Weijie Su, Junjie Wang, Xuan Dong, Wenhai Wang, Lewei Lu, Bin Li, Jie Zhou, Yu Qiao, Jifeng Dai Arxiv Preprint, 2024.
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks [Code] [Blog] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, Jifeng Dai IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. (Oral)
Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory [Project Page] [Blog] [Demo] Xizhou Zhu^†, Yuntao Chen^†, Hao Tian^†, Chenxin Tao^†, Weijie Su^†, Chenyu Yang^†, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, Jifeng Dai Arxiv Preprint, 2023.
Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Information [Code] [Slides] [Presentation] Weijie Su^†, Xizhou Zhu^†, Chenxin Tao^†, Lewei Lu, Bin Li, Gao Huang, Yu Qiao, Xiaogang Wang, Jie Zhou, Jifeng Dai IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Siamese Image Modeling for Self-Supervised Vision Representation Learning Chenxin Tao^†, Xizhou Zhu^†, Weijie Su^†, Gao Huang, Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, Jifeng Dai IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Deformable DETR: Deformable Transformers for End-to-End Object Detection [Code] [Blog] [Slides] [Presentation] Xizhou Zhu^†, Weijie Su^†, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai International Conference on Learning Representations (ICLR), 2021. (Oral)
VL-BERT: Pre-training of Generic Visual-Linguistic Representations [Code] [Slides] [Presentation] Weijie Su^†, Xizhou Zhu^†, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai International Conference on Learning Representations (ICLR), 2020.