Weijie Su

Short Bio

Weijie Su is a final-year PhD student at University of Science and Technology of China (USTC), under the supervision of Prof. Bin Li and Prof. Jifeng Dai. Before that, he received his Bachelor’s degree from USTC in June 2019, majoring in Information and Computing Science (Computational Mathematics). Currently, he also works closely with Dr. Xizhou Zhu.
Research Interests Multimodal Foundation Model; Pre-training / Representation Learning; Visual Recognition; General Capable Agents.

If you are interested in an internship at Shanghai AI Laboratory / SenseTime Research related to my research field, please send me an email.

News

2023.05 Introducing GITM, a Generally Capable Agent that Fully Masters Minecraft. 2023.02 Our M3I Pre-training and SiameseIM are accepted by CVPR 2023. 2021.06 Presentation slides and video of our Deformable DETR on ICLR 2021 are available now. 2021.01 Deformable DETR is accepted for oral presentation by ICLR 2021. 2020.11 Code of Deformable DETR is now available at this github repository. 2020.05 Presentation slides and video of our VL-BERT on ICLR 2020 are available now. 2019.12 Our paper VL-BERT got accepted by ICLR 2020. 2019.11 Code for our paper VL-BERT has been made public available at github.

Experiences

2024.1 - Present Research Intern, Shanghai AI Laboratory (Mentor: Prof. Jifeng Dai).

2020.1 - Present Research Intern, SenseTime Research (Mentor: Prof. Jifeng Dai and Dr. Xizhou Zhu).

2018.7 - 2019.9 Research Intern, Microsoft Research Asia (Mentor: Prof. Jifeng Dai).

2017.7 - 2018.6 Research Intern, GCL Lab at USTC (Mentor: Prof. Ligang Liu).

Honors

Future Star Award (未来之星奖), SenseTime, 2024.

National Scholarship (国家奖学金), Ministry of Education of the People's Republic of China, 2020.

Outstanding Graduate, University of Science and Technology of China, 2019.

Bronze Medal, The 30th Chinese Mathematical Olympiad (CMO), 2014。

Publications

(^† Equal Contribution)

InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks [Code] [Blog] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, Jifeng Dai IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory [Project Page] [Blog] [Demo] Xizhou Zhu^†, Yuntao Chen^†, Hao Tian^†, Chenxin Tao^†, Weijie Su^†, Chenyu Yang^†, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, Jifeng Dai Arxiv Preprint, 2023.
Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Information [Code] [Slides] [Presentation] Weijie Su^†, Xizhou Zhu^†, Chenxin Tao^†, Lewei Lu, Bin Li, Gao Huang, Yu Qiao, Xiaogang Wang, Jie Zhou, Jifeng Dai IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Siamese Image Modeling for Self-Supervised Vision Representation Learning Chenxin Tao^†, Xizhou Zhu^†, Weijie Su^†, Gao Huang, Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, Jifeng Dai IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Deformable DETR: Deformable Transformers for End-to-End Object Detection [Code] [Blog] [Slides] [Presentation] Xizhou Zhu^†, Weijie Su^†, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai International Conference on Learning Representations (ICLR), 2021. (Oral)
VL-BERT: Pre-training of Generic Visual-Linguistic Representations [Code] [Slides] [Presentation] Weijie Su^†, Xizhou Zhu^†, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai International Conference on Learning Representations (ICLR), 2020.