Beidi Chen

Check InfiniAI Lab @ CMU for latest update.

I am starting as an Assistant Professor of Electrical and Computer Engineering at Carnegie Mellon University in Fall 2023.

My lab is recruiting for 23' (Application Link)! I am looking for students and interns who are excited to tackle efficiency problems in ML together from an algorithm, modeling, or system/hardware perspective. Ph.D., master's, undergraduate, and visiting students are welcome to reach out!

I am currently a visiting researcher at Meta/Facebook AI Research (FAIR). Previously, I was a postdoc researcher at Stanford working with Dr. Chris Ré. I received my Ph.D. in Computer Science from Rice University under the supervision of Dr. Anshumali Shrivastava in 2020. I received my B.S. from University of California, Berkeley in 2015. My mentors were Dr. Sara Alspaugh, Dr. Kaifei Chen and my advisor was Dr. Randy Katz. My research focuses on large-scale machine learning. Specifically, I design and optimize randomized algorithms (algorithm-hardware co-design) to accelerate large machine learning systems for real-world problems.

NEWS:

Research

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen. " H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models". NeurIPS 2023.

Yuandong Tian, Yiping Wang, Beidi Chen, Simon Shaolei Du. " Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer". NeurIPS 2023.

Stefano Massaroli, Michael Poli, Daniel Y Fu, Hermann Kumbong, David W. Romero, Rom Nishijima Parnichkun, Aman Timalsina, Quinn McIntyre, Beidi Chen, Atri Rudra, Ce Zhang, Christopher Ré, Stefano Ermon, Yoshua Bengio. " Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions". NeurIPS 2023.

Moses Charikar, Beidi Chen, Christopher Ré, Erik Waingarten. " Fast Algorithms for a New Relaxation of Optimal Transport ". Conference on Learning Theory (COLT) 2023.

Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, Beidi Chen. " Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time". ICML 2023. (Oral).

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang. " High-throughput Generative Inference of Large Language Models with a Single GPU". ICML 2023. (Oral). [ code ]

Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Ré, Ce Zhang. " CocktailSGD: Fine-tuning Foundation Models over 500Mbps Networks ". ICML 2023.

Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Ré, Ce Zhang. " Decentralized Training of Foundation Models in Heterogeneous Environments". NeurIPS 2022. (Oral).

Jue Wang, Binhang Yuan, Luka Rimanic, Yongjun He, Tri Dao, Beidi Chen, Christopher Ré, Ce Zhang. " Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees". NeurIPS 2022.

Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, Christopher Ré. " Monarch: Expressive Structured Matrices for Efficient and Accurate Training". ICML 2022. (Long Talk, Outstanding Paper Runner Up). [ code ][ poster ]

Beidi Chen*, Tri Dao*, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, Christopher Ré. " Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models". In Proceedings of International Conference on Learning Representations, ICLR 2022 (Spotlight). [video][ code ]

Zichang Liu, Zhaozhuo Xu, Alan Ji, Junyan Zhang, Jonathan Li, Beidi Chen, Anshumali Shrivastava. "HALOS: Hashing Large Output Space for Cheap Inference". In Proceedings of the 5rd Conference on Machine Learning and Systems, MLSys 2022.

Beidi Chen*, Tri Dao*, Eric Winsor, Zhao Song, Atri Rudra, Christopher Ré. " Scatterbrain: Unifying Sparse and Low-rank Attention Approximation ". In Neural Information Processing Systems, NeurIPS 2021. [ code ]

Beidi Chen, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, Christopher Re. "MONGOOSE: A Learnable LSH Framework for Efficient Neural Network Training". In Proceedings of International Conference on Learning Representations, ICLR 2021. (Oral) [video][ code]

Zhaozhuo Xu, Beidi Chen, Chaojian Li, Weiyang Liu, Le Song, Yingyan Lin, and Anshumali Shrivastava. " Locality Sensitive Teaching". In Neural Information Processing Systems, NeurIPS 2021.

Shabnam Daghaghi, Nicholas Meisburger, Mengnan Zhao, Beidi Chen, Tharun Medini, and Anshumali Shrivastava. "A Tale of Two Efficient and Informative Negative Sampling Distributions". In Proceedings of International Conference on Machine Learning, ICML 2021. (Long Talk)

Tharun Medini, Beidi Chen, Anshumali Shrivastava. "SOLAR: Sparse Orthogonal Learned and Random Embeddings". In Proceedings of International Conference on Learning Representations, ICLR 2021.

Beidi Chen, Weiyang Liu, Animesh Garg, Zhiding Yu, Anshumali Shrivastava, Jan Kautz, Anima Anandkumar. "Angular Visual Hardness". In Proceedings of the 37th International Conference on Machine Learning, ICML 2020. [Video]

Beidi Chen, Tharun Medini, James Farwell, Sameh Gobriel, Charlie Tai, Anshumali Shrivastava. "SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems". In Proceedings of the 3rd Conference on Machine Learning and Systems, MLSys 2020.

Beidi Chen, Yingchen Xu, and Anshumali Shrivastava. "LGD: Fast and Accurate Stochastic Gradient Estimation". In Neural Information Processing Systems, NeurIPS 2019. (LSH-Sampling Breaks the Computational Chicken-and-Egg Loop in Adaptive Stochastic Gradient Estimation in ICLR 2018 Workshop)

Beidi Chen, M. Sadegh Riazi, Anshumali Shrivastava, DanWallach, Farinaz Koushanfar. "Sub-linear Privacy-preserving Search with Untrusted Server and Semi-honest Parties". Manuscript.

Beidi Chen, Anshumali Shrivastava. " Revisiting Winner Take All (WTA) Hashing for Sparse Datasets". In Proceedings of the 34th Conference in Uncertainty in Artificial Intelligence, UAI 2018.

Beidi Chen, Anshumali Shrivastava, Rebecca C. Steorts. "Unique Entity Estimation with Application to the Syrian Conflict". The Annals of Applied Statistics 12.2 (2018). (Also Won IISA 2018 Best Student Paper in Applied Statistics with this paper.)

Kaifei Chen, Siyuan He, Beidi Chen, John Kolb, Randy H. Katz, David E. Culler. "BearLoc: A Composable Distributed Framework for Indoor Localization Systems". In Proceedings of the 2015 Workshop on IoT challenges in Mobile and Industrial Systems, IoT-Sys@MobiSys 2015, pages 7-12, May. 2015. Florence, Italy.

S. Alspaugh, Beidi Chen, Jessica Lin, Archana Ganapathi, Marti Hearst, and Randy Katz. "Analyzing Log Analysis: An Empirical Study of User Log Mining". In Proceedings of the 28th Large Installation System Administration Conference, LISA 2014. (Best Student Paper)

Experience

Research Intern

Microsoft Research, Redmond, WA
May 2019 - Present

Machine Learning Scientist Intern

Nvidia, Santa Clara, CA

Feb 2019 - May 2019

Applied Scientist Intern

A9.com, Palo Alto, CA

May 2018 - Dec 2018

Applied Scientist Intern

Amazon Web Services, Inc., Palo Alto, CA
March 2017 - June 2017

Software Engineering Intern

Apple Inc., Cupertino, CA

Feb 2015 - Aug 2015

Research and Development Intern

VMware Inc., Palo Alto, CA
May 2014 - Aug 2014

Software Engineering Intern

Broadcom Corporation, Sunnyvale, CA

May 2013 - Aug 2013

Education

Stanford University

Postdoc
Computer Science
August 2020 - Present

Rice University

Doctor of Philosophy
Computer Science
August 2015 - May 2020

University of California, Berkeley

Bachelor of Science
Electrical Engineering and Computer Science
August 2011 - May 2015

Wuhan Foreign Languages School

High School
Sep 2008 - June 2011

Honors and Awards

  • ICML 2022 Outstanding Paper Runner Up
  • MIT EECS Rising Stars 2021
  • UIUC EECS Rising Stars 2019
  • IISA 2018 Best Student Paper Award
  • Ken Kennedy Institute for Information Technology 2017 Fellowship
  • USENIX LISA 2014 Best Student Paper Award
  • ANITA BORG INSTITUTE GHC Scholarship 2014
  • Qualcomm Undergraduate Experiences in Science & Technology Scholar 2013

Invited Talks

LightOn AI Meetup [webpage]

VMware ML

TWIML AI Podcast [video]

Microsoft Machine Translation Group

Microsoft Rearch Talks [video]

Record Linkage workshop at CIMAT [webpage]

IISA2018 Conference

JSM 2018 Topic-Contributed Session

Activity and Service

Teaching

TA in Comp530 (Database System Implementation)
Rice University, Spring 2018

TA in Comp382 (Reasoning about Algorithms)
Rice University, Fall 2017, 2019

TA in Comp330 (Tools and Models For Data Science)
Rice University, Fall 2016, 2018

TA in EE122 (Introduction to Communication Networks)
UC Berkeley, Spring 2013, 2014

TA in EE40 (Introduction to Microelectronic Circuits)
UC Berkeley, Fall 2014

Service

Reviewer for Science Advances, NeurIPS, ICML, ICLR, AISTATS, AAAI, UAI, AISTATS

Activity

Stanford CS Undergraduate Mentoring Program

Women in Science and Engineering at Stanford

Member of CSters at Rice University

Member of Association of Women in EECS (AWE) at UC Berkeley