Shuaiwen Leon Song, Ph.D. (he/him/his)

SOAR Associate Professor (Tenured)
Director of Future System Architecture (FSA) Lab
School of Computer Science
The University of Sydney

Affiliated Professor
Electrical & Computer Engineering Department
University of Washington, Seattle
Affiliated Faculty, Sydney Quantum Academy

Google Scholar DBLP LinkedIn


Room 438, School of Computing
The University of Sydney
Sydney, NSW 2006, Australia


shuaiwen.song |at| sydney | dot | edu | dot | au


+61 2 8627 9613

USYD

Recent News

2023/3/20, I have been elevated to the associate editor for TPDS.

2022/12/15, I have been promoted .Thank you USYD!

2022/12/6, I have received IEEE mid-career award for scalable computing. Thanks IEEE!

2022/10/4, Google Brain collaboratiohn award. Thanks Google!

2022/9/5, our paper has been accepted to EuroSys 2023. Congrats everyone !

2022/8/23, serving PC for ISCA and NIPS 2023.

2022/5/31, I gave an invited talk at Microsoft on multi-dimensional and multi-scale machine learning system design with compilation techniques. Thanks for all the great discussion!

2022/4/10, our FSA lab website goes live at https://www.fsa-lab.org/ !

2022/3/25, I will be serving MICRO'22 PC. Please submit your best work!

2022/2/26, Congrats to Donglin to receive ASPLOS'22 student travel award.

2022/2/25, I have received Alibaba AIR faculty award to support my work on building multi-dimensional optimization framework for extreme large models on complex enterprise system architectures. Thanks Alibaba!

2022/1/15, collaborated paper with Google Brain has been accepted to MLSys 2022. Congrats to the team! It is Donglin's first leading author paper.

2021/12/4, I am awarded with SOAR faculty fellowship. Thank you for USYD for great support to my research.

2021/12/1, interview with ABC Radio on Metaverse.

2021/11/18, work with my visiting student on temporal graph processing has been accepted to ICDE'22. congrats to everyone!

2021/11/16, lossy compression for DNN paper has been accepted to VLDB'22.

2021/11/15, our paper on desgining compiler optimization for large-scale memory intensive computation in emerging ML models has been accepted to ASPLOS'22. Congrats to Dr. Zheng and the team!

2021/9/20, attending NYU entrepreneurship training workshop with my collaborator.

2021/7/24, congrats to my phd student Alan Robertson to make his first service as ASPLOS'22 ERC.

2021/7/15, I am elected to serve as the next area chair for Supercomputing 2022 (IEEE/ACM SC'22). Please submit your best works !

2021/7/15, our paper on exploration into designing general robust probalistic neural networks'patterns and optimization strategies for Safety-critial applications is accepted by MICRO21. Congrats everyone!

2021/7/11, my student Donglin is invited to give a talk at Google Brain research (hosted by Anna Goldie) on our recent collaborative work with Sara Hooker@Google!

2021/7/10, two papers on high performance computing are accepted to SC'21! Congrats everyone!

2021/5/22, our collaborated paper on the performance discrepancy between python and native libraries has been accepted FSE 2021. Congrats to Xu and his team!

2021/5/18, Google GCP award for Google Brain collaboration. Thanks Google!

2021/3/31, Efficient and Accurate End-to-End Deep LearningTraining via Fine-Grained Architecture-Preserving Pruning is accepted to ICS'21. Nice job everyone !

2021/3/24, giving guest lecture on future XR system design and optimizations for UW CSE548 on April 23rd. Come join in the discussion!˜

2021/3/6, our large LSTM training software-hardware design paper has been accepted to ISCA'21! Congrats to all the folks invovled from USYD FSA lab and UW Bespoke Group.

2021/3/3, I am excited to serve as ERC for MICRO21. Please submit your work!

2021/1/30, our HPCA'20 paper has been included as architecture research highlights for 2019˜2020 by IEEE Transactions on Computers (TC). Congrats everyone !

2021/1/20, I am excited to become ACM distinguished speaker.

2020/12/15, I have received Facebook Faculty Award. Thank you FB for supporting my research!

2020/12/8, invited to speak at AMD research Asia Tech talk. Super excited!

2020/11/19, Coallborative VR system design paper is accepted at ASPLOS'21!

2020/11/18, kick-start project with Google Brain !

2020/11/13, received Australian Research Council's (Austrilian NSF) discovery project for 3 years. Thanks ARC for supporting my research !

2020/11/05, PC @ ISCA'21. Please submit your best work !

2020/09/20, invited to speak at Monash University Engineering Event on Oct 13th.

2020/09/17, PC @ SC'21. Please submit your best work !

2020/09/17, PC @ HPDC'21, machine learning track. Please submit your best work !˜

2020/7/1, I am awarded the 2020 Austrilia's Most Innovative Engineers award. Thanks for USYD's tremendous support!

2020/7/1, chairing architecture track@IPDPS'21

2020/6/4, ERC@HPCA-27

2020/5/1, Associate editor@ IEEE transactions on Sustainable Computing

2020/4/6, ERC@MICRO-53

2020/3/25, IPDRM workshop proposal accepted @SC'20!

2020/3/15, Awarded for GCP and TPU Pod V2 and V3 resource for my VR research! Thank you Google!

2020/3/04, panelist for MLperf workshp at System ML conference at Austin Texas.

2020/2/28, presenting VR research work @ Google Platform research.

2020/2/15, review board@TPDS.

2019/11/10, visiting CS@UNSW.

2019/11/2, visiting virtual reality lab@ UNSW art school.

2019/11/1, ERC@ISCA'20 .

2019/10/15, CapsuleNet PIM design paper accepted to HPCA-26.

2019/10/1: Area chair@ICPP'20.

2019/9/20, PC@ICDCS'20, HPDC'20.

2019/9/28, start my academic life @ U of Sydney. Love our beautiful downtown campus in beautiful Sydney!˜

2019/8/15, visiting professor Yan at ECE@Rice and give a talk.

2019/8/1, soft tensorcore for approximate neural nets is accepted to Supercomputing'19.

2019/6/10, ISCA paper is presented by my postdoc Chenhao@FRC.

2019/05/01, formally affiliated with UW ECE department as affiliated professor.

2019/4/2, Panelist for Berkeley lab AI workshop.

2019/3/28, PC@PPoPP'20.

2019/3/15, future cloud server design for VR services is accepted to ISCA'19.

2019/2/14, granted $350k as PI from DoD/DoE HPDA project to develop highly scalable BLAS library on many-accelerator systems.

2019/1/9, PC@PACT'19.

2018/11/8, co-design for enabling motion-anomaly free virtual reality devices accepted to HPCA-25.

2018/9/13, Tartan benchmark for multi-GPU evaluation nominated for best paper finalists at IISWC'18.

2018/6/29, serving 2018 IEEE TCHPC early career award selection committee.

2018/5/16, R&D 100 award judge.

2018/4/10, WarpConsolidation model accepted to ICS'18@Beijing.

2018/3/14, U.S. DOE research highlight: Unlocking On-Package Memory’s Effects on High-Performance Computing’s Scientific Kernels

2018/2/8, invited to serve on review board for Concurrency and Computation, Practice and Experience (CCPE) journal.

2018/2/6, Four papers presented at HPCA-PPoPP-CGO 2018: SuperNeuron (PPoPP), CUDAAdvisor (CGO), Low-cost real-time memory profiling (CGO), and Efficient Approximate design for 3D rendering architecture (HPCA).

2018/1/24, serving PC for PPoPP'19.

2018/1/24, participating DOE ASCR Heterogeneous workshop to help ASCR draft strategies for beyond exascale computing.

2018/1/13, CSE@UW project meeting with Micheal's group.

2017/11/15, receiving IEEE early career award for HPC @Supercomputing'17.

2017/11/15, presenting our paper@Supercomputing'17 in best paper session.

2017/10/27, giving a talk @ MSR distributed computing group.

2017/10/02, giving a talk @ Intel research portland.

2017/8/15, invited talk @ SIAM PP18 in Tokyo.

2017/8/3, ASPLOS'17 paper receives HiPEAC paper award.

2017/7/2, paper accepted to MICRO-50.


"It is never too late to become what you might have been." -- George Elliot

Who am I?

I am the SOAR associate professor (tenured) at the School of Computer Science at University of Sydney, and I direct the Future System Architecture Lab (FSA). I am also a Senior Principal Scientist at Microsoft, leading DeepSpeed4Science initative and other pathfinding projects at DeepSpeed. I hold an Affiliated Professor position with University of Washington . Prior to my appointment at University of Sydney, I worked for U.S. Department of Energy Lab for five years as a senior staff scientist and technical lead. In 2017 and 2022, I was awarded with IEEE HPC early career award and IEEE mid-career award for scalable computing, respectively. I was also awarded with 2022 Alibaba Gloab Faculty Award (AIR), 2022 SOAR Fellowship, 2022/2021 Google Brain Collaboration Award, 2021 Facebook faculty award, 2020 Australia's Most Innovative Engineer Award and a ACM distinguished speaker. I am also a Lawrence Scholar and a recipient of Paul E. Torgersen Excellent research award, a 2018 DOE pathway to excellence research award, 2015 and 2017 DOE PNNL lab outstanding research award, two Supercomputing (IEEE/ACM SC) best paper runners-up (2015 and 2017), and 2017 HiPEAC paper award. I have published in the top HPC and computer architecture conferences including ISCA, HPCA, ASPLOS, MICRO, and Supercomputing (SC). My past and current research has been supported by Microsoft, Google, NVIDIA, Intel, U.S. government agencies including DOE office of science (ASCR), DoD, DARPA and DoE Lab LDRD, and Australian Research Council (ARC). During my tenure at PNNL, I led two DOE lab LDRD projects on AI-driven architecture design and large-scale data analytics acceleration. At University of Sydney, I run Future System Architecture (FSA) Lab with my wonderful students. Currently, we are actively working with our collaborators from UW Seattle, UT Austin, NYU, Google Brain, Facebook Reality Lab and Alibaba Research. In my spare time, I am also consulting for tech startups.

I do research at the boundary of system software and hardware, breaking down abstraction barriers, and rethinking the hardware–software interface. I have a particular interest of holistic system design and software-hardware co-design. More broadly, my expertise lies in the general areas of computer system architecture and high performance computing (HPC). I hold the strong belief that future beyond Moore’s system architectures will become increasingly heterogeneous which demands new software (programming system, compiler, runtime) and hardware design paradigm to accommodate such complex many-accelerator integrated systems. As a computer system researcher, I am inspired to push the concept of co-design to create efficient and scalable solutions for emerging systems and applications, including future planet-scale Extended-Reality (XR) system, System ML and AI-driven System Designs, and even future quantum accelerator based heterogeneous architectures., In the recent years, with my amazing students and collaborators, we have published some of the first papers (HPCA'17, HPCA'18, HPCA'19, ISCA'19, ASPLOS'21, HPCA'23) on future VR system characterizations and system-level design & optimizations (including both multi-accelerator based HMD SoC and cloud server designs) in the field of computer architecture. Additionally, my recent work in industry research on machine learning system optimizations and scalability are being delopyed to real-world large-scale enterprise usage for millions of users.

  • *** I am currently on leave at Microsoft Seattle. If you need immediate response, please contact me at leonsong@microsoft.com

  • Research Interests

    • Hardware-Software System Co-design
    • Emerging architectures and systems (e.g., heterogenous architectures, emerging many-core accelerators, novel memory technologies and quantum architectures)
    • High Performance Computing (HPC)
    • Machine Learning System and SystemML
    • Metaverse System Design: designing future planet-scale XR system and exploring its efficiency, scalability, quality of experience and social impact.

    Some Recent Research Highlights (Full Publication List;FSA Group)


    Recent Awards and Recognition


    Professional Services and Activities

      Organizing Committee

    • Area chair: Architecture and Networks,Supercomputing (SC) 2022
    • Area chair: Architecture (IPDPS 2021)
    • Area chair: Performance (ICPP2020)
    • Best paper selection panel chair: IPDPS21
    • ACM ASPLOS'18, chair for poster and ACM student research compeition (SRC)
    • Publicity chair ,ACM ICS'22
    • Publicity Chair, ACM HPDC, 2016, 2017, 2018 and 2019
    • Publication chair, 2018 ACM International Conference on Supercomputing (ICS)
    • Workshop chair and steering committee, International Workshop on Emerging Parallel and Distributed Runtime Systems and Middleware (Supercomputing)
    • Workshop chair, International workshop on High-Performance, Power-Aware Computing (HPPAC)
    • Journal Editorial

    • Editor, Elsevier High-Confidence Computing.
    • Associate editor: IEEE transaction on sustainable computing.
    • Reivew Board: IEEE Transactions on Parallel and Distributed Systems (TPDS)
    • Review Board: Journal for Concurrency and Computation, Practice and Experience (CCPE)
    • Recent Program Committee

    • PC, the 50th IEEE/ACM International Symposium on Computer Architecture (ISCA), 2023
    • PC, Neural Information Processing Systems (NIPS) 2023.
    • PC, ACM SIGPLAN Annual Symposium Principles and Practice of Parallel Programming (PPoPP),2023
    • PC, 54th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2022
    • ERC, 49th International Symposium on Computer Architecture (ISCA), 2022
    • Session chair, 48th International Symposium on Computer Architecture (ISCA), 2021
    • ERC, 54th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2021
    • PC, 48th International Symposium on Computer Architecture (ISCA), 2021
    • PC, 2021 IEEE/ACM Supercomputing (SC)
    • PC, ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC), 2021
    • ERC, 27th IEEE international symposium on High-Performance Computer Architecture (HPCA),2021
    • ERC, IEEE/ACM International Symposium on Microarchitecture (MICRO-53), 2020
    • ERC, 47th International Symposium on Computer Architecture (ISCA), 2020
    • PC, ACM SIGPLAN Annual Symposium Principles and Practice of Parallel Programming (PPoPP), 2019˜2020
    • PC, ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC), 2016˜2018, 2020, 2021
    • PC, IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2015˜2017
    • PC, IEEE/ACM International Conference on Parallel Architectures and Compilation Techniques (PACT), 2019
    • PC,ACM International Conference on Supercomputing (ICS), 2017,2021
    • PC, IEEE International Conference on Distributed Computing Systems (ICDCS), 2020
    • PC, IEEE International Parallel & Distributed Processing Symposium (IPDPS), 2016˜2018,2020,2021
    • NSF proposal panelist, 2015
    • R&D award judge, 2018˜2019
    • Journal Reviewer

    • ACM Transactions on Computer Systems (TOCS), 2020
    • ACM Transactions on Architecture and Code Optimization (TACO), 2018
    • IEEE Transactions on Parallel and Distributed Systems (TPDS), 2014˜2020
    • IEEE Transactions on Computers (TC), 2015˜2016
    • IEEE Transactions on Sustainable Computing, 2019˜2020
    • Journal of Parallel and Distributed Computing - Elsevier (JPDC)
    • The International Journal of High Performance Computing (IJHPCA)
    • The Journal of Supercomputing Elsevier (JOS)
    • Parallel Computing-Elsevier (ParCo)

    Recent Talks

    • 2023 DeepSpeed4Science global announcement [WinBuzzer] [Metaverse Post] [Microsoft Research Blog] [今日头条] [知乎]
    • 2023 Keynote speaker for IPDRM workshop at Supercomputing'23
    • 2023 Keynote speaker for TIES workshop@PEARC'23
    • 2023 Opening keynote for DoE MODSIM workshop 2023 at University of Washington
    • "Designing Future Planet-Scale XR System”, UW Seattle CSE guest lecture (April 2021), AMD Research Asia, Future System Research Trends @HPDC'2021, Monash University Engineering Society (2020), Digital Science Initiative (2020), Googel Research (Feb 2020) and Rice ECE (August 2019).
    • "Unleashing the Power of Holistic System Design for Modern Emerging Workloads", Intel Research, University of Virginia, U of Minnesota, UC Irvine (ECE), U of Sydney, U of Pittsburgh, U of Connecticut, Rutgers U (ECE), UC Riverside, NUS, 2019.
    • "Designing Future Non-Von Neumann Architectures for Big Data Analytics", multi-agency review, DOE/DOD HPDA project, 2018 and 2019.
    • Invited to Microsoft Research Faculty Summit 2018.
    • “Advanced HPC System Research: State-of-the-Practice and Future Roadmap”, invited Lecture, Bio-engineering Department, Duke University, Nov 2018.
    • “Exploring and Analyzing the Real Impact of Modern On-Package Memory on HPCScientific Kernels”, conference talk, Supercomputing’18, Denver, Colorado, Nov 15th, 2017.
    • “Binarzied Software Tensor Core”, invited talk, Microsoft Research, Redmond, WA, Nov 3rd, 2017.
    • “Software-Hardware Co-Design for Future Complex HPC Architectures: DOE perspectives”, invited talk, Intel Research Lab, Hillsboro, OR, Oct 30th, 2017.
    • Invited to Microsoft Research Faculty Summit 2017.
    • “Whither Advanced GPU Research in HPC? Where We Are, Where We are Going”, Invited Speaker, Tsinghua University, CS department; Peking University, ECE department; Chinese Academy of Science, Institute of Computing Technologies, April 14th, 2017.
    • “Locality-Aware CTA Clustering For Modern GPUs", conference talk, ACM ASPLOS’17, Xi’An, China, April 2017.

    Publicity

    • 2022 IEEE mid-career award for Scalable Computing.
    • 2022 Google Brain Collaboration award.
    • 2022 AIR Faculty Award
    • SOAR Faculty Fellowship Award.
    • ACM distinguished speker.
    • 2021 Facebook Faculty Award.
    • 2021 Google Brain Colalboration Award.
    • 2020 Australia's Most Innovative Engineer Award; University of Sydney News: https://www.sydney.edu.au/news-opinion/news/2020/07/06/engineers-australia.html.
    • DOE featured research highlight: [DoECheck the news!]
    • Digital Trends (Boosting graphics performance through processing in-memory): [digital-trendsCheck the news!]
    • Yahoo Tech: [yahooCheck the news!]
    • Bit-Tech: [bit-techCheck the news!]
    • PNNL featured research news: http://www.pnnl.gov/news/release.aspx?id=4385
    • PNNL research highlight: Changing the game, http://www.pnnl.gov/science/highlights/highlight.asp?id=4495
    • PNNL research spotlight: Improving computing system performance, http://www.pnnl.gov/science/highlights/highlight.asp?id=4238, March 2016.
    • "Powering Down", article about my work on power management on large-scale system, published on DOE DEIXIS magazine featured article, [Check the news!], written by Monte Basgall.
    • PNNL Science Research Highlight: Energy Star: Novel models of HPC systems depict the interplay between energy efficiency and resilience", Link: http://www.pnnl.gov/science/highlights ,2015.
    • PNNL ACMDD staff award and honors: PNNL HPC Staff Take on Energy E.ciency, Resilience at scale", Link: http://www.pnnl.gov/science/highlights, 2015.
    • PNNL ACMDD staff award and honors: PNNL HPC Staff research: Improving Energy, Performance Efficiency for High Performance Computing", Link: http://www.pnnl.gov/science/highlights, 2015.
    • Current Magazine: \HPC system modeling: Depicting interplay between energy efficiency and resilience", June 2015 issue.
    • InsideHPC: "PNNL looks at undervolting to meet exascale goals", written by Rich Bruecker: [insider-hpcCheck the news!]
    • HPC Wire Top Feature Article: "Tackling the Power and Energy wall for Future HPC Systems", [hpc-wireCheck the news!], Dec, 2013.

    Current Students and Alumni

    • Please check our group site at FSA-Lab

    Past and Current Sponsors and Collaborators

    MSFT MSFT FB ARC UW Nvidia Intel DoE DoE Office of Science PNNL Sydney Nano instituten Department of Defense DARPA

    Contact


    Room 438, School of Computing
    The University of Sydney
    Camperdown NSW 2006, Australia


    shuaiwen.song |at| sydney | dot | edu | dot | au


    +61 2 8627 9613