In this talk, I will talk about the tools and predictors built in our group, EcoSystem, for machine learning applications. First, we will talk about Skyline (UIST’20), interactive in-editor computational performance profiling, visualization, and debugging tool for deep neural networks. Second, we will talk about Habitat (USENIX ATC’21), a runtime-based computational performance predictor for DNN training. Third, while Skyline proved to be efficient for supervised learning algorithms, system-level bottlenecks in unsupervised learning workloads (e.g., reinforcement learning) are poorly understood by ML developers. Our work seeks to understand fundamental structural differences in those workloads that make them inherently less compute-bound than supervised learning ones. To explain where training time is spent in such cases, we have proposed RL-Scope (MLSys’21), a cross-stack profiler that scopes low-level CPU/GPU resource usage to high-level algorithmic operations, and provides accurate insights by correcting for profiling overhead. Fourth, we have developed a new profiling tool, Daydream (USENIX ATC’20), to help programmers efficiently explore the efficacy of DNN optimizations without implementing them. I will show that Daydream is able to model most mainstream DNN optimization techniques and accurately predict the efficacy of optimizations that will result in significant performance improvements.
Gennady Pekhimenko is an Assistant Professor at the University of Toronto, CS department and (by courtesy) ECE department, where he is leading the EcoSystem (Efficient Computing Systems) group. Gennady is also a Faculty Member at Vector Institute and a CIFAR AI chair. Before joining Univ. of Toronto, he spent a year in 2017 at Microsoft Research in Redmond in the Systems Research group. He got his PhD from the Computer Science Department at Carnegie Mellon University in 2016. Gennady is a recipient of Amazon Machine Learning Research Award, Facebook Faculty Research Award, Connaught New Researcher Award, NVIDIA Graduate, Microsoft Research, Qualcomm Innovation, and NSERC CGS-D Fellowships. He was also recently inducted into the ISCA Hall of Fame and was a recipient of the MICRO Top Picks Award in 2021. His research interests are in the areas of systems, computer architecture, compilers, and applied machine learning.