profile_picture
Karsten Kreis
Senior Research Scientist, NVIDIA
kkreis [at] nvidia [dot] com

I am a Senior Research Scientist in the fundamental generative AI research team of NVIDIA Research. I am interested both in generative AI algorithm development and in applying deep generative models in areas such as representation learning, graphics, content creation and digital artistry. I am also interested in research that takes inspirations from physics to improve machine learning techniques as well as in applying state-of-the-art generative learning methods to problems in the natural sciences.

I am trained as a physicist and completed my master’s in quantum information theory. For my Ph.D. in computational and statistical physics, I developed multiscale models and sampling algorithms for molecular dynamics simulation. Before joining NVIDIA, I worked on deep generative learning at D-Wave Systems and I co-founded Variational AI, a startup leveraging generative modeling for drug discovery.

Publications

*,† indicate equal contribution / co-first authorship.
Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models
Computer Vision and Pattern Recognition (CVPR), 2024 (Highlight)
Huan Ling* , Seung Wook Kim* , Antonio Torralba , Sanja Fidler , Karsten Kreis
arxiv project page

Outdoor Scene Extrapolation with Hierarchical Generative Cellular Automata
Computer Vision and Pattern Recognition (CVPR), 2024 (Highlight)
Dongsu Zhang , Francis Williams , Zan Gojcic , Karsten Kreis , Sanja Fidler , Young Min Kim , Amlan Kar
coming soon

WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space
International Conference on Learning Representations (ICLR), 2024
Katja Schwarz , Seung Wook Kim , Jun Gao , Sanja Fidler , Andreas Geiger , Karsten Kreis
arxiv proceedings project page

A Unified Approach for Text- and Image-guided 4D Scene Generation
arXiv preprint, 2023
Yufeng Zheng , Xueting Li , Koki Nagano , Sifei Liu , Karsten Kreis , Otmar Hilliges , Shalini De Mello
arxiv project page

TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models
International Conference on Computer Vision (ICCV), 2023 (Oral Presentation)
Tianshi Cao , Karsten Kreis , Sanja Fidler , Nicholas Sharp* , Kangxue Yin*
arxiv proceedings project page

DreamTeacher: Pretraining Image Backbones with Deep Generative Models
International Conference on Computer Vision (ICCV), 2023
Daiqing Li* , Huan Ling* , Amlan Kar , David Acuna , Seung Wook Kim , Karsten Kreis , Antonio Torralba , Sanja Fidler
arxiv proceedings project page

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
Computer Vision and Pattern Recognition (CVPR), 2023
Andreas Blattmann* , Robin Rombach* , Huan Ling* , Tim Dockhorn* , Seung Wook Kim , Sanja Fidler , Karsten Kreis
arxiv proceedings project page

NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models
Computer Vision and Pattern Recognition (CVPR), 2023
Seung Wook Kim* , Bradley Brown* , Kangxue Yin , Karsten Kreis , Katja Schwarz , Daiqing Li , Robin Rombach , Antonio Torralba , Sanja Fidler
arxiv proceedings project page

Magic3D: High-Resolution Text-to-3D Content Creation
Computer Vision and Pattern Recognition (CVPR), 2023 (Highlight)
Chen-Hsuan Lin* , Jun Gao* , Luming Tang* , Towaki Takikawa* , Xiaohui Zeng* , Xun Huang , Karsten Kreis , Sanja Fidler† , Ming-Yu Liu† , Tsung-Yi Lin
arxiv proceedings project page

Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion
Computer Vision and Pattern Recognition (CVPR), 2023
Davis Rempe* , Zhengyi Luo* , Xue Bin Peng , Ye Yuan , Kris Kitani , Karsten Kreis , Sanja Fidler , Or Litany
arxiv proceedings project page

Differentially Private Diffusion Models
Transactions on Machine Learning Research (TMLR), 2023
Tim Dockhorn , Tianshi Cao , Arash Vahdat , Karsten Kreis
arxiv code proceedings project page

Score-based Diffusion Models in Function Space
arXiv preprint, 2023
Jae Hyun Lim* , Nikola B. Kovachki* , Ricardo Baptista* , Christopher Beckham , Kamyar Azizzadenesheli , Jean Kossaifi , Vikram Voleti , Jiaming Song , Karsten Kreis , Jan Kautz , Christopher Pal , Arash Vahdat , Anima Anandkumar
arxiv

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
arXiv preprint, 2022
Yogesh Balaji , Seungjun Nah , Xun Huang , Arash Vahdat , Jiaming Song , Qinsheng Zhang , Karsten Kreis , Miika Aittala , Timo Aila , Samuli Laine , Bryan Catanzaro , Tero Karras , Ming-Yu Liu
arxiv project page

Latent Space Diffusion Models of Cryo-EM Structures
Machine Learning for Structural Biology Workshop, NeurIPS, 2022 (Oral Presentation)
Karsten Kreis* , Tim Dockhorn* , Zihao Li , Ellen Zhong
arxiv

GENIE: Higher-Order Denoising Diffusion Solvers
Neural Information Processing Systems (NeurIPS), 2022
Tim Dockhorn , Arash Vahdat , Karsten Kreis
arxiv code proceedings project page

LION: Latent Point Diffusion Models for 3D Shape Generation
Neural Information Processing Systems (NeurIPS), 2022
Xiaohui Zeng , Arash Vahdat , Francis Williams , Zan Gojcic , Or Litany , Sanja Fidler , Karsten Kreis
arxiv code proceedings project page

Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps
Computer Vision and Pattern Recognition (CVPR), 2022 (Oral Presentation)
Seung Wook Kim , Karsten Kreis , Daiqing Li , Antonio Torralba , Sanja Fidler
arxiv proceedings project page

BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations
Computer Vision and Pattern Recognition (CVPR), 2022
Daiqing Li , Huan Ling , Seung Wook Kim , Karsten Kreis , Adela Barriuso , Sanja Fidler , Antonio Torralba
arxiv code proceedings project page

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion
International Conference on Learning Representations (ICLR), 2022 (Spotlight Presentation)
Tim Dockhorn , Arash Vahdat , Karsten Kreis
arxiv code proceedings project page

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs
International Conference on Learning Representations (ICLR), 2022 (Spotlight Presentation)
Zhisheng Xiao , Karsten Kreis , Arash Vahdat
arxiv code proceedings project page

Score-based Generative Modeling in Latent Space
Neural Information Processing Systems (NeurIPS), 2021
Arash Vahdat* , Karsten Kreis* , Jan Kautz
arxiv code proceedings project page

Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence
Neural Information Processing Systems (NeurIPS), 2021
Tianshi Cao , Alex Bie , Arash Vahdat , Sanja Fidler , Karsten Kreis
arxiv code proceedings project page

EditGAN: High-Precision Semantic Image Editing
Neural Information Processing Systems (NeurIPS), 2021
Huan Ling* , Karsten Kreis* , Daiqing Li , Seung Wook Kim , Antonio Torralba , Sanja Fidler
arxiv code nvidia blog post proceedings project page two minute papers

ATISS: Autoregressive Transformers for Indoor Scene Synthesis
Neural Information Processing Systems (NeurIPS), 2021
Despoina Paschalidou , Amlan Kar , Maria Shugrina , Karsten Kreis , Andreas Geiger , Sanja Fidler
arxiv code proceedings project page

Causal Scene BERT: Improving object detection by searching for challenging groups of data
International Conference on Computer Vision (ICCV), 2nd AVVision Workshop, 2021
Cinjon Resnick , Or Litany , Amlan Kar , Karsten Kreis , James Lucas , Kyunghyun Cho , Sanja Fidler
arxiv proceedings

Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
Computer Vision and Pattern Recognition (CVPR), 2021
Daiqing Li , Junlin Yang , Karsten Kreis , Antonio Torralba , Sanja Fidler
arxiv code proceedings project page

Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes
Computer Vision and Pattern Recognition (CVPR), 2021 (Oral Presentation)
Towaki Takikawa* , Joey Litalien* , Kangxue Yin , Karsten Kreis , Charles Loop , Derek Nowrouzezahrai , Alec Jacobson , Morgan McGuire , Sanja Fidler
arxiv code proceedings project page

VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
International Conference on Learning Representations (ICLR), 2021 (Spotlight Presentation)
Zhisheng Xiao , Karsten Kreis , Jan Kautz , Arash Vahdat
arxiv code proceedings

Variational Amodal Object Completion
Neural Information Processing Systems (NeurIPS), 2020
Huan Ling , David Acuna , Karsten Kreis , Seung Wook Kim , Sanja Fidler
proceedings project page

ESPResSo++ 2.0: Advanced methods for multiscale molecular simulation
Computer Physics Communications (CPC), 2019
Horacio V. Guzman , Nikita Tretyakov , Hideki Kobayashi , Aoife C. Fogarty , Karsten Kreis , Jakub Krajniak , Christoph Junghans , Kurt Kremer , Torsten Stuehn
paper

From classical to quantum and back: Hamiltonian adaptive resolution path integral, ring polymer, and centroid molecular dynamics
Journal of Chemical Physics (JCP), 2017
Karsten Kreis , Kurt Kremer , Raffaello Potestio , Mark E. Tuckerman
arxiv paper

The relative entropy is fundamental to adaptive resolution simulations
Journal of Chemical Physics (JCP), 2016
Karsten Kreis , Raffaello Potestio
paper

Adaptive Resolution Simulations with Self-Adjusting High-Resolution Regions
Journal of Chemical Theory and Computation (JCTC), 2016 (Featured on the Journal Cover)
Karsten Kreis , Raffaello Potestio , Kurt Kremer , Aoife C. Fogarty
journal cover paper

From Classical to Quantum and Back: A Hamiltonian Scheme for Adaptive Multiresolution Classical/Path-Integral Simulations
Journal of Chemical Theory and Computation (JCTC), 2016
Karsten Kreis , Mark E. Tuckerman , Davide Donadio , Kurt Kremer , Raffaello Potestio
paper

Advantages and challenges in coupling an ideal gas to atomistic models in adaptive resolution simulations
The European Physical Journal Special Topics (EPJ ST), 2015
Karsten Kreis , Aoife C. Fogarty , Kurt Kremer , Raffaello Potestio
arxiv paper

A unified framework for force-based and energy-based adaptive resolution simulations
Europhysics Letters (EPL), 2014
Karsten Kreis , Davide Donadio , Kurt Kremer , Raffaello Potestio
paper

Classifying, quantifying, and witnessing qudit-qumode hybrid entanglement
Physical Review A (PRA), 2012
Karsten Kreis , Peter van Loock
arxiv paper


Theses

Advanced Adaptive Resolution Methods for Molecular Simulation
Ph.D. Thesis, 2018
Carried out at the Max Planck Institute for Polymer Research and at New York University
pdf
Characterizing And Exploiting Hybrid Entanglement
Master Thesis ("Diplomarbeit"), 2011
Carried out at the Max Planck Institute for the Science of Light
arxiv

Prizes, Awards and Scholarships

  • Highlighted Reviewer at the International Conference on Learning Representations (ICLR), 2022 --- Outstanding Reviewer (Top 10%) at the International Conference on Machine Learning (ICML), 2022 --- Notable Reviewer (Top 1%) at the International Conference on Learning Representations (ICLR), 2023
    •
  • Prize of the Department of Physics, Mathematics and Computer Science of the Johannes Gutenberg University Mainz for an outstanding dissertation, 2019
    •
  • Ph.D. thesis awarded highest grade "summa cum laude", 2018
    •
  • Moore/Sloan & Washington Research Foundation Innovation in Data Science Postdoctoral Fellowship at the University of Washington, Seattle (declined), 2017
    •
  • Ph.D. Scholarship of the "Graduate School of Excellence Materials Science in Mainz", 2013-2016
    •
  • Poster prize for the best poster at the “Meet-Your-Collegue-Day 2013” of the Max Planck Institute for Polymer Research, 2013
    •
  • "Ohm Prize" of the Department for Physics of the Friedrich-Alexander University Erlangen-Nuremberg for an outstanding diploma thesis, 2011
    •
  • "Erasmus Grant" to study abroad at Imperial College London, 2007-2008
    •
  • "Book Prize" of the German Physical Society for excellent achievements in the subject of physics in high school, 2005
    •

    Miscellaneous

    Tutorials

  • Tutorial “Diffusion Models: A Generative AI Big Bang” at NVIDIA GTC 2024.
    •
  • Tutorial “Latent Diffusion Models: Is the Generative AI Revolution Happening in Latent Space?" at Neural Information Processing Systems (NeurIPS), 2023.
    •
  • Course on Diffusion Models at SIGGRAPH, 2023.
    •
  • Tutorial “Denoising Diffusion-based Generative Modeling: Foundations and Applications” at the Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
    •
  • Two introductory blog posts (part 1 and part 2) about our three recent works on Latent Score-based Generative Models, Denoising Diffusion GANs and Critically-Damped Langevin Diffusion.
    •

    Invited Talks

  • Apr 2024: “Visual Generative AI with Diffusion Models: From Static Pixels to Video, 3D and 4D Synthesis”, at the RISE Learning Machines seminars.
    •
  • Feb 2024: “Visual Generative AI: From its Beginnings to Modern Diffusion Models for Image, Video, 3D and 4D Synthesis”, lecture at the University of Toronto and Vector Institute (slides).
    •
  • Dec 2023: “Beyond Static Pixels: From Video to Text-to-4D Synthesis with Diffusion Models” at the NeurIPS 2023 Workshop on Diffusion Models.
    •
  • Nov 2023: “Kuenstliche Intelligenz: Ueberblick und Trends” at Mercedes-Benz, Bremen.
    •
  • Oct 2023: “Generative Diffusion Models: Inspirations from Physics and Applications in Digital Content Creation” at Structured Learning Workshop, Chalmers AI.
    •
  • Aug 2023: “Diffusion Models: From Algorithms to Image, Video, 3D and Cryo-EM Structure Synthesis” at MIT CSAIL.
    •
  • Aug 2023: “Generative Diffusion Models: Inspirations from Physics and Applications in Digital Content Creation” at IAIFI Summer Workshop.
    •
  • Jul 2023: “Diffusion Models: From Foundations to Image, Video and 3D Content Creation” at the University of British Columbia.
    •
  • Jun 2023: “Image, Video and 3D Content Creation with Diffusion Models” at the Vision Models Forum at the 2023 BAAI Conference.
    •
  • Jun 2023: “From Images and Video to 3D Shapes: Content Creation with Diffusion Models” at the Conference on Robotics and Vision 2023.
    •
  • Apr 2023: “Diffusion Models: From Foundations to Image, Video and 3D Content Creation” at Simon Fraser University.
    •
  • Apr 2023: “Diffusion Models: From Foundations to Image, Video and 3D Content Creation” at the Computer Vision Group at the University of Illinois Urbana-Champaign.
    •
  • Jan 2023: “Latent Space Diffusion Models of Cryo-EM Structures” at Variational AI.
    •
  • Dec 2022: “Advanced Diffusion Models: Accelerated Sampling, Smooth Diffusion, and 3D Shape Generation” at the Computer Vision Group at the University of Bern.
    •
  • Dec 2022: “Accelerated Sampling and Improved Synthesis in Diffusion Models” at the NeurIPS 2022 Workshop on Score-Based Methods.
    •
  • Feb 2022: “Score-Based Generative Modeling with Critically-Damped Langevin Diffusion” at Deep Learning: Classics and Trends by ML Collective.
    •
  • Sep 2015: “Adaptive Resolution: From Atomistic and Coarse-Grained Hybrid Simulations to Quantum-Classical Coupling” at D. E. Shaw Research.
    •
  • Aug 2015: “Adaptive Resolution: From Atomistic and Coarse-Grained Hybrid Simulations to Quantum-Classical Coupling” at the Noid Lab at Pennsylvania State University.
    •

    Supervision and Mentoring

    I have had the opportunity to supervise and mentor several talented interns at NVIDIA:
  • Amirmojtaba Sabour
    •
  • Yilun Xu
    •
  • Katja Schwarz
    •
  • Andreas Blattmann
    •
  • Robin Rombach
    •
  • Tim Dockhorn
    •
  • Xiaohui Zeng
    •
  • Tianshi Cao
    •
  • Zhisheng Xiao (as co-mentor)
    •

    Reviewing

    I have been serving as a reviewer for these conferences and journals:
  • NeurIPS 2021, 2022, 2023
    •
  • ICLR 2022, 2023, 2024
    •
  • ICML 2022, 2023, 2024
    •
  • CVPR 2022
    •
  • SIGGRAPH 2022, 2023
    •
  • Nature
    •

    Software Development

    I used to be a software developer (between 2013-2019) of ESPResSo++, which stands for “Extensible Simulation Package for Research on Soft Matter”. I developed and implemented novel molecular dynamics algorithms and multiscale models for efficient and accurate computer simulations of relevant chemical and biological systems. I also performed code reviews, wrote tests, prepared documentation and tutorials, and taught new users in seminars and courses. Coding languages were C++ and Python with MPI-based parallelization.