profile_picture
Karsten Kreis
Senior Research Scientist, NVIDIA
kkreis [at] nvidia [dot] com

I am a Senior Research Scientist in the fundamental generative AI research team of NVIDIA Research. I am interested both in generative AI algorithm development and in applying deep generative models in areas such as representation learning, graphics, content creation and digital artistry. I am also interested in research that takes inspirations from physics to improve machine learning techniques as well as in applying state-of-the-art generative learning methods to problems in the natural sciences.

I am trained as a physicist and completed my master’s thesis in quantum information theory. For my Ph.D. in computational and statistical physics, I developed multiscale models and sampling algorithms for molecular dynamics simulation. Before joining NVIDIA, I worked on deep generative learning at D-Wave Systems and I co-founded Variational AI, a startup leveraging generative modeling for drug discovery.

Publications

*,† indicate equal contribution / co-first authorship.
EquiJump: Protein Dynamics Simulation via SO(3)-Equivariant Stochastic Interpolants
arXiv preprint, 2024
Allan dos Santos Costa* , Ilan Mitnikov* , Franco Pellegrini* , Ameya Daigavane , Mario Geiger , Zhonglin Cao , Karsten Kreis , Tess Smidt , Emine Kucukbenli , Joseph Jacobson
arxiv

Energy-Based Diffusion Language Models for Text Generation
arXiv preprint, 2024
Minkai Xu , Tomas Geffner , Karsten Kreis , Weili Nie , Yilun Xu , Jure Leskovec , Stefano Ermon , Arash Vahdat
arxiv

Truncated Consistency Models
arXiv preprint, 2024
Sangyun Lee , Yilun Xu , Tomas Geffner , Giulia Fanti , Karsten Kreis , Arash Vahdat , Weili Nie
arxiv project page

Multi-student Diffusion Distillation for Better One-step Generators
arXiv preprint, 2024
Yanke Song , Jonathan Lorraine , Weili Nie , Karsten Kreis , James Lucas
arxiv project page

Molecule Generation with Fragment Retrieval Augmentation
Neural Information Processing Systems (NeurIPS), 2024
Seul Lee , Karsten Kreis , Srimukh Prasad Veccham , Meng Liu , Danny Reidenbach , Saee Gopal Paliwal , Arash Vahdat* , Weili Nie*
proceedings

Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models
Neural Information Processing Systems (NeurIPS), 2024
Giannis Daras , Weili Nie , Karsten Kreis , Alexandros G. Dimakis , Morteza Mardani , Nikola B. Kovachki , Arash Vahdat
arxiv proceedings project page

Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimization
Neural Information Processing Systems (NeurIPS), 2024
Siyi Gu* , Minkai Xu* , Alexander Powers , Weili Nie , Tomas Geffner , Karsten Kreis , Jure Leskovec , Arash Vahdat , Stefano Ermon
arxiv code proceedings

L4GM: Large 4D Gaussian Reconstruction Model
Neural Information Processing Systems (NeurIPS), 2024
Jiawei Ren , Kevin Xie , Ashkan Mirzaei , Hanxue Liang , Xiaohui Zeng , Karsten Kreis , Ziwei Liu , Antonio Torralba , Sanja Fidler , Seung Wook Kim , Huan Ling
arxiv proceedings project page

DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents
International Conference on Machine Learning (ICML), 2024
Yilun Xu , Gabriele Corso , Tommi Jaakkola , Arash Vahdat , Karsten Kreis
arxiv code proceedings project page

Align Your Steps: Optimizing Sampling Schedules in Diffusion Models
International Conference on Machine Learning (ICML), 2024
Amirmojtaba Sabour , Sanja Fidler , Karsten Kreis
arxiv code proceedings project page

Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models
Computer Vision and Pattern Recognition (CVPR), 2024 (Highlight)
Huan Ling* , Seung Wook Kim* , Antonio Torralba , Sanja Fidler , Karsten Kreis
arxiv proceedings project page

Outdoor Scene Extrapolation with Hierarchical Generative Cellular Automata
Computer Vision and Pattern Recognition (CVPR), 2024 (Highlight)
Dongsu Zhang , Francis Williams , Zan Gojcic , Karsten Kreis , Sanja Fidler , Young Min Kim , Amlan Kar
arxiv proceedings project page

WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space
International Conference on Learning Representations (ICLR), 2024
Katja Schwarz , Seung Wook Kim , Jun Gao , Sanja Fidler , Andreas Geiger , Karsten Kreis
arxiv proceedings project page

A Unified Approach for Text- and Image-guided 4D Scene Generation
arXiv preprint, 2023
Yufeng Zheng , Xueting Li , Koki Nagano , Sifei Liu , Karsten Kreis , Otmar Hilliges , Shalini De Mello
arxiv

TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models
International Conference on Computer Vision (ICCV), 2023 (Oral Presentation)
Tianshi Cao , Karsten Kreis , Sanja Fidler , Nicholas Sharp* , Kangxue Yin*
arxiv proceedings project page

DreamTeacher: Pretraining Image Backbones with Deep Generative Models
International Conference on Computer Vision (ICCV), 2023
Daiqing Li* , Huan Ling* , Amlan Kar , David Acuna , Seung Wook Kim , Karsten Kreis , Antonio Torralba , Sanja Fidler
arxiv proceedings project page

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
Computer Vision and Pattern Recognition (CVPR), 2023
Andreas Blattmann* , Robin Rombach* , Huan Ling* , Tim Dockhorn* , Seung Wook Kim , Sanja Fidler , Karsten Kreis
arxiv proceedings project page

NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models
Computer Vision and Pattern Recognition (CVPR), 2023
Seung Wook Kim* , Bradley Brown* , Kangxue Yin , Karsten Kreis , Katja Schwarz , Daiqing Li , Robin Rombach , Antonio Torralba , Sanja Fidler
arxiv proceedings project page

Magic3D: High-Resolution Text-to-3D Content Creation
Computer Vision and Pattern Recognition (CVPR), 2023 (Highlight)
Chen-Hsuan Lin* , Jun Gao* , Luming Tang* , Towaki Takikawa* , Xiaohui Zeng* , Xun Huang , Karsten Kreis , Sanja Fidler† , Ming-Yu Liu† , Tsung-Yi Lin
arxiv proceedings project page

Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion
Computer Vision and Pattern Recognition (CVPR), 2023
Davis Rempe* , Zhengyi Luo* , Xue Bin Peng , Ye Yuan , Kris Kitani , Karsten Kreis , Sanja Fidler , Or Litany
arxiv proceedings project page

Differentially Private Diffusion Models
Transactions on Machine Learning Research (TMLR), 2023
Tim Dockhorn , Tianshi Cao , Arash Vahdat , Karsten Kreis
arxiv code proceedings project page

Score-based Diffusion Models in Function Space
arXiv preprint, 2023
Jae Hyun Lim* , Nikola B. Kovachki* , Ricardo Baptista* , Christopher Beckham , Kamyar Azizzadenesheli , Jean Kossaifi , Vikram Voleti , Jiaming Song , Karsten Kreis , Jan Kautz , Christopher Pal , Arash Vahdat , Anima Anandkumar
arxiv

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
arXiv preprint, 2022
Yogesh Balaji , Seungjun Nah , Xun Huang , Arash Vahdat , Jiaming Song , Qinsheng Zhang , Karsten Kreis , Miika Aittala , Timo Aila , Samuli Laine , Bryan Catanzaro , Tero Karras , Ming-Yu Liu
arxiv project page

Latent Space Diffusion Models of Cryo-EM Structures
Machine Learning for Structural Biology Workshop, NeurIPS, 2022 (Oral Presentation)
Karsten Kreis* , Tim Dockhorn* , Zihao Li , Ellen Zhong
arxiv

GENIE: Higher-Order Denoising Diffusion Solvers
Neural Information Processing Systems (NeurIPS), 2022
Tim Dockhorn , Arash Vahdat , Karsten Kreis
arxiv code proceedings project page

LION: Latent Point Diffusion Models for 3D Shape Generation
Neural Information Processing Systems (NeurIPS), 2022
Xiaohui Zeng , Arash Vahdat , Francis Williams , Zan Gojcic , Or Litany , Sanja Fidler , Karsten Kreis
arxiv code proceedings project page

Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps
Computer Vision and Pattern Recognition (CVPR), 2022 (Oral Presentation)
Seung Wook Kim , Karsten Kreis , Daiqing Li , Antonio Torralba , Sanja Fidler
arxiv proceedings project page

BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations
Computer Vision and Pattern Recognition (CVPR), 2022
Daiqing Li , Huan Ling , Seung Wook Kim , Karsten Kreis , Adela Barriuso , Sanja Fidler , Antonio Torralba
arxiv code proceedings project page

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion
International Conference on Learning Representations (ICLR), 2022 (Spotlight Presentation)
Tim Dockhorn , Arash Vahdat , Karsten Kreis
arxiv code proceedings project page

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs
International Conference on Learning Representations (ICLR), 2022 (Spotlight Presentation)
Zhisheng Xiao , Karsten Kreis , Arash Vahdat
arxiv code proceedings project page

Score-based Generative Modeling in Latent Space
Neural Information Processing Systems (NeurIPS), 2021
Arash Vahdat* , Karsten Kreis* , Jan Kautz
arxiv code proceedings project page

Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence
Neural Information Processing Systems (NeurIPS), 2021
Tianshi Cao , Alex Bie , Arash Vahdat , Sanja Fidler , Karsten Kreis
arxiv code proceedings project page

EditGAN: High-Precision Semantic Image Editing
Neural Information Processing Systems (NeurIPS), 2021
Huan Ling* , Karsten Kreis* , Daiqing Li , Seung Wook Kim , Antonio Torralba , Sanja Fidler
arxiv code nvidia blog post proceedings project page two minute papers

ATISS: Autoregressive Transformers for Indoor Scene Synthesis
Neural Information Processing Systems (NeurIPS), 2021
Despoina Paschalidou , Amlan Kar , Maria Shugrina , Karsten Kreis , Andreas Geiger , Sanja Fidler
arxiv code proceedings project page

Causal Scene BERT: Improving object detection by searching for challenging groups of data
International Conference on Computer Vision (ICCV), 2nd AVVision Workshop, 2021
Cinjon Resnick , Or Litany , Amlan Kar , Karsten Kreis , James Lucas , Kyunghyun Cho , Sanja Fidler
arxiv proceedings

Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
Computer Vision and Pattern Recognition (CVPR), 2021
Daiqing Li , Junlin Yang , Karsten Kreis , Antonio Torralba , Sanja Fidler
arxiv code proceedings project page

Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes
Computer Vision and Pattern Recognition (CVPR), 2021 (Oral Presentation)
Towaki Takikawa* , Joey Litalien* , Kangxue Yin , Karsten Kreis , Charles Loop , Derek Nowrouzezahrai , Alec Jacobson , Morgan McGuire , Sanja Fidler
arxiv code proceedings project page

VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
International Conference on Learning Representations (ICLR), 2021 (Spotlight Presentation)
Zhisheng Xiao , Karsten Kreis , Jan Kautz , Arash Vahdat
arxiv code proceedings

Variational Amodal Object Completion
Neural Information Processing Systems (NeurIPS), 2020
Huan Ling , David Acuna , Karsten Kreis , Seung Wook Kim , Sanja Fidler
proceedings project page

ESPResSo++ 2.0: Advanced methods for multiscale molecular simulation
Computer Physics Communications (CPC), 2019
Horacio V. Guzman , Nikita Tretyakov , Hideki Kobayashi , Aoife C. Fogarty , Karsten Kreis , Jakub Krajniak , Christoph Junghans , Kurt Kremer , Torsten Stuehn
paper

From classical to quantum and back: Hamiltonian adaptive resolution path integral, ring polymer, and centroid molecular dynamics
Journal of Chemical Physics (JCP), 2017
Karsten Kreis , Kurt Kremer , Raffaello Potestio , Mark E. Tuckerman
arxiv paper

The relative entropy is fundamental to adaptive resolution simulations
Journal of Chemical Physics (JCP), 2016
Karsten Kreis , Raffaello Potestio
paper

Adaptive Resolution Simulations with Self-Adjusting High-Resolution Regions
Journal of Chemical Theory and Computation (JCTC), 2016 (Featured on the Journal Cover)
Karsten Kreis , Raffaello Potestio , Kurt Kremer , Aoife C. Fogarty
journal cover paper

From Classical to Quantum and Back: A Hamiltonian Scheme for Adaptive Multiresolution Classical/Path-Integral Simulations
Journal of Chemical Theory and Computation (JCTC), 2016
Karsten Kreis , Mark E. Tuckerman , Davide Donadio , Kurt Kremer , Raffaello Potestio
paper

Advantages and challenges in coupling an ideal gas to atomistic models in adaptive resolution simulations
The European Physical Journal Special Topics (EPJ ST), 2015
Karsten Kreis , Aoife C. Fogarty , Kurt Kremer , Raffaello Potestio
arxiv paper

A unified framework for force-based and energy-based adaptive resolution simulations
Europhysics Letters (EPL), 2014
Karsten Kreis , Davide Donadio , Kurt Kremer , Raffaello Potestio
paper

Classifying, quantifying, and witnessing qudit-qumode hybrid entanglement
Physical Review A (PRA), 2012
Karsten Kreis , Peter van Loock
arxiv paper


Theses

Advanced Adaptive Resolution Methods for Molecular Simulation
Ph.D. Thesis, 2018
Carried out at the Max Planck Institute for Polymer Research and at New York University
pdf
Characterizing And Exploiting Hybrid Entanglement
Master Thesis ("Diplomarbeit"), 2011
Carried out at the Max Planck Institute for the Science of Light
arxiv

Prizes, Awards and Scholarships

  • Highlighted Reviewer at the International Conference on Learning Representations (ICLR), 2022 --- Outstanding Reviewer (Top 10%) at the International Conference on Machine Learning (ICML), 2022 --- Notable Reviewer (Top 1%) at the International Conference on Learning Representations (ICLR), 2023
    •
  • Prize of the Department of Physics, Mathematics and Computer Science of the Johannes Gutenberg University Mainz for an outstanding dissertation, 2019
    •
  • Ph.D. thesis awarded highest grade "summa cum laude", 2018
    •
  • Moore/Sloan & Washington Research Foundation Innovation in Data Science Postdoctoral Fellowship at the University of Washington, Seattle (declined), 2017
    •
  • Ph.D. Scholarship of the "Graduate School of Excellence Materials Science in Mainz", 2013-2016
    •
  • Poster prize for the best poster at the “Meet-Your-Collegue-Day 2013” of the Max Planck Institute for Polymer Research, 2013
    •
  • "Ohm Prize" of the Department for Physics of the Friedrich-Alexander University Erlangen-Nuremberg for an outstanding diploma thesis, 2011
    •
  • "Erasmus Grant" to study abroad at Imperial College London, 2007-2008
    •
  • "Book Prize" of the German Physical Society for excellent achievements in the subject of physics in high school, 2005
    •

    Miscellaneous

    Tutorials

  • Tutorial “Diffusion Models: A Generative AI Big Bang” at NVIDIA GTC 2024.
    •
  • Tutorial “Latent Diffusion Models: Is the Generative AI Revolution Happening in Latent Space?" at Neural Information Processing Systems (NeurIPS), 2023.
    •
  • Course on Diffusion Models at SIGGRAPH, 2023.
    •
  • Tutorial “Denoising Diffusion-based Generative Modeling: Foundations and Applications” at the Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
    •
  • Two introductory blog posts (part 1 and part 2) about our three recent works on Latent Score-based Generative Models, Denoising Diffusion GANs and Critically-Damped Langevin Diffusion.
    •

    Invited Talks

  • Jun 2024: “The Rise of Diffusion Models in Computer Vision and Lessons for Protein Design” at the Institute for Protein Design.
    •
  • Jun 2024: “Generative AI as Artists’ Paintbrush: From Diffusion Models to 4D Dynamic Content Creation”, at the CVPR Workshop on Computer Vision for Fashion, Art, and Design.
    •
  • May 2024: “Generative AI - Vision”, lecture at the Oxford Machine Learning Summer School, MLx Fundamentals track.
    •
  • Apr 2024: “Visual Generative AI with Diffusion Models: From Static Pixels to Video, 3D and 4D Synthesis”, at the RISE Learning Machines seminars.
    •
  • Feb 2024: “Visual Generative AI: From its Beginnings to Modern Diffusion Models for Image, Video, 3D and 4D Synthesis”, lecture at the University of Toronto and Vector Institute (slides).
    •
  • Dec 2023: “Beyond Static Pixels: From Video to Text-to-4D Synthesis with Diffusion Models” at the NeurIPS 2023 Workshop on Diffusion Models.
    •
  • Nov 2023: “Kuenstliche Intelligenz: Ueberblick und Trends” at Mercedes-Benz, Bremen.
    •
  • Oct 2023: “Generative Diffusion Models: Inspirations from Physics and Applications in Digital Content Creation” at Structured Learning Workshop, Chalmers AI.
    •
  • Aug 2023: “Diffusion Models: From Algorithms to Image, Video, 3D and Cryo-EM Structure Synthesis” at MIT CSAIL.
    •
  • Aug 2023: “Generative Diffusion Models: Inspirations from Physics and Applications in Digital Content Creation” at IAIFI Summer Workshop.
    •
  • Jul 2023: “Diffusion Models: From Foundations to Image, Video and 3D Content Creation” at the University of British Columbia.
    •
  • Jun 2023: “Image, Video and 3D Content Creation with Diffusion Models” at the Vision Models Forum at the 2023 BAAI Conference.
    •
  • Jun 2023: “From Images and Video to 3D Shapes: Content Creation with Diffusion Models” at the Conference on Robotics and Vision 2023.
    •
  • Apr 2023: “Diffusion Models: From Foundations to Image, Video and 3D Content Creation” at Simon Fraser University.
    •
  • Apr 2023: “Diffusion Models: From Foundations to Image, Video and 3D Content Creation” at the Computer Vision Group at the University of Illinois Urbana-Champaign.
    •
  • Jan 2023: “Latent Space Diffusion Models of Cryo-EM Structures” at Variational AI.
    •
  • Dec 2022: “Advanced Diffusion Models: Accelerated Sampling, Smooth Diffusion, and 3D Shape Generation” at the Computer Vision Group at the University of Bern.
    •
  • Dec 2022: “Accelerated Sampling and Improved Synthesis in Diffusion Models” at the NeurIPS 2022 Workshop on Score-Based Methods.
    •
  • Feb 2022: “Score-Based Generative Modeling with Critically-Damped Langevin Diffusion” at Deep Learning: Classics and Trends by ML Collective.
    •
  • Sep 2015: “Adaptive Resolution: From Atomistic and Coarse-Grained Hybrid Simulations to Quantum-Classical Coupling” at D. E. Shaw Research.
    •
  • Aug 2015: “Adaptive Resolution: From Atomistic and Coarse-Grained Hybrid Simulations to Quantum-Classical Coupling” at the Noid Lab at Pennsylvania State University.
    •

    Supervision and Mentoring

    I have had the opportunity to supervise and mentor several talented interns at NVIDIA:
  • Zuobai Zhang
    •
  • Hannes Staerk
    •
  • Bowen Jing
    •
  • Amirmojtaba Sabour
    •
  • Yilun Xu
    •
  • Katja Schwarz
    •
  • Andreas Blattmann
    •
  • Robin Rombach
    •
  • Tim Dockhorn
    •
  • Xiaohui Zeng
    •
  • Tianshi Cao
    •
  • Zhisheng Xiao (as co-mentor)
    •

    Reviewing

    I have been serving as a reviewer for these conferences and journals:
  • NeurIPS 2021, 2022, 2023, 2024
    •
  • ICLR 2022, 2023, 2024, 2025
    •
  • ICML 2022, 2023, 2024
    •
  • CVPR 2022
    •
  • SIGGRAPH 2022, 2023
    •
  • Nature
    •

    Software Development

    I used to be a software developer (between 2013-2019) of ESPResSo++, which stands for “Extensible Simulation Package for Research on Soft Matter”. I developed and implemented novel molecular dynamics algorithms and multiscale models for efficient and accurate computer simulations of relevant chemical and biological systems. I also performed code reviews, wrote tests, prepared documentation and tutorials, and taught new users in seminars and courses. Coding languages were C++ and Python with MPI-based parallelization.