Survey of 3D Compression and Streaming

Media formats are moving beyond 1D audio and 2D images which are viewed at static viewing angles and scale to 3D content which are viewed from dynamic viewing angles and scale. This has broad implications in gaming, virtual concerts, television, augmented reality, and more. Compression and streaming are at the core of the design of the next generation of media systems, and 3D brings new challenges which require innovations in algorithms as well as systems.

This repository collects papers and talks and blog posts which broadly discuss compression and streaming for 3D content. This is an exciting field which combines computer graphics, computer vision, machine learning, multimedia, systems, signal processing, information theory, security, and more!

This is very much work in progress and missings lots of work. The categorizations etc are very likely to change.

Related Surveys / Position Papers / Broadly Applicable Papers

Compression is Again Driving the Evolution of Media
Philip A. Chou

State-of-the-art in Compressed GPU-Based Direct Volume Rendering
Marcos Balsa Rodríguez, Enrico Gobbetti, Jose Antonio Iglesias Guitian, Maxim Makhinya, Fabio Marton, Renato Pajarola, Susanne K Suter
Eurographics State-of-the-Art Reports 2014

3D Mesh Compression: Survey, Comparisons, and Emerging Trends
Adrien Maglo, Guillaume Lavoue, Florent Dupont, Celine Hudelot
Eurographics State-of-the-Art Reports 2015

Nonlinear Transform Coding
Johannes Ballé, Philip A. Chou, David Minnen, Saurabh Singh, Nick Johnston, Eirikur Agustsson, Sung Jin Hwang, George Toderici
IEEE Journal of Selected Topics in Signal Processing 2020

Toward Truly Immersive Holographic-Type Communication: Challenges and Solutions
Alexander Clemm, Maria Torres Vega, Hemanth Kumar Ravuri, Tim Wauters, Filip De Turkc
IEEE Communications Magazine 2020

Extremely Interactive and Low-Latency Services in 5G and Beyond Mobile Systems
Tarik Taleb, Zinelaabidine Nadir, Hannu Flinck, JaeSeung Song
IEEE Communications Standards Magazine 2021

Neural Fields in Visual Computing and Beyond
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, Srinath Sridhar
Eurographics State-of-the-Art Reports 2022

Standards

MPEG-I: Coded Representation of Immersive Media

Emerging MPEG Standards for Point Cloud Compression
Sebastian Schwarz, Marius Preda, Madhukar Budagavi, Pablo Cesar, Philip A. Chou, Robert A. Cohen, Maja Krivokuca, Sebastien Lasserre, Zhu Li, Joan Llach, Khaled Mammou, Rufael Mekuria, Ohji Nakagami, Ernestasia Siahaan, Ali Tabatabai, Alexis M. Tourapis, Vladyslav Zakharchenko
IEEE JETCAS 2019

A Comprehensive Study and Comparison of Core Technologies for MPEG 3-D Point Cloud Compression
Hao Liu, Hui Yuan, Qi Liu, Junhui Hou, Ju Liu
IEEE Transactions on Broadcasting 2019

An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC)
D. Graziosi, O. Nakagami, S. Kuma, A. Zaghetto, T. Suzuki, A. Tabatabai APSIPA Transactions on Signal and Information Processing 2020

Systems Papers

GigaVoxels: ray-guided streaming for efficient and detailed voxel rendering
Cyril Crassin, Fabrice Neyret, Sylvain Lefebvre, Elmar Eisemann
I3D 2009

Holoportation: Virtual 3D Teleportation in Real-time
Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, Vladimir Tankovich, Charles Loop, Qin Cai, Philip A Chou, Sarah Mennicken, Julien Valentin, Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli, Yuliya Lutchyn, Cem Keskin, Shahram Izadi
ACM UIST 2016

Project Startline: A high-fidelity telepresence system
Jason Lawrence Dan B Goldman Supreeth Achar Gregory Major Blascovich Joseph G. Desloge Tommy Fortes Eric M. Gomez Sascha Häberling Hugues Hoppe Andy Huibers Claude Knaus Brian Kuschak Ricardo Martin-Brualla Harris Nover Andrew Ian Russell Steven M. Seitz Kevin Tong
ACM Transactions on Graphics 2021

Nanite: A Deep Dive
Brian Karis, Rune Stubbe, Graham Wihlidal
ACM SIGGRAPH 2021, Advances in Real-time Rendering in Games

Continuous Point Clouds

Compression

Self-similarity for accurate compression of point sampled surfaces
Julie Digne, Raphaelle Chaine, Sebastien Valette
Eurographics 2014

3D point cloud compression using conventional image compression for efficient data transmission
Hamidreza Houshiar, Andreas Nuchter
IEEE ICAT 2015

Compression and Rendering of Textured Point Clouds via Sparse Coding
Kersten Schuster, Philip Trettner, Patric Schmitz, Julian Schakib, Leif Kobbelt
HPG 2021

Voxelized Point Clouds

We draw a slight distinction between point clouds and voxelized point clouds. The latter is quantized into integer coordinates, while the general class of point clouds are still often quantized but not necessarily into a regular grid.

A Volumetric Approach to Point Cloud Compression—Part I: Attribute Compression
Philip A. Chou, Maxim Koroteev, Maja Krivokuca
IEEE Transactions of Image Processing

A Volumetric Approach to Point Cloud Compression—Part II: Geometry Compression
Maja Krivokuca, Philip A. Chou, Maxim Koroteev
IEEE Transactions of Image Processing

Compression

Point Cloud Attribute Compression with Graph Transform
Cha Zhang, Dinei Florencio, Charles Loop
IEEE ICIP 2014

Compression of 3D Point Clouds Using a Region-Adaptive Hierarchical Transform
Ricardo L. de Queiroz, Philip A. Chou
IEEE Transactions on Image Processing 2016

Geometry and attribute compression for voxel scenes
Bas Dado, Timothy R. Kol, Pablo Bauszat, Jean-Marc Thiery, Elmar Eisemann
Eurographics 2016

Real-time compression and streaming of 4D performances
Danhang Tang, Mingsong Dou, Peter Lincoln, Philip Davidson, Kaiwen Guo, Jonathan Taylor, Sean Fanello, Cem Keskin, Adarsh Kowdle, Sofien Bouaziz, Shahram Izadi, Andrea Tagliasacchi
SIGGRAPH Asia 2018

Deep Implicit Volume Compression
Danhang Tang, Saurabh Singh, Philip A. Chou, Christian Haene, Mingsong Dou, Sean Fanello, Jonathan Taylor, Philip Davidson, Onur G. Guleryuz, Yinda Zhang, Shahram Izadi, Andrea Tagliasacchi, Sofien Bouaziz, Cem Keskin
CVPR 2020

OctSqueeze: Octree-Structured Entropy Model for LiDAR Compression
Lila Huang, Shenlong Wang, Kelvin Wong, Jerry Liu, Raquel Urtasun
CVPR 2020

MuSCLE: Multi Sweep Compression of LiDAR using Deep Entropy Models
Sourav Biswas, Jerry Liu, Kelvin Wong, Shenlong Wang, Raquel Urtasun
NeurIPS 2020

Region adaptive graph fourier transform for 3d point clouds
Eduardo Pavez, Benjamin Girault, Antonio Ortega, Philip A. Chou
IEEE ICIP 2020

VoxelContext-Net: An Octree based Framework for Point Cloud Compression
Zizheng Que, Guo Lu, Dong Xu
CVPR 2021

Moving Basis Decomposition for Precomputed Light Transport
Ari Silvennoinen, Peter-Pike Sloan
EGSR 2021

Volumetric Attribute Compression for 3D Point Clouds using Feedforward Network with Geometric Attention
Tam Thuc Do, Philip A. Chou, Gene Cheung
ICASSP 2023

Post-processing

Deep Geometry Post-Processing for Decompressed Point Clouds
Xiaoqing Fan, Ge Li, Dingquan Li, Yurui Ren, Wei Gao, Thomas H. Li
ArXiv, 2022

Adaptive Streaming

Rate-Utility Optimized Streaming of Volumetric Media for Augmented Reality
Jounsup Park, Philip A. Chou, Jenq-Neng Hwang
IEEE JETCAS 2019

Towards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression
Jeroen van der Hooft, Tim Wauters, Filip De Turck, Christian Timmerer, Hermann Hellwagner
ACM MM 2019

Neural Fields

Neural fields offer a new paradigm for compression which is based on function fitting (akin to some earlier works on compression via polynomial fitting, etc). Eventually I will write something which explains why neural fields might be a cool alternative / complementary to traditional voxel-adjacent compression methods.

Representation

There is a lot of work in this area. Here, we only list papers which make an experimental comparison in terms of compression (broadly defined).

On the Effectiveness of Weight-Encoded Neural Implicit 3D Shapes
Thomas Davies, Derek Nowrouzezahrai, Alec Jacobson
Technical Report

Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Surfaces
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, Sanja Fidler
CVPR 2021

ACORN: Adaptive Coordinate Networks for Neural Scene Representation
Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, Gordon Wetzstein
SIGGRAPH 2021

BACON: Band-limited Coordinate Networks for Multiscale Scene Representation
David B. Lindell, Dave Van Veen, Jeong Joon Park, Gordon Wetzstein
ArXiv 2021

Compression

Compressive Neural Representations of Volumetric Scalar Fields
Yuzhe Lu, Kairong Jiang, Joshua A. Levine, Matthew Berger
EuroVis 2021

3D Scene Compression through Entropy Penalized Neural Representation Functions
Thomas Bird, Johannes Balle, Saurabh Singh, Philip A. Chou
IEEE PCS 2021

NeRV: Neural Representations for Videos
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav Shrivastava
NeurIPS 2021

COIN: COmpression with Implicit Neural Representations
Emilien Dupont, Adam Goliński, Milad Alizadeh, Yee Whye Teh, Arnaud Doucet
ICLR 2021 Neural Compression Workshop

Implicit Neural Representations for Image Compression
Yannick Strümpler, Janis Postels, Ren Yang, Luc Van Gool, and Federico Tombari
ECCV 2022

COIN++: Data Agnostic Neural Compression
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Goliński, Yee Whye Teh, Arnaud Doucet
ArXiv 2022

LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks
Berivan Isik, Philip A. Chou, Sung Jin Hwang, Nick Johnston, George Toderici
ArXiv 2021

Implicit Neural Video Compression
Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, Taco Cohen
ArXiv 2021

Neural Motion Compression with Frequency-adaptive Fourier Feature Network
Kenji Tojo, Yifei Chen, Nobuyuki Umetani
Eurographics 2022

Variable Bitrate Neural Fields
Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Muller, Morgan McGuire, Alec Jacobson, Sanja Fidler
SIGGRAPH 2022

Streamable Neural Fields
Junwoo Cho, Seungtae Nam, Daniel Rho, Jong Hwan Ko, Eunbyung Park
ECCV 2022

On Quantizing Implicit Neural Representations
Cameron Gordon, Shin-Fang Chng, Lachlan MacDonald, Simon Lucey
WACV 2023

Masked Wavelet Representation for Compact Neural Radiance Fields
Daniel Rho, Byeonghyeon Lee, Seungtae Nam, Joo Chan Lee, Jong Hwan Ko, Eunbyung Park
CVPR 2023

MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes
Christian Reiser, Richard Szeliski, Dor Verbin, Pratul P. Srinivasan, Ben Mildenhall, Andreas Geiger, Jonathan T. Barron, Peter Hedman
ArXiv 2023