by Chen Liu and Tobias Ritschel
Transactions on Graphics (SIGGRAPH Asia 2024)
Please also check out our (Paper | Project Page)
This repo provides the official implementation in JAX for our paper, and our dataset.
Our dependencies can be installed by
# recommended installing in a new virtual env with python 3.10, like conda
pip install -r requirements.txt
- We mainly require these packages:
jax
,diffrax
,equinox
, andoptax
. - We also need
torch
(cpu) to load pre-trained VGG weights in.pth
format. - We use
tensorboard
to log our training - (Optional) Please also install
ffmpeg
for inference code to generate.avi
and.mp4
outputs
Tested in Ubuntu 22.04.4 x64 and NVIDIA RTX4090 GPU with CUDA 12.
Please download our dataset here, and extract RGB_dynamic_textures.zip
to data/RGB_dynamic_textures
and SVBRDF_dynamic_flash_textures.zip
to data/SVBRDF_dynamic_flash_textures
.
The folder structure should look like:
data
├── README.md
├── RGB_dynamic_textures
│ ├── changingcolor_leaves2.gif
│ ├── changingcolor_leaves.gif
│ ├── ...
│ └── vapouring_ice.gif
└── SVBRDF_dynamic_flash_textures
├── cheese_melting
│ ├── IMG_0794.JPG
│ ├── ...
├── clay_solidifying
├── ...
└── watercolor_painting
# to train with dynamic textures
bash scripts/odetexture.sh data/RGB_dynamic_textures experiments_texture
# to train with dynamic flash textures
bash scripts/odeBRDF.sh data/SVBRDF_dynamic_flash_textures experiments_svbrdf
# to visualize your training logs by tensorboard
tensorboard --logdir experiments_texture
# to sample with trained dynamic texture ODEs
bash scripts/sample_odetexture.sh data/RGB_dynamic_textures experiments_texture
# to sample with trained dynamic BRDF ODEs
bash scripts/sample_odeBRDF.sh data/SVBRDF_dynamic_flash_textures experiments_svbrdf
Please cite our paper if you use the code or data.
@article{liuNeuralDifferentialAppearance2024,
title={Neural Differential Appearance Equations},
author={Liu, Chen and Ritschel, Tobias},
journal={ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia)},
year={2024},
}