JVT: Just Video Transformer makes pixel-space video diffusion simple

Just Video Transformer makes
pixel-space video diffusion simple

Jianjun Zhou1,2,3*Hua Geng1*Haoyu Guo3†

1 Zhejiang University2 Shanghai Innovation Institute3 Shanghai AI Laboratory

* Equal contribution   † Corresponding author

VAE-free DiT for video generation

Following JiT, JVT predicts a clean video from noisy pixels under a rectified-flow objective. Large spatiotemporal patches make direct video modeling tractable.

JVT architecture: noisy video is patch embedded, processed by Transformer blocks with register tokens introduced after the first L1 blocks, then projected back to clean video.
Key findings: (1) patch embedding is a major bottleneck for generation quality and efficiency in the current architecture, (2) registers without guidance bring no significant benefit in pixel-space video diffusion, (3) standard AdaLN remains a effective design for conditioning.

Abstract

Direct diffusion modeling of high-dimensional raw video data is challenging, and existing pixel-space methods often rely on specialized architectures. We introduce JVT (Just Video Transformer), a simple large-patch DiT-style Transformer for pixel-space video diffusion, without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net-style skip connections. We systematically examine video patch embeddings, register tokens, and adaptive layer normalization to identify design choices that balance generation quality and computational efficiency. The resulting JVT design achieves generation quality competitive with latent diffusion baselines in both class-conditioned and unconditional settings. The pixel-space formulation of JVT also naturally supports joint generation of pixel-aligned modalities. We demonstrate this capability through camera-conditioned RGB-disparity generation, which improves camera following relative to RGB-only baselines and yields explicit scene geometry without an additional reconstruction model at inference. These results highlight the potential of simple DiT-style Transformers for pixel-space video diffusion, and we hope our findings offer useful insights and practical guidance for future research.

Qualitative results on classical benchmarks

16-frame video generation at 256×256 resolution.

UCF101

Class-conditioned

FaceForensics

Unconditional

SkyTimelapse

Unconditional

Taichi-HD

Unconditional

Pixel-aligned joint generation

Joint generate RGB with its pixel-aligned modalities (e.g. depth, raymap, segmentation, scene flow) directly in pixel space, without information loss in VAE latent space.

RealEstate10K examples showing JVT-generated RGB frames, aligned disparity maps and colored point clouds reconstructed with camera parameters.
Joint RGB-disparity generation on RealEstate10K. Point clouds come from the generated disparity and RGB, without an additional reconstruction model at inference.

Joint generated geometry with videos

input cameras camera path current camera

Static preview · scroll here to load the interactive results.

RGB
Depth
Point cloud
3D view
Ground truth
JVT
Aether w/ depth
Aether w/o depth

Select a point cloud or 3D view to explore it. Drag to orbit, scroll to zoom, right-drag to pan.

Details

All methods share the initial RGB frame and target camera trajectory from RealEstate10K. JVT and Aether w/ depth also receive the same Pi3X depth for the initial frame. Aether w/o depth receives none. Each method's depth for its 8 frames is unprojected with the target cameras into a single colored point cloud. Cross-frame inconsistency can appear as doubled or drifting geometry. “Ground truth” combines real RGB frames with Pi3X-estimated depth, rather than measured depth. Depth receives the same display filtering for every method: latent-patch deblocking and a guided filter.

Camera following

Depth (disparity) helps improve translation of camera following

reference region, boxed at the same position in every method
GT
DFoT
GF
LIVE
JVT
JVT w/ disp.
Frame 0
Frame 4
Frame 7

Loading camera comparisons…

Quantitative evaluation of camera following

Camera-conditioned generation on RealEstate10K · 8 frames, 256 × 256
MethodObjectiveFVD ↓RRA ↑RTA ↑AUC ↑
@5°@30°@5°@30°@5°@30°
DFoTRGB35.399.910077.797.458.387.2
Geometry ForcingRGB46.899.810073.696.452.184.7
LIVERGB42.599.110049.892.628.473.3
JVT (Ours)RGB27.099.910072.496.052.884.1
JVT (Ours)RGB-disparity27.599.910079.598.458.688.6

RRA and RTA report relative rotation and translation-direction accuracy at 5° and 30° thresholds. AUC summarizes their joint error distribution. Camera poses are estimated with Pi3X.

Quantitative results on classical benchmarks

Selected results from the paper's main benchmark table · FVD@2048 ↓
MethodResolutionClass-conditionedUnconditional
UCF-101UCF-101FaceForensicsSkyTimelapseTaichi-HD
Latte256²477.97—34.0059.82159.60
JVT (Ours)256²136.5413.532.8646.58119.1
JVT+RG (Ours)256²129.2215.931.8047.80124.3

JVT denotes the register-free model, JVT+RG uses register guidance. RG improves UCF-101 results substantially, while its effect on the other datasets varies. Bold and underlined values indicate the best and second-best scores in the full table under the paper's ranking protocol.

All benchmark results and evaluation protocols
Complete benchmark comparison · FVD@2048 ↓
MethodResolutionClass-conditionedUnconditional
UCF-101UCF-101FaceForensicsSkyTimelapseTaichi-HD
MoCoGAN256²—2886.9†124.7†206.6†—
MoCoGAN-HD256²—1729.6†111.8†164.1†144.7‡
VideoGPT256²—2880.6†185.9†222.7†—
StyleGAN-V256²—1431.047.4179.52—
MoStGAN-V256²—1380.339.765.3—
CogVideo160²626————
DIGAN128²465577—114.6128.1*
DIGAN256²—1630.2†62.5†83.11†—
LVDM256²—372—95.299.0*
PVDM256²—343.6—55.41—
VIDM128²————121.9
VIDM256²—294.7—57.4—
GLOBER128²151.5239.5——124.2
GLOBER256²168.9252.7—78.1—
OmniTokenizer256²191————
VDT64²—225.7———
FVDM256²468.23—55.01106.09194.61
Latte256²477.97—34.0059.82159.60
JVT (Ours)256²136.5413.532.8646.58119.1
JVT+RG (Ours)256²129.2215.931.8047.80124.3

† Scores taken from StyleGAN-V, ‡ scores taken from DIGAN. Other baseline scores are sourced from their original papers. Taichi-HD scores marked * use both training and test splits and are excluded from ranking. — denotes an unreported result. JVT uses both training and test splits of UCF-101, and only the training split of the other datasets, following Latte's setting.

Citation

@article{zhou2026jvt,
  title   = {{JVT}: Just Video Transformer makes pixel-space video diffusion simple},
  author  = {Jianjun Zhou and Hua Geng and Haoyu Guo},
  year    = {2026}
  journal = {arXiv preprint arXiv: 2611.18549}
}