CL4D: Contrastive Language–4D Pretraining for Vision-Language Reasoning in Dynamic Scenes

4D understanding and reasoning is a fundamental capability for embodied AI agents operating in dynamic physical environments. However, existing vision encoders are largely limited to static 2D images or 3D point clouds without temporal modeling, or to 2D videos that lack accurate geometric depth reasoning. Consequently, current approaches fail to jointly capture spatial structure and motion evolution in dynamic scenes.

We present CL4D, the first foundational 4D vision encoder that directly operates on dynamic point clouds, trained with a contrastive learning objective to align spatio-temporal geometric representations with natural language descriptions. By learning a shared embedding space between text and 4D scene dynamics, CL4D enables zero-shot motion-to-text and text-to-motion retrieval in dynamic environments and serves as a foundational 4D vision encoder for downstream 4D vision–language tasks.

Building on this encoder, we introduce 4DVLM, a 4D vision–language model that conditions language generation on dynamic geometric representations. 4DVLM is the first VLM designed to operate directly on 4D point clouds without relying on 2D images, 2D videos, or static 3D point clouds. We train CL4D and subsequently 4DVLM on a newly constructed dataset termed DynAction4D capturing diverse human motions across varying object interactions and scene environments.

Extensive experiments across multiple 4D human action benchmarks demonstrate that CL4D achieves state-of-the-art performance, with improvements of approximately ~16.75% over prior methods. Furthermore, 4DVLM outperforms frontier video VLMs such as Gemini and GPT-5 even when these models are provided with RGB video sequences corresponding to the same scenes represented as 4D point clouds for 4DVLM.

To facilitate Language–4D representation learning, we introduce DynAction4D, a large-scale dynamic human action point cloud dataset. DynAction4D consists of four distinct segments, each capturing unique dynamic scenarios with paired natural language descriptions:

• HumanOnly: Captures a wide variety of human-only actions (walking, jumping, dancing, etc.) across diverse environments, featuring 23k training and 4k testing sequences.
• ObjInteractions: Focuses on human actions involving object interactions, derived from Humoto. Includes 73 objects and 736 distinct interaction types.
• Cluttered: Places human characters performing various actions in environments cluttered with multiple objects at varying placements and density conditions.
• 4D-VQA: Specifically built for evaluating downstream 4D spatial and temporal reasoning capabilities, generated by prompting Gemini on Unity-rendered video sequences.

Our framework features a two-stage training strategy to bridge 4D scenes (dynamic point clouds) and natural language:

1. Stage 1: Contrastive Language–4D Pre-training (CL4D)
   A Spatio-Temporal Vision Encoder (V_st) operates directly on dynamic point clouds (x, y, z, t). It comprises a frame-level Pointnet Point Encoder (PE), a Spatial Transformer (V_s) modeling global spatial relationships within each frame, and a Temporal Transformer (V_t) modeling inter-frame temporal dependencies. These 4D visual embeddings are aligned with text embeddings from a Text Encoder (TE) using a symmetric cross-entropy objective.
2. Stage 2: 4D Large Vision-Language Model (4DVLM)
   The pre-trained V_st serves as a frozen foundational encoder. Projected 4D visual tokens are fused with Vicuna-7B (LLaVA architecture) to perform complex spatial, temporal, and action-based reasoning over dynamic environments, outputting descriptive language generation.

We adapt prior state-of-the-art 4D encoders using the same contrastive pre-training objective. CL4D consistently outperforms all methods across the three segments of DynAction4D and the real-world RH20T benchmark, showing up to ~16.75% R@1 improvement.

For a fair comparison with existing video-based VLMs that cannot process raw point clouds directly, we render the 4D point clouds into mesh videos. Equipped with a foundational geometry-aware encoder, our 4DVLM outperforms frontier video models across all metrics, validating the importance of 3D spatial grounding.