--- id: video-understanding title: "Video Understanding: Action Recognition, Temporal Action Detection, and Video-Language Models" schema_type: article category: ai language: en confidence: high last_verified: "2026-05-24" created_date: "2026-05-24" generation_method: ai_assisted ai_models: - claude-4.5-sonnet derived_from_human_seed: true conflict_of_interest: none_declared is_live_document: false data_period: static completeness: 0.85 atomic_facts: - id: af-video-understanding-1 statement: >- Springer IJCV (May 2025) published a comprehensive survey of action understanding -- tracing the evolution from handcrafted features (HOG3D, improved dense trajectories, pre-2014) to 3D CNNs (I3D, C3D, 2014-2019) to video transformers (TimeSformer, VideoMAE, 2020-present) -- finding that VideoMAE v2 achieves 90.8% top-1 on Kinetics-400 and 86.6% on Something-Something V2, matching supervised training with 90% less labeled data through masked autoencoding pretraining. source_title: "Springer IJCV (2025) -- About Time: Advances, Challenges, and Outlooks of Action Understanding" source_url: https://link.springer.com/article/10.1007/s11263-025-02478-4 confidence: high - id: af-video-understanding-2 statement: >- ScienceDirect (February 2026) published a comprehensive survey of action recognition consolidating a decade of progress -- documenting that multimodal video-language models (Video-LLaMA, VideoChat) combining visual, audio, and text modalities achieve 5-10% accuracy gains over vision-only models on complex activity understanding (cooking, assembly) and that temporal action detection (localizing start/end times) remains the key open challenge with SOTA mAP of 45-55% on ActivityNet, far below image-level recognition. source_title: "ScienceDirect AEJ (2026) -- Action recognition: A comprehensive survey of tasks, methods, and datasets" source_url: https://www.sciencedirect.com/science/article/pii/S2405959525001869 confidence: high primary_sources: - id: ps-video-understanding-1 title: "About Time: Advances, Challenges, and Outlooks of Action Understanding" type: academic_paper year: 2025 institution: Springer International Journal of Computer Vision (IJCV) url: https://link.springer.com/article/10.1007/s11263-025-02478-4 - id: ps-video-understanding-2 title: "Action recognition: A comprehensive survey of tasks, methods, and datasets" type: academic_paper year: 2026 institution: Alexandria Engineering Journal / Elsevier url: https://www.sciencedirect.com/science/article/pii/S2405959525001869 known_gaps: - Long-term video understanding (>10 minutes) with hierarchical temporal reasoning - Fine-grained action detection in egocentric (first-person) video disputed_statements: [] secondary_sources: - title: "CNNs, RNNs and Transformers in Human Action Recognition: A Survey and A Hybrid Proposal" type: survey_paper year: 2025 authors: - multiple institution: Artificial Intelligence Review (Springer) url: https://doi.org/10.1007/s10462-025-11388-3 - title: "Action Recognition: A Comprehensive Survey of Tasks, Methods, and Applications" type: survey_paper year: 2026 authors: - multiple institution: Array (ScienceDirect) url: https://doi.org/10.1016/j.array.2025.100386 - title: "From CNNs to Transformers in Multimodal Human Activity Recognition: A Survey" type: survey_paper year: 2024 authors: - multiple institution: ACM Computing Surveys url: https://doi.org/10.1145/3664815 - title: "Understanding Video Transformers: A Review on Architecture, Tasks, and Applications" type: survey_paper year: 2024 authors: - multiple institution: Intelligence & Computing (SPJ) url: https://doi.org/10.34133/icomputing.0143 updated: "2026-05-24" --- ## TL;DR Video understanding teaches AI to comprehend what happens in video -- recognizing actions (jumping, cooking, playing guitar), detecting when actions start and end, and answering natural language questions about video content. From surveillance and sports analytics to robot learning and content moderation, action recognition is the visual backbone of temporal AI. ## Core Explanation Three task granularities: (1) Action classification -- given a trimmed video clip (3-10 seconds), classify the action (Kinetics-400: 400 classes). One label per clip; (2) Temporal action detection -- given an untrimmed video (minutes to hours), detect all action instances with start time, end time, and class label. Much harder due to variable duration and background frames; (3) Spatio-temporal action detection -- add bounding boxes around the person performing each action in each frame (AVA dataset). Architecture evolution: Two-stream (RGB + optical flow) -> 3D CNNs (C3D, I3D, SlowFast -- two pathways: slow for spatial, fast for temporal) -> Video Transformers (TimeSformer, VideoSwin, VideoMAE). Key insight: video has massive temporal redundancy; masked autoencoding (VideoMAE -- mask 90% of video patches, reconstruct) is extremely effective for self-supervised pretraining. ## Detailed Analysis TimeSformer (2021, Meta): applies self-attention along spatial dimensions and temporal dimensions separately (divided space-time attention), reducing compute from O(T^2*S^2) to O(T^2+S^2). VideoMAE (2022): randomly mask 90% of spacetime patches, train to reconstruct -- the extreme masking forces the model to learn high-level semantics rather than copying nearby frames. Achieves SOTA with efficient training. SlowFast (2019): Slow pathway (low frame rate, high spatial resolution) captures spatial semantics (objects, scenes); Fast pathway (high frame rate, low spatial resolution) captures motion. Temporal action detection: THUMOS, ActivityNet benchmarks. Approaches: two-stage (propose segments, classify) and single-stage (anchor-free detection). SOTA around 45-55% mAP on ActivityNet -- far below image detection (60-80% mAP). Video-language models (TemporalVLM): answer questions about video content ("Did the person stir the pot before or after adding salt?"). Applications: surveillance (anomaly detection), sports analytics (player action tracking), content moderation (violent/sensitive content), robot learning (learning manipulation from video).