Journal of University of Chinese Academy of Sciences >
Skeleton-based video anomaly detection with memory enhancement and diffusion model
Received date: 2024-11-14
Revised date: 2025-04-02
Online published: 2025-04-16
Video anomaly detection （VAD） is a technique aimed at identifying abnormal events within videos and has significant applications in public safety and video content understanding. Traditional VAD methods primarily rely on extracting pixel features from either the entire video frame or specific regions. To reduce the impact of unstructured noise， VAD methods based on human skeletal data have garnered considerable attention. However， in open-set scenarios， these methods face two major challenges： erroneous reconstruction of abnormal behaviors and insufficient generalization to diverse normal behaviors. To address these issues， this paper proposes a skeleton-based video anomaly detection framework with memory enhancement and diffusion model （MEDM-SVAD）. This framework incorporates a memory enhancement module to expand the model’s memory capacity for normal samples， preventing abnormal behaviors from being misclassified due to minimal reconstruction error. Additionally， the diffusion model significantly improves the framework’s ability to generalize to out-of-domain normal behaviors. Experimental results on three public datasets， HR-STC， HR-Avenue， and HR-UBnormal， show that the proposed method MEDM-SVAD achieves AUC scores of 78.5%， 90.1%， and 69.7%， respectively， demonstrating various levels of performance improvement over the current state-of-the-art MoCoDAD algorithm， thus verifying its effectiveness and superiority across multiple scenarios.
Qi ZHANG , Yuan LI , Yanzhao ZHOU , Jianbin JIAO . Skeleton-based video anomaly detection with memory enhancement and diffusion model[J]. Journal of University of Chinese Academy of Sciences, 2026 , 43(5) : 677 -686 . DOI: 10.7523/j.ucas.2025.017
| [1] | Sultani W， Chen C， Shah M. Real-world anomaly detection in surveillance videos［C］//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR）. June 18-23， 2018， Salt Lake City， UT， USA. IEEE， 2018： 6479-6488. DOI： 10.1109/CVPR.2018.00678 . |
| [2] | Hasan M， Choi J， Neumann J， et al. Learning temporal regularity in video sequences［C］//2016 IEEE Conference on Computer Vision and Pattern Recognition（CVPR）. June 27-30， 2016. Las Vegas， NV， USA. IEEE， 2016： 733-742. DOI： 10.1109/CVPR.2016.86 . |
| [3] | Liu W， Luo W X， Lian D Z， et al. Future frame prediction for anomaly detection： a new baseline［C］//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR）. June 18-23， 2018， Salt Lake City， UT， USA. IEEE， 2018： 6536-6545. DOI： 10.1109/CVPR.2018.00684 . |
| [4] | Luo W X， Liu W， Gao S H. A revisit of sparse coding based anomaly detection in stacked RNN framework［C］//2017 IEEE International Conference on Computer Vision（ICCV）. October 22-29， 2017， Venice， Italy. IEEE， 2017： 341-349. DOI： 10.1109/ICCV.2017.45 . |
| [5] | Chong Y S， Tay Y H. Abnormal event detection in videos using spatiotemporal autoencoder［C］//Advances in Neural Networks-ISNN 2017. Cham： Springer International Publishing， 2017： 189-196. DOI： 10.1007/978-3-319-59081-3_23 . |
| [6] | Xu D， Yan Y， Ricci E， et al. Detecting anomalous events in videos by learning deep representations of appearance and motion［J］. Computer Vision and Image Understanding， 2017， 156： 117-127. DOI： 10.1016/j.cviu.2016.10.010 . |
| [7] | Hinami R， Mei T， Satoh S. Joint detection and recounting of abnormal events by learning deep generic knowledge［C］//2017 IEEE International Conference on Computer Vision（ICCV）. October 22-29， 2017， Venice， Italy. IEEE， 2017： 3639-3647. DOI： 10.1109/ICCV.2017.391 . |
| [8] | Co?ar S， Donatiello G， Bogorny V， et al. Toward abnormal trajectory and event detection in video surveillance［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2017， 27（3）： 683-695. DOI： 10.1109/TCSVT.2016.2589859 . |
| [9] | Zimek A， Schubert E， Kriegel H P. A survey on unsupervised outlier detection in high‐dimensional numerical data［J］. Statistical Analysis and Data Mining： the ASA Data Science Journal， 2012， 5（5）： 363-387. DOI： 10.1002/sam.11161 . |
| [10] | Morais R， Le V， Tran T， et al. Learning regularity in skeleton trajectories for anomaly detection in videos［C］//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR）. June 15-20， 2019， Long Beach， CA， USA. IEEE， 2019： 11988-11996. DOI： 10.1109/CVPR.2019.01227 . |
| [11] | Flaborea A， Collorone L， D’ Amely Di Melendugno G M， et al. Multimodal motion conditioned diffusion model for skeleton-based video anomaly detection［C］//2023 IEEE/CVF International Conference on Computer Vision（ICCV）. October 1-6， 2023， Paris， France. IEEE， 2023： 10284-10295. DOI： 10.1109/ICCV51070.2023.00947 . |
| [12] | Jain Y， Sharma A K， Velmurugan R， et al. PoseCVAE： anomalous human activity detection［C］//2020 25th International Conference on Pattern Recognition（ICPR）. January 10-15， 2021， Milan， Italy. IEEE， 2021： 2927-2934. DOI： 10.1109/ICPR48806.2021.9412132 . |
| [13] | Markovitz A， Sharir G， Friedman I， et al. Graph embedded pose clustering for anomaly detection［C］//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR）. June 13-19， 2020， Seattle， WA， USA. IEEE， 2020： 10536-10544. DOI： 10.1109/CVPR42600.2020.01055 . |
| [14] | Rodrigues R， Bhargava N， Velmurugan R， et al. Multi-timescale trajectory prediction for abnormal human activity detection［C］//2020 IEEE Winter Conference on Applications of Computer Vision（WACV）. March 1-5， 2020， Snowmass Village， CO， USA. IEEE， 2020： 2626-2634. DOI： 10.1109/WACV45572.2020.9093633 . |
| [15] | Barbalau A， Ionescu R T， Georgescu M I， et al. SSMTL++： revisiting self-supervised multi-task learning for video anomaly detection［J］. Computer Vision and Image Understanding， 2023， 229： 103656. DOI： 10.1016/j.cviu.2023.103656 . |
| [16] | Flaborea A， D’ Amely di Melendugno G M， D’Arrigo S， et al. Contracting skeletal kinematics for human-related video anomaly detection［J］. Pattern Recognition， 2024， 156： 110817. DOI： 10.1016/j.patcog.2024.110817 . |
| [17] | Kanu-Asiegbu A M， Vasudevan R， Du X X. BiPOCO： Bi-directional trajectory prediction with pose constraints for pedestrian anomaly detection［EB/OL］. arXiv 2022：2207.02281v1. （2022-07-05） ［2025-03-18］. . |
| [18] | Li N J， Chang F L， Liu C S. Human-related anomalous event detection via spatial-temporal graph convolutional autoencoder with embedded long short-term memory network［J］. Neurocomputing， 2022， 490： 482-494. DOI： 10.1016/j.neucom.2021.12.023 . |
| [19] | Luo W X， Liu W， Gao S H. Normal graph： spatial temporal graph convolutional networks based prediction network for skeleton based video anomaly detection［J］. Neurocomputing， 2021， 444： 332-337. DOI： 10.1016/j.neucom.2019.12.148 . |
| [20] | 刘禹含， 吉根林， 张红苹. 基于骨架图与混合注意力的视频行人异常检测方法［J］. 计算机应用， 2024， 44（8）： 2551-2557. DOI： 10.11772/j.issn.1001-9081.2023081157 . |
| [21] | 杨学存， 李杰华， 陈丽媛， 等. 基于人体骨架的扶梯乘客异常行为识别方法［J］. 安全与环境学报， 2024， 24（2）： 636-643. DOI： 10.13637/j.issn.1009-6094.2022.2404 . |
| [22] | 朱高伟. 基于骨架数据的列车司机异常行为检测［J］. 计算机科学与应用， 2024， 14（7）： 42-50. DOI： 10.12677/csa.2024.147162 . |
| [23] | Acsintoae A， Florescu A， Georgescu M I， et al. UBnormal： new benchmark for supervised open-set video anomaly detection［C］//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. June 18-24， 2022， New Orleans， LA， USA. IEEE， 2022： 20111-20121. DOI： 10.1109/CVPR52688.2022.01951 . |
| [24] | Gong D， Liu L Q， Le V， et al. Memorizing normality to detect anomaly： memory-augmented deep autoencoder for unsupervised anomaly detection［C］//2019 IEEE/CVF International Conference on Computer Vision（ICCV）. October 27-November 2， 2019， Seoul， Korea. IEEE， 2019： 1705-1714. DOI： 10.1109/ICCV.2019.00179 . |
| [25] | Nguyen T N， Meunier J. Anomaly detection in video sequence with appearance-motion correspondence［C］//2019 IEEE/CVF International Conference on Computer Vision （ICCV）. October 27-November 2， 2019， Seoul， Korea. IEEE， 2019： 1273-1283. DOI： 10.1109/ICCV.2019.00136 . |
| [26] | Maas A L， Hannun A Y， Ng A Y. Rectifier nonlinearities improve neural network acoustic models［C］//2013 International Conference on Machine Learning （ICML）. June 16-21， 2013， Atlanta， Georgia， USA. PMLR， 2013， 30（1）： 3. |
| [27] | Fan Y X， Wen G J， Li D R， et al. Video anomaly detection and localization via gaussian mixture fully convolutional variational autoencoder［J］. Computer Vision and Image Understanding， 2020， 195： 102920. DOI： 10.1016/j.cviu.2020.102920 . |
| [28] | Lu Y W， Kumar K M， Nabavi S S， et al. Future frame prediction using convolutional VRNN for anomaly detection［C］//2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance（AVSS）. September 18-21， 2019， Taipei， China. IEEE， 2019： 1-8. DOI： 10.1109/avss.2019.8909850 . |
| [29] | Yu G， Wang S Q， Cai Z P， et al. Cloze test helps： effective video anomaly detection via learning to complete video events［C］//Proceedings of the 28th ACM International Conference on Multimedia. October 12-16， 2020， Seattle， WA， USA. ACM， 2020： 583-591. DOI： 10.1145/3394171.3413973 . |
| [30] | Liu Z A， Nie Y W， Long C J， et al. A hybrid video anomaly detection framework via memory-augmented flow reconstruction and flow-guided frame prediction［C］//2021 IEEE/CVF International Conference on Computer Vision（ICCV）. October 10-17， 2021， Montreal， QC， Canada. IEEE， 2021： 13568-13577. DOI： 10.1109/ICCV48922.2021.01333 . |
| [31] | Zhang X Y， Li N Q， Li J W， et al. Unsupervised surface anomaly detection with diffusion probabilistic model［C］//2023 IEEE/CVF International Conference on Computer Vision（ICCV）. October 1-6， 2023， Paris， France. IEEE， 2023： 6759-6768. DOI： 10.1109/ICCV51070.2023.00624 . |
| [32] | Ho J， Jain A， Abbeel P. Denoising diffusion probabilistic models［EB/OL］. arXiv 2020： 2006.11239v2. （2020-12-16） ［2025-03-18］. . |
| [33] | Xiao Z S， Kreis K， Vahdat A. Tackling the generative learning trilemma with denoising diffusion GANs［EB/OL］. arXiv 2022： 2112.07804v2. （2022-04-04） ［2025-03-18］. . |
| [34] | Krizhevsky A， Sutskever I， Hinton G E. ImageNet classification with deep convolutional neural networks［J］. Communications of the ACM， 2017， 60（6）： 84-90. DOI： 10.1145/3065386 . |
| [35] | Sofianos T， Sampieri A， Franco L， et al. Space-time-separable graph convolutional network for pose forecasting［C］//2021 IEEE/CVF International Conference on Computer Vision（ICCV）. October 10-17， 2021， Montreal， QC， Canada. IEEE， 2021： 11189-11198. DOI： 10.1109/ICCV48922.2021.01102 . |
| [36] | He K M， Zhang X Y， Ren S Q， et al. Deep residual learning for image recognition［C］//2016 IEEE Conference on Computer Vision and Pattern Recognition（CVPR）. June 27-30， 2016， Las Vegas， NV， USA. IEEE， 2016： 770-778. DOI： 10.1109/CVPR.2016.90 . |
| [37] | Ronneberger O， Fischer P， Brox T. U-Net： convolutional networks for biomedical image segmentation［C］// Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015. October 5-9， 2015， Munich， Germany. Cham： Springer International Publishing， 2015： 234-241. DOI： 10.1007/978-3-319-24574-4_28 . |
| [38] | Shi L S， Wang L， Zhou S P， et al. Trajectory unified transformer for pedestrian trajectory prediction［C］//2023 IEEE/CVF International Conference on Computer Vision（ICCV）. October 1-6， 2023， Paris， France. IEEE， 2023： 9641-9650. DOI： 10.1109/ICCV51070.2023.00887 . |
| [39] | 余保玲， 虞松坤， 孙耀然， 等. 基于DeepPose和Faster RCNN的多目标人体骨骼节点检测算法［J］. 中国科学院大学学报， 2020， 37（6）： 828-834. DOI：10.7523/j.issn.2095-6134.2020.06.015 . |
| [40] | Zaremba W， Sutskever I， Vinyals O. Recurrent neural network regularization［EB/OL］. arXiv 2015： 1409.2329v5. （2015-02-19） ［2025-03-18］. . |
/
| 〈 |
|
〉 |