一种基于long short-term memory的唇语识别方法

doi:10.7523/j.issn.2095-6134.2018.01.015

中国科学院大学学报 ›› 2018, Vol. 35 ›› Issue (1): 109-117.DOI: 10.7523/j.issn.2095-6134.2018.01.015

一种基于long short-term memory的唇语识别方法

马宁^1,2, 田国栋², 周曦²

1. 中国科学院大学, 北京 100049;
2. 中国科学院重庆绿色智能技术研究院, 重庆 400714

收稿日期:2016-11-23 修回日期:2017-03-15 发布日期:2018-01-15
通讯作者: 马宁
基金资助:
国家自然科学基金（61472386，61502444）和中国科学院战略性先导科技专项子课题（XDA06040103）资助

A lip-reading recognition approach based on long short-term memory

MA Ning^1,2, TIAN Guodong², ZHOU Xi²

1. University of Chinese Academy of Sciences, Beijing 100049, China;
2. Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China

Received:2016-11-23 Revised:2017-03-15 Published:2018-01-15

摘要/Abstract

摘要： 唇动视觉信息是说话内容的重要载体。受嘴唇外观、背景信息和说话习惯等影响，即使说话者说相同的内容，唇动视觉信息也会相差很大。为解决唇语视觉信息多样性的问题，提出一种基于long short-term memory（LSTM）的新的唇语识别方法。以往大多数的方法从嘴唇外表信息入手。本方法用嘴唇关键点坐标描述嘴唇形变信息作为唇语视频的特征，它具有类内一致性和类间区分性的特点。然后利用LSTM对特征进行时序编码，它能学习具有区分性和泛化性的空间-时序特征。在公开的唇语数据集GRID、MIRACL-VC和OuluVS上对本方法做了针对分割的单词或短语的说话者独立的唇语识别评估。在GRID和MIRACL-VC上，本方法的准确率比传统方法至少高30%；在OuluVS上，本方法的准确率接近于最优结果。以上实验结果表明，本文提出的基于LSTM的唇语识别方法有效地解决了唇语视觉信息多样性的问题。

关键词: 唇语识别, long short-term memory, 计算机视觉

Abstract: Visual speech information is the important carrier of conversation. However, visual speech informations from different speakers are different due to various appearances of lips, various backgrounds, and various talking ways even the content of the conversation is the same. To address the problem of variety of visual speech information, we propose a new approach for lip-reading recognition based on long short-term memory (LSTM). We compute the positions of lip landmarks which describe the dynamic information of the shape as the features of the lip-reading video, and it has the characteristics of within-class consistency and between-class distinctiveness. Then we use LSTM to encode temporal information, and it learns spatio-temporal features which have the ability of discrimination and generalization. Our approach is evaluated on three public databases (GRID, MRIALC, and OuluVS) for lip-reading recognition of isolated words or phrases in speaker independent experiments. On GRID and MRIALC, the accuracy of our approach is more than 30% highter than that of the conventional approach. On OuluVS, the accuracy of our approach is comparable to state of the art. The experiment results indicate that our lip-reading recognition approach solves the problem of variety of visual speech information effectively.

Key words: lip-reading recognition, long short-term memory, computer vision

中图分类号:

TP391

马宁, 田国栋, 周曦. 一种基于long short-term memory的唇语识别方法[J]. 中国科学院大学学报, 2018, 35(1): 109-117.

MA Ning, TIAN Guodong, ZHOU Xi. A lip-reading recognition approach based on long short-term memory[J]. , 2018, 35(1): 109-117.

参考文献

[1] McGurk H, MacDonald J. Hearing lips and seeing voices[J]. Nature, 1976, 264:746-748.
[2] Graves A. Supervised sequence labelling with recurrent neural networks[M]. Berlin:Springer Berlin Heidelberg, 2012.
[3] Zhou Z, Zhao G, Hong X, et al. A review of recent advances in visual speech decoding[J]. Image & Vision Computing, 2014, 32:590-605.
[4] Zhao G, Barnard M, Pietikainen M. Lipreading With Local Spatiotemporal Descriptors[J].IEEE transactions on multimedia, 2009, 11:1254-1265.
[5] Zhou Z, Zhao G, Pietikainen M, Towards a practical lipreading system[C]//Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011:137-144.
[6] Bakry A, Elgammal A. Mkpls:manifold kernel partial least squares for lipreading and speaker identification[C]//Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013:684-691.
[7] Lan Y, Harvey R, Theobald B J, et al. Comparing visual features for lipreading[C]//International Conference on Auditory-Visual Speech Processing. 2009:102-106.
[8] Pei Y, Kim T K, Zha H. Unsupervised random forest manifold alignment for lipreading[C]//Proceedings of the IEEE International Conference on Computer Vision. 2013:129-136.
[9] Rekik A, Ben-Hamadou A, Mahdi W. A new visual speech recognition approach for RGB-D cameras[C]//International Conference Image Analysis and Recognition. Springer International Publishing, 2014:21-28.
[10] Noda K, Yamaguchi Y, Nakadai K, et al. Lipreading using convolutional neural network[J]. Interspeech, 2014:1149-1153.
[11] Graves A, Jaitly N. Towards end-to-end speech recognition with recurrent neural networks[C]//Proceedings of the 31st International Conference on Machine Learning, 2014:1764-1772.
[12] Sutskever I, Vinyals O, Le Q V. Sequence to sequence learning with neural networks[C]//Advances in Neural Information Processing Systems, 2014:3104-3112.
[13] Donahue J, Hendricks L A, Guadarrama S, et al. Long-term recurrent convolutional networks for visual recognition and description[C]//Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. IEEE, 2015:2625-2634.
[14] Wand M, Koutn J. Lipreading with long short-term memory[C]//Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016:6115-6119.
[15] Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos[C]//Advances in Neural Information Processing Systems, 2014:568-576.
[16] Cooke M, Barker J, Cunningham S, et al. An audio-visual corpus for speech perception and automatic speech recognition[J]. The Journal of the Acoustical Society of America, 2006, 120:2421-2424.
[17] Xiong X, Torre F D L. Supervised descent method and its applications to face alignment[C]//Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013:532-539.
[18] Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural computation, 1997, 9:1735-1780.
[19] Yarin G, Ghahramani Z. A theoretically grounded application of dropout in recurrent neural networks[C]//Advances in Neural Information Processing Systems, 2016:1019-1027.

一种基于long short-term memory的唇语识别方法

A lip-reading recognition approach based on long short-term memory

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

访问统计

联系我们

[1]	杨雨龙, 郭田德, 韩丛英. 基于原型学习改进的伪标签半监督学习算法[J]. 中国科学院大学学报, 2021, 38(6): 841-851.
[2]	史达亨, 刘立刚, 周斌, 卜智勇. 跨时间迁移的多源无线信号指纹融合定位方法[J]. 中国科学院大学学报, 2021, 38(6): 817-824.
[3]	庄子俊, 袁晓兵, 裴俊, 王国辉, 刘建坡. 基于无监督表达学习的森林地貌特征建模及林火易发性评估[J]. 中国科学院大学学报, 0, (): 1-1.
[4]	陈汉钦, 秦进, 赵彤, 阎瑶. 面向识别的人脸三维信息估计^*[J]. 中国科学院大学学报, 0, (): 2-2.
[5]	赵敏钧, 赵亚伟, 赵雅捷, 罗刚. 一种新的基于深度学习的重叠关系联合抽取模型[J]. 中国科学院大学学报, 0, (): 55-55.
[6]	马曌月, 肖俊, 王颖. 基于面提取的三维岩体点云孔洞检测与修复方法^*[J]. 中国科学院大学学报, 0, (): 64-64.
[7]	林大权, 范睿, 张良峰. 基于高维特征的图像对抗攻击算法^*[J]. 中国科学院大学学报, 0, (): 74-74.
[8]	王贺, 卜智勇, 谭冲. 基于核相关滤波视觉检测的多旋翼无人机对地目标跟踪与逼近[J]. 中国科学院大学学报, 0, (): 21532-21532.
[9]	赵心驰, 姜策, 何为. 基于注意力生成对抗网络的图像强光去除[J]. 中国科学院大学学报, 0, (): 21548-21548.
[10]	马莹莹, 吴幼龙, 唐华. 基于特征编码和图嵌入的姓名消歧方法[J]. 中国科学院大学学报, 0, (): 21549-21549.
[11]	张韬, 肖俊, 王颖. 基于匹配点逐层过滤的岩体点云配准方法[J]. 中国科学院大学学报, 0, (): 21550-21550.
[12]	杨善强, 李华旺, 常亮, 高才栋, 虞业泺. 一种基于RPC的组态化卫星模拟器故障注入方法[J]. 中国科学院大学学报, 2021, 38(3): 402-408.
[13]	盛晓光, 王颖, 张迎伟, 项若曦, 付红萍. WILS：面向学业预警的非均衡增量式学习方法*[J]. 中国科学院大学学报, 0, (): 17-18.
[14]	周龙, 王伟强, 吕科. FACR：一种快速且准确的车辆识别器[J]. 中国科学院大学学报, 2021, 38(1): 130-136.
[15]	胡莉娜, 高盛华. 基于层次化标签的人体解析[J]. 中国科学院大学学报, 2020, 37(6): 820-827.