点集匹配是计算机视觉和模式识别中的重要问题，在目标识别、医学图像配准、姿态估计等方面都得到广泛应用。提出基于机器学习的端对端模型——multi-pointer network（MPN）来解决点集匹配问题。该网络模型利用多标签分类的思想，改进pointer network。以前的模型只输出输入序列的一个元素，而MPN模型选择输入序列中的一组元素作为输出。首先，把点集匹配问题转换为序列问题。这样，网络的输入为顶点的坐标序列，输出为点对之间的对应关系。利用这种方式，可以解决相对于整个空间的平移变换和其他大幅度的刚性变换。实验结果表明，模型也可以被推广解决其他带结构的组合优化问题，如三角剖分等。
Point matching is an important issue of computer vision and pattern recognition, and it is widely used in target recognition, medical image, pose estimation, etc. In this study, we propose a novel end-to-end model (multi-pointer network) based on machine learning method to solve this problem. We capitalize on the idea of multi-label classification to ameliorate the pointer network. Instead of outputting a member of input sequence, our model selects a set of input elements as output. Considering matching problem as a sequential manner, our model takes the coordinates of points as input and outputs correspondences directly. Using this new method, we can effectively solve the translation of the whole space and other large-scale rigid transformations. Furthermore, experiment results show that our model can be generalized to other combinatorial optimization problems in which the output is a subset of input, like Delaunay triangulation.
[1] Brendel W, Todorovic S. Learning spatiotemporal graphs of human activities[C]//International Conference on Computer Vision. Barcelona:IEEE, 2011:778-785.
[2] Zheng D, Xiong H, Zheng Y F, et al. A structured learning-based graph matching for dynamic multiple object tracking[C]//International Conference on Image Processing. Brussels:IEEE, 2011:2333-2336.
[3] Chu L, Jiang S, Wang S, et al. Robust spatial consistency graph model for partial duplicate image retrieval[J]. IEEE Transactions on Multimedia, 2013, 15(8):1982-1996.
[4] Fischler M A, Bolles R C. Random sample consensus:a paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM, 1981, 24(6):381-395
[5] Bolles R C. Robust feature matching through maximal cliques[J]. Proceedings of SPIE-The International Society for Optical Engineering, 1979, 182:140-149.
[6] Rumelhart D E, Hinton G E, Williams R J, et al. Learning representations by back-propagating errors[J]. Nature, 1988, 323(6088):696-699.
[7] Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 1997, 9(8):1735-1780.
[8] Sutskever I, Vinyals O, Le Q V, et al. Sequence to sequence learning with neural networks[C]//Neural Information Processing Systems. Montreal:2014:3104-3112.
[9] Vinyals O, Kaiser L, Koo T, et al. Grammar as a foreign language[C]//Neural Information Processing Systems. Montreal:2015:2773-2781.
[10] Vinyals O, Toshev A, Bengio S, et al. Show and tell:a neural image caption generator[C]//Computer Vision and Pattern Recognition. Boston:IEEE, 2015:3156-3164.
[11] Donahue J, Hendricks L A, Guadarrama S, et al. Long-term recurrent convolutional networks for visual recognition and description[C]//Computer Vision and Pattern Recognition. Boston:IEEE, 2015:2625-2634.
[12] Vinyals O, Fortunato M, Jaitly N, et al. Pointer networks[C]//Neural Information Processing Systems. Montreal:2015:2692-2700.
[13] Milan A, Rezatofighi S H, Garg R, et al. Data-driven approximations to NP-hard problems[C]//Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. San Francisco:AAAI Press, 2017:1453-1459.
[14] Bello I, Pham H, Le Q V, et al. Neural combinatorial optimization with reinforcement learning[J]. arXiv preprint arXiv:1611.09940, 2016.
[15] Cho K, Van Merrienboer B, Gulcehre C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]//Empirical Methods in Natural Language Processing. Doha:Association for Computational Linguistics, 2014:1724-1734.
[16] Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate[J]. arXiv preprint arXiv:1409.0473, 2014.
[17] Zhang M, Zhou Z. A review on multi-label learning algorithms[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(8):1819-1837.
[18] Tsoumakas G, Katakis I. Multi-label classification:an overview[J]. International Journal of Data Warehousing and Mining, 2007, 3(3):1-13.
[19] Liu I, Ramakrishnan B. Bach in 2014:music composition with recurrent neural network[J]. arXiv preprint arXiv:1412.3191, 2014.
[20] Yeung S, Russakovsky O, Jin N, et al. Every moment counts:dense detailed labeling of actions in complex videos[J]. International Journal of Computer Vision, 2018, 126(2-4):375-389.
[21] Lipton Z C, Kale D C, Elkan C, et al. Learning to diagnose with LSTM recurrent neural networks[J]. arXiv preprint arXiv:1511.03677, 2015.
[22] Schuster M, Paliwal K K. Bidirectional recurrent neural networks[J]. IEEE Transactions on Signal Processing, 1997, 45(11):2673-2681.
[23] Vinyals O, Bengio S, Kudlur M. Order matters:sequence to sequence for sets[J]. arXiv preprint arXiv:1511.06391, 2015.
[24] Abadi M, Barham P, Chen J, et al. Tensorflow:a system for large-scale machine learning[C]//Operating Systems Design and Implementation. Savannah:USENIX, 2016, 16:265-283.
