A lightweight FPGA image preprocessing accelerator scheme for visual navigation

doi:10.7523/j.ucas.2024.063

Abstract

Abstract:

An image preprocessing accelerator scheme based on lightweight and low-cost FPGA has been proposed in this paper to meet the accelerated processing requirements of the visual navigation image frontend. Through efficient pipeline design and parallel processing technology， the designed accelerator integrates key functions such as histogram equalization， FAST feature point detection， and multi-source sensor data time synchronization. This solution solves technical difficulties such as achieving multifunctional integration， meeting real-time requirements， balancing cost and performance， synchronizing multi-source sensor information time， and achieving software hardware collaborative design under limited hardware resources. The proposed solution is based on Xilinx's Zynq-7000 series lightweight FPGA implementation， which greatly reduces image processing latency while achieving low cost. When the FPGA operates at a frequency of 160 MHz， it achieves a processing speed of 150 frame/s for 1 280×720 images， providing a low-cost and high-performance visual navigation image front-end acceleration solution.

Key words: image accelerator, histogram equalization, feature point extraction, time synchronization, visual navigation, field programmable gate array（FPGA）

CLC Number:

TP911.73

Renkui XUE, Jie ZHANG, Bin LI, Meng LI, Yang WU. A lightweight FPGA image preprocessing accelerator scheme for visual navigation[J]. Journal of University of Chinese Academy of Sciences, 2026, 43(2): 277-287.

Figures/Tables 15

Fig.1 Composition of image accelerator

Fig.2 FPGA MIPI receiving interface resistance matching network

Fig.3 FPGA MIPI transmission interface resistance matching network

Fig.4 Principle of histogram equalization implementation

Fig.5 FAST feature point detection algorithm IP design process

Fig.6 Design of multi-source sensor time synchronization based on FPGA

Fig.7 Schematic diagram of time synchronization timing

Fig.8 Hardware testing platform

Fig.9 Comparison of FPGA and OpenCV processing results

Fig.10 FPGA processing results of histogram equalization and FAST feature point extraction results in different scenarios

Table 1 Comparison of image processing results between FPGA and OpenCV

图像	直方图均衡化结果		特征点检测数量
图像	误差像素数量	灰度值误差	FPGA	OpenCV
图10（a）	15 479	1	5 484	5 484
图10（c）	37 296	1	13 845	13 845
图10（e）	102 358	1	30 807	30 807

Table 2 Time delay between camera and IMU data

序号	时延平均值/μs
1	774.90
2	724.37
3	755.81
4	687.05
5	717.91
6	741.06
均值	733.52
均方差	28.15

Fig.11 Distribution of reprojection errors for AprilTag calibration board and visual IMU joint calibration

Table 3 Reprojection error between camera and IMU data

序号	重投影误差均值/像素
1	0.20
2	0.18
3	0.20
4	0.25
5	0.18
6	0.17

Table 4 FPGA resource consumption

分类	占用	总量	占比/%
LUT	21 180	53 200	40
BRAM	42	140	30
DSP	13	220	6

References 22

[1]	Floreano D， Wood R J. Science， technology and the future of small autonomous drones［J］. Nature， 2015， 521（7553）： 460-466. DOI： 10.1038/nature14542 .
[2]	房德国，王伟，李自然，等. VIO-SLAM综述［J］. 电光与控制， 2020， 27（12）： 58-62， 100. DOI： 10.3969/j.issn.1671-637X.2020.12.013 .
[3]	田野，陈宏巍，王法胜，等. 室内移动机器人的SLAM算法综述［J］. 计算机科学， 2021， 48（9）： 223-234. DOI： 10.11896/jsjkx.200700152 .
[4]	Peng T， Zhang D N， Liu R X， et al. Evaluating the power efficiency of visual SLAM on embedded GPU systems［C］//2019 IEEE National Aerospace and Electronics Conference （NAECON）. Dayton， OH， USA. IEEE， 2019： 117-121. DOI： 10.1109/NAECON46414.2019.9058059 .
[5]	Zhao Y， Xiong Z， Duan S Q， et al. Improved ORB based image registration acceleration algorithm in visual-inertial navigation system［C］//2020 Chinese Automation Congress （CAC）. Shanghai， China. IEEE， 2020： 5714-5718. DOI： 10.1109/CAC51589.2020.9326928 .
[6]	郑泰皓，方子彬. 视觉SLAM特征与提取算法综述［J］. 计算机时代， 2020（7）： 16-21. DOI： 10.16644/j.cnki.cn33-1094/tp.2020.07.005 .
[7]	Vemulapati V， Chen D M. FSLAM： an efficient and accurate SLAM accelerator on SoC FPGAs［C］//2022 International Conference on Field-Programmable Technology （ICFPT）. Hong Kong， China. IEEE， 2022： 1-9. DOI： 10.1109/ICFPT56656.2022.9974562 .
[8]	Taranco R， Arnau J M， González A. A low-power hardware accelerator for ORB feature extraction in self-driving cars［C］//2021 IEEE 33rd International Symposium on Computer Architecture and High Performance Computing （SBAC-PAD）. Belo Horizonte， Brazil. IEEE， 2021： 11-21. DOI： 10.1109/SBAC-PAD53543.2021.00013 .
[9]	Liu Y J， Wu X F. Parallel and pipelining design of SLAM feature detection algorithm in hardware［C］//2021 IEEE 16th Conference on Industrial Electronics and Applications （ICIEA）. Chengdu， China. IEEE， 2021： 1388-1393. DOI： 10.1109/ICIEA51954.2021.9516430 .
[10]	Li J Y， Liu Y， Huang K， et al. An FPGA-based high-throughput keypoint detection accelerator using convolutional neural network for mobile robot applications［C］//2022 IEEE Asia Pacific Conference on Postgraduate Research in Microelectronics and Electronics （PrimeAsia）. Shenzhen， China. IEEE， 2022： 81-84. DOI： 10.1109/PrimeAsia 56064.2022.10104021 .
[11]	Xu Z L， Yu J C， Yu C， et al. CNN-based feature-point extraction for real-time visual SLAM on embedded FPGA［C］//2020 IEEE 28th Annual International Symposium on Field-Programmable Custom Computing Machines （FCCM）. Fayetteville， AR， USA. IEEE， 2020： 33-37. DOI： 10.1109/FCCM48280.2020.00014 .
[12]	丁畅，董丽丽，许文海. “直方图” 均衡化图像增强技术研究综述［J］. 计算机工程与应用， 2017， 53（23）： 12-17. DOI： 10.3778/j.issn.1002-8331.1710-0031 .
[13]	Gauch J M. Investigations of image contrast space defined by variations on histogram equalization［J］. CVGIP： Graphical Models and Image Processing， 1992， 54（4）： 269-280. DOI： 10.1016/1049-9652（92）90074-8 .
[14]	Zuiderveld K. Contrast limited adaptive histogram equalization［M］//Graphics Gems. Amsterdam： Elsevier， 1994： 474-485. DOI： 10.1016/b978-0-12-336156-1.50061-6 .
[15]	彭波，王一鸣. 低照度图像增强算法的研究与实现［J］. 计算机应用， 2007， 27（8）： 2001-2003. DOI： 10.1360/crad20071222 .
[16]	Rosten E， Porter R， Drummond T. Faster and better： a machine learning approach to corner detection［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2010， 32（1）： 105-119. DOI： 10.1109/TPAMI.2008.275 .
[17]	Rosten E， Drummond T. Machine learning for high-speed corner detection［M］//Leonardis A， Bischof H， Pinz A， eds. Computer Vision-ECCV 2006. Berlin， Heidelberg： Springer Berlin Heidelberg， 2006： 430-443. DOI： 10.1007/11744023_34 .
[18]	程彪，黄鲁. 自适应阈值FAST特征点检测算法的FPGA实现［J］. 信息技术与网络安全， 2018， 37（10）： 82-86. DOI： 10.19358/j.issn.2096-5133.2018.10.019 .
[19]	Defossen M. XAPP894（v1.0.1） D-PHY solutions［EB/OL］. Xilinx，（2021-02-01）［2024-04-20］. .
[20]	UG 871（v 2014.1） Vivado design suite tutorial： high-level synthesis［EB/OL］. Xilinx，（2014-05-06）［2024-04-20］. .
[21]	刘延飞，杨铁阡，李琪，等. 多总线地面测试设备中双SDRAM控制器的设计［J］. 电讯技术， 2011， 51（4）： 31-34. DOI： 10.3969/j.issn.1001-893x.2011.04.007 .
[22]	何凯，梁蓓，杨发顺. 基于Vivado HLS的求取特征点图像坐标的设计［J］. 电子科技， 2018， 31（4）： 87-90. DOI： 10.16180/j.cnki.issn1007-7820.2018.04.023 .