一种基于双控节点的Ceph写性能优化方法*

doi:10.7523/j.ucas.2020.0051

中国科学院大学学报

一种基于双控节点的Ceph写性能优化方法^*

黄遵祥, 朱磊基, 熊勇^†

中国科学院上海微系统与信息技术研究所,上海 201800;
中国科学院大学,北京 100049;
中国科学院无线传感网与通信重点实验室,上海 201800

收稿日期:2020-07-13 修回日期:2020-11-04 发布日期:2021-05-31
通讯作者: ^†E-mail：yong.xiong@mail.sim.ac.cn
基金资助:
^*全军共用信息系统装备预研专用技术项目（31511030302）资助

A Ceph write performance optimization method based on double-control nodes

HUANG Zunxiang, ZHU Leiji, XIONG Yong

Shanghai Institute of Microsystem and Information Technology Chinese Academy of Sciences,Shanghai 201800,China;
University of Chinese Academy of Sciences,Beijing 100049,China;
Key Lab of Wireless Sensor Network and Communication Chinese Academy of Sciences,Shanghai 201800,China

Received:2020-07-13 Revised:2020-11-04 Published:2021-05-31

摘要/Abstract

摘要： 分布式存储系统Ceph由于采用多副本强一致性写入机制,造成集群写性能不理想。针对该问题,提出一种基于双控节点的Ceph写性能优化方法,首先利用双控双存储阵列节点,当一个控制器出现故障时,该节点中的另一个伙伴控制器创建新的OSD进程并快速接管故障控制器的存储阵列,从而保证数据存储的安全性和高可靠性,同时将写入机制优化为主副本OSD在本地写入日志盘（journal）后,就向客户端返回写完成,之后写入数据盘和其余从副本的完成情况则由主副本OSD继续收集并完成后续各类回调操作,从而降低非必要写操作对集群写性能的影响。最后对数据可用性和集群写性能进行实验测试,其中写性能测试分别从写延迟、吞吐量和IOPS三个角度,对优化后的方法和Ceph原生写入机制在顺序写和随机写两方面进行比较,进一步验证了优化方法在维护数据高可用的同时,对写性能提升的效果。

关键词: 分布式存储, Ceph, 双控节点, 副本一致性

Abstract: Because the distributed storage system Ceph uses a multi-copy strong consistency write mechanism, the cluster write performance is not ideal. To solve this problem, this paper proposes a Ceph write optimization method based on double-control nodes. With double-control double-RAID nodes, when one controller fails, another partner controller in the node creates a new OSD process and quickly takes over the RAID of the failed controller, thereby ensuring the safety and high reliability of data storage. At the same time, the write mechanism is optimized as follows: after the primary OSD is written to the journal, the write completion is returned to the client. After that, the primary OSD continues to collect the completion status of the write data disk and other slave copies, and then completes callback operations. Thereby reducing the impact of unnecessary write operations on the write performance of the cluster. Finally, the data availability and cluster write performance are tested experimentally. The write performance test compares the optimized method and Ceph's native write mechanism in terms of sequential write and random write from three perspectives of write latency, throughput and IOPS. It further verifies the effect of the optimization method on improving write performance while maintaining high data availability.

Key words: distributed storage, Ceph, double-control node, replica consistency

中图分类号:

TP399

黄遵祥, 朱磊基, 熊勇. 一种基于双控节点的Ceph写性能优化方法^*[J]. 中国科学院大学学报, DOI: 10.7523/j.ucas.2020.0051.

HUANG Zunxiang, ZHU Leiji, XIONG Yong. A Ceph write performance optimization method based on double-control nodes[J]. Journal of University of Chinese Academy of Sciences, DOI: 10.7523/j.ucas.2020.0051.

参考文献

[1] Weil S.Ceph: reliable, scalable, and high-performance distributed storage[D]. California: University of California at Santa Cruz, 2007.
[2] Weil S, Leung A, Brandt S, et al.RADOS: A scalable, reliable storage service for petabyte-scale storage clusters[C]// Gibson G A. Proceedings of the 2nd International Petascale Data Storage Workshop. Reno Nevada: Association for Computing Machinery, 2007: 35-44.
[3] Weil S, Brandt S, Miller E, et al.Ceph: A Scalable, High-Performance Distributed File System[C]// Bershad B. Symposium on Operating Systems Design & Implementation. California: USENIX Association, 2006: 307-320.
[4] Weil S, Brandt S, Miller E, et al.CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data[C]// Miller B H. IEEE Sc Conference. New York: ACM, 2006: 122-130.
[5] Weil S, Pollack K, Brandt S, et al.Dynamic Metadata Management for Petabyte-Scale File Systems[C]// Huskamp J. conference on high performance computing. Washington: IEEE Computer Society, 2004: 4-4.
[6] Oh M, Eom J, Yoon J, et al.Performance Optimization for All Flash Scale-Out Storage[C]// Kim S. IEEE International Conference on Cluster Computing. Taipei: IEEE, 2016: 316-325.
[7] Michael B, David D, Sarp O, et al.Asynchronous object storage with QoS for scientific and commercial big data[C]// Hildebrand D. International Conference for High Performance Computing, Networking, Storage and Analysis. Denver Colorado: Association for Computing Machinery, 2013: 7-13.
[8] Zhang X, Wang Y, Wang Q, et al.A New Approach to Double I/O Performance for Ceph Distributed File System in Cloud Computing[C]// Groves B. 2019 2nd International Conference on Data Intelligence and Security. South Padre Island: IEEE, 2019: 68-75.
[9] 刘鑫伟. 基于Ceph分布式存储系统副本一致性研究[D].湖北:华中科技大学,2016.
[10] 姚朋成. Ceph异构存储优化机制研究[D].重庆:重庆邮电大学,2019.
[11] Zhang J, Wu Y, Chung Y C.PROAR: A Weak Consistency Model for Ceph[C]// Jia X H. IEEE International Conference on Parallel & Distributed Systems. WuHan: IEEE, 2017: 347-353.
[12] Hilmi M, Mulyana E, Hendrawan H, et al.Analysis of Network Capacity Effect on Ceph Based Cloud Storage Performance[C]// Garnida H. 2019 IEEE 13th International Conference on Telecommunication Systems, Services, and Applications. Indonesia: IEEE, 2019: 22-24.
[13] Yusuf I N, Mulyana E, Hendrawan H, et al.Utilizing CRUSH Algorithm on Ceph to Build a Cluster of Reliable Data Storage[C]// Arseno D. international conference on telecommunication systems services and applications. Indonesia: IEEE, 2019: 17-21.
[14] Zhan K, Xu L, Yuan Z, et al.Performance Optimization of Large Files Writes to Ceph Based on Multiple Pipelines Algorithm[C]// Hagersten E 2018 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications. Melbourne: IEEE, 2018: 525-532.
[15] Fan Y, Wang Y, Ye M.An Improved Small File Storage Strategy in Ceph File System[C]// Guo P. 2018 14th International Conference on Computational Intelligence and Security (CIS). Hangzhou: IEEE Computer Society, 2018: 488-491.
[16] 邵曦煜, 李京, 周志强. 一种Ceph块设备跨集群迁移算法[J]. 中国科学技术大学学报, 2018, 48(9): 61-67.
[17] Oh M, Park S, Yoon J, et al.Design of Global Data Deduplication for a Scale-Out Distributed Storage System[C]// Dinu F. international conference on distributed computing systems. Vienna: IEEE, 2018: 1063-1073.
[18] Zhan K, Piao A H.Optimization of Ceph Reads/Writes Based on Multi-threaded Algorithms[C]// Brownlee N. high performance computing and communications. Sydney: IEEE, 2016: 719-725.

一种基于双控节点的Ceph写性能优化方法^*

A Ceph write performance optimization method based on double-control nodes

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 2

编辑推荐

Metrics

本文评价

访问统计

联系我们

[1]	黄遵祥, 朱磊基, 熊勇. 一种基于双控节点的Ceph写性能优化方法[J]. 中国科学院大学学报, 2022, 39(6): 817-826.
[2]	侯金钟, 张立波, 罗铁坚. 一种均衡视频资源的分布存储方法[J]. 中国科学院大学学报, 2016, 33(5): 686-692.