On the efficiency of user-level parallel file systems

doi:10.7523/j.ucas.2020.0027

Abstract

Abstract: A common and easier way to deploy parallel filesystem is using the user-space file system framework (FUSE). However, it serves as an I/O interposition layer which crosses user and kernel space that may introduce significant overhead due to the user-kernel mode switches, context switches and additional memory copies. This paper designs a method based on dynamic linking techniques to bypass FUSE, and implements the interposition layer in user space. This method is implemented and evaluated in parallel log-structured file system which is a popular parallel file system for checkpointing. The experiments and evaluation show that it can improve 131% of the read performance when the transfer block size is large, and improve 5 times write performance at small transfer block size and guarantee no significant write performance downgrade in other scenarios.

Key words: file system, storage system, FUSE, parallel file system

CLC Number:

TP302

DENG Tongliang, CHEN Chen, YIN Shu. On the efficiency of user-level parallel file systems[J]. Journal of University of Chinese Academy of Sciences, 2022, 39(2): 275-282.

References

[1] Bland B. Titan-early experience with the titan system at oak ridge national laboratory[C]//2012 SC Companion: High Performance Computing, Networking Storage and Analysis. November 10-16, 2012, Salt Lake City, UT, USA. IEEE Press, 2012: 2189-2211.DOI:10.1109/SC.Companion.2012.356.
[2] Livermore’s HPC Innovation Center. Catalyst[EB/OL]. (2017-10-08)[2018-01-01]. http://computation.llnl.gov/computers/catalyst.
[3] Oak Ridge National Laboratory. Summit[EB/OL]. (2018-07-30) [2019-08-07]. https://www.olcf.ornl.gov/summit/.
[4] Ross R B, Thakur R. PVFS: a parallel file system for Linux clusters[C]//Proceedings of the 4th annual Linux showcase and conference. CA, USA: USENIX Association, 2000: 391-430.
[5] Bent J, Gibson G, Grider G, et al. PLFS: a checkpoint filesystem for parallel applications[C]//Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, 2009: 1-12. DOI: 10.1145/1654081.
[6] Braam P. The Lustre Storage Architecture[EB/OL]. (2019-03-05) [2019-10-15]. http://lustre.org/.
[7] 周恩强, 董勇, 张伟,等. 对象存储并行文件系统小文件性能优化研究[J]. 计算机工程与科学, 2013, 35(12): 8-13.DOI:10.3969/j.issn.1007-130X.2013.12.002.
[8] 刘恋, 郑彪, 龚奕利. 分布式文件系统中元数据操作的优化[J]. 计算机应用, 2012, 32(12): 3271-3273.DOI:10.3724/SP.J.1087.2012.03271.
[9] Szeredi M. Fuse: File system in user space[EB/OL]. (2017-09-27) [2017-10-01]. http://fuse.sourceforge.net/.
[10] Rath N. List of fuse based file systems[EB/OL]. (2019-12-22) [2020-04-15]. https://github.com/libfuse/libfuse/wiki/Filesystems.
[11] Tarasov V, Gupta A, Sourav K, et al. Terra incognita: on the practicality of user-space file systems[C]//7th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage 15). CA, USA: USENIX Association, 2015:15-15.
[12] Red Hat Inc. GlusterFS[EB/OL]. (2011-10-07) [2020-05-08]. https://www.gluster.org/.
[13] Hoskins M E. Sshfs: super easy file access over ssh[EB/OL]. (2006-04-28) [2020-05-08]. https://www.linuxjournal.com/article/8904.
[14] Vangoor B K R, Tarasov V, Zadok E. To fuse or not to fuse: performance of user-space file systems[C]//Proceedings of the 15th USENIX Conference on File and Storage Technologies, Santa Clara, CA, USA: USENIX Association, 2017:59-72.
[15] Lipp M, Schwarz M, Gruss D, et al. Meltdown: reading kernel memory from user space[C]//27th USENIX Security Symposium. Baltimore, MD, USA: USENIX Association, 2018: 973-990.
[16] Ishiguro S, Murakami J, Oyama Y, et al. Optimizing local file accesses for FUSE-based distributed storage[C]//2012 SC Companion: High Performance Computing, Networking Storage and Analysis. November 10-16,2012,Salt Lake City, UT, USA. IEEE, 2012: 760-765.DOI:10.1109/SC.Companion.2012.104.
[17] Zhu Y, Wang T, Mohror K, et al. Direct-FUSE: removing the middleman for high-performance FUSE file system support[C]//Proceedings of the 8th International Workshop on Runtime and Operating Systems for Supercomputers. 2018:1-8.DOI:10.1145/3217189.3217195.
[18] Ward L. The SYSIO library[EB/OL]. (2003-02-20) [2019-12-17]. https://sourceforge.net/projects/libsysio/.
[19] Rajgarhia A, Gehani A. Performance and extension of user space file systems[C]//Proceedings of the 2010 ACM Symposium on Applied Computing. 2010: 206-213.DOI:10.1145/1774088.1774130.
[20] 王卫锋, 杨林. 基于Hadoop的邮政寄递大数据分析系统设计与实现[J]. 中国科学院大学学报, 2017, 34(3): 395-400.DOI:10.7523/j.issn.2095-6134.2017.03.014.
[21] Zadok E, Nieh J. FiST: a language for stackable file systems[C]//Proceedings of the Annual Conference on USENIX Annual Technical Conference. CA, USA: USENIX Association, 2000:5-21.
[22] Li C P, Ding C, Shen K. Quantifying the cost of context switch[C]//Proceedings of the 2007 workshop on Experimental computer science. 2007: 2-5.DOI:10.1145/1281700.1281702.
[23] Egwutuoha I P, Levy D, Selic B, et al. A survey of fault tolerance mechanisms and checkpoint/restart implementations for high performance computing systems[J]. The Journal of Supercomputing, 2013, 65(3): 1302-1326.DOI:10.1007/S11227-013-0884-0.
[24] Rosenblum M, Ousterhout J K. The design and implementation of a log-structured file system[J]. ACM Transactions on Computer Systems, 1992, 10(1): 26-52.DOI:10.1145/146941.146943.
[25] Linux Programmer’s Manual. ld.so(8)[EB/OL]. (2020-04-30) [2020-05-08]. http://man7.org/linux/man-pages/man8/ld.so.8.html.
[26] Weil S A, Brandt S A, Miller E L, et al. Ceph: a scalable, high-performance distributed file system[C]//Proceedings of the 7th Symposium on Operating Systems Design and Implementation. CA, USA: USENIX Association, 2006: 307-320.
[27] Los Alamos National Laboratory. LANL fs test [EB/OL]. (2017-05-20) [2017-09-05]. https://github.com/fstest/fs_test.