Design and implementation of postal delivery big data analytic system based on Hadoop

doi:10.7523/j.issn.2095-6134.2017.03.014

›› 2017, Vol. 34 ›› Issue (3): 395-400.DOI: 10.7523/j.issn.2095-6134.2017.03.014

Design and implementation of postal delivery big data analytic system based on Hadoop

WANG Weifeng, YANG Lin

Information Dynamics and Engineering Applications Laboratory, School of Computer and Control Engineering, University of Chinese Academy of Sciences, Beijing 100049, China

Received:2016-09-14 Revised:2016-11-18 Online:2017-05-15

Abstract

Abstract: Facing massive postal delivery data, the existing data warehouse system based on the traditional relational database has problems of high construction cost and analysis capacity bottleneck. Nowadays, Hadoop is widely used in large data storage and analysis, and it has the advantages of high scalability, high performance, and so on. On the basis of studies of the open source framework of Hadoop, combining with practical engineering project, we proposed a delivery data analysis system based on Hadoop. we implemented some parts of the system. We obtained the performance parameters of this system. The parameters can be widely used in future building of the project.

Key words: postal delivery data, Hadoop, big data storage, big data analysis

CLC Number:

TP311

WANG Weifeng, YANG Lin. Design and implementation of postal delivery big data analytic system based on Hadoop[J]. , 2017, 34(3): 395-400.

References

[1] 中国产业研究院. 2015-2022年中国电子商务市场全景调研及投资战略咨询报告[EB/OL].(2015)[2016-09-14]. http://www.chyxx.com/research/201510/349060.html.
[2] Pavlo A, Paulson E, Rasin A. A comparison of approaches to large-scale data analysis[C]//ACM. SIGMOD International Conference on Management of Data. Rhode Island: SIGMOD, 2009: 165-178.
[3] Gunther N, Puglia P, Tomasette K. Hadoop superlinear scalability[J]. Communications of the ACM, 2015, 58(4): 46-55.
[4] Apache Foundation. Apache Hadoop [EB/OL]. (2016-01-20) [2016-09-14]. https://wiki.apache.org/hadoop.
[5] Dean J, Ghemawat S. MapReduce: simplified data processing on large clusters[C]//Conference on Symposium on Opearting Systems Design & Implementation. USENIX Association, 2004: 107-113.
[6] Ghemawat S, Gobioff H, Leung S T. The Google file system[J]. Acm Sigops Operating Systems Review, 2003, 37(5): 29-43.
[7] Chang F, Dean J, Ghemawat S, et al. Bigtable: a distributed storage system for structured data[J]. ACM Transactions on Computer Systems, 2008, 26(2):205-218.
[8] Cohen J, Dolan B, Dunlap M, et al. MAD skills: new analysis practices for big data[J]. Proceedings of the Vldb Endowment, 2009, 2(2): 1 481-1 492.
[9] 覃雄派, 王会举, 杜小勇,等. 大数据分析：RDBMS与MapReduce的竞争与共生[J]. 计算机光盘软件与应用, 2013, 23(7): 55-56.
[10] Laurie B, Laurie P. Apache: the defini-tive guide[M]. 3rd ed. O'Reilly & Associates, 2005:14-60.
[11] 蔡斌,陈湘萍. Hadoop 技术内幕: 深入解析Hadoop Common和HDFS架构设计与实现原理[M]. 北京：机械工业出版社, 2013: 151-184.
[12] 董西成. Hadoop技术内幕: 深入解析MapReduce架构设计与实现原理: in-depth study of mapreduce[M]. 北京：机械工业出版社, 2013:228-240.
[13] 魏迪. 基于Hadoop的海量业务数据分析平台的设计与实现[D]. 北京：北京邮电大学, 2013.
[14] Thusoo A, Sarma J S, Jain N, et al. Hive: a warehousing solution over a map-reduce framework[J]. Proceedings of the Vldb Endowment, 2009, 2(2):1 626-1 629.
[15] Hoffman S. Apache flume: distributed log collection for Hadoop[M]. Birmingham: Packt Publishing, 2013: 24-88.
[16] Gupta S, Dutt N, Gupta R, et al. SPARK: a high-level synthesis framework for applying parallelizing compiler trans-formations[C]//International Conference on Vlsi Design. IEEE Computer Society, 2003:461-466.

Design and implementation of postal delivery big data analytic system based on Hadoop

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 1

Recommended Articles

Metrics

Comments