基于Hadoop的邮政寄递大数据分析系统设计与实现

doi:10.7523/j.issn.2095-6134.2017.03.014

中国科学院大学学报 ›› 2017, Vol. 34 ›› Issue (3): 395-400.DOI: 10.7523/j.issn.2095-6134.2017.03.014

• 简报 • 上一篇

基于Hadoop的邮政寄递大数据分析系统设计与实现

王卫锋, 杨林

中国科学院大学计算机与控制学院信息动态学与工程应用实验室, 北京 100049

收稿日期:2016-09-14 修回日期:2016-11-18 发布日期:2017-05-15
通讯作者: 王卫锋,E-mail:ry_009@126.com

Design and implementation of postal delivery big data analytic system based on Hadoop

WANG Weifeng, YANG Lin

Information Dynamics and Engineering Applications Laboratory, School of Computer and Control Engineering, University of Chinese Academy of Sciences, Beijing 100049, China

Received:2016-09-14 Revised:2016-11-18 Published:2017-05-15

摘要/Abstract

摘要： 面对海量邮政寄递数据，现有的构建于关系数据库上的数据仓库系统在做数据分析时具有建设成本高、分析能力会遇到瓶颈等缺点。Hadoop具有高可扩展、高性能和低成本等优点，被广泛应用于大数据的存储和分析。基于对Hadoop开源框架的研究，设计邮政寄递大数据分析系统，并对该系统进行部分实现。结合邮政安监系统工程需求展开实验，得出大数据分析系统的性能参数，为后续工程建设提供依据。

关键词: 邮政寄递数据, Hadoop, 大数据存储, 大数据分析

Abstract: Facing massive postal delivery data, the existing data warehouse system based on the traditional relational database has problems of high construction cost and analysis capacity bottleneck. Nowadays, Hadoop is widely used in large data storage and analysis, and it has the advantages of high scalability, high performance, and so on. On the basis of studies of the open source framework of Hadoop, combining with practical engineering project, we proposed a delivery data analysis system based on Hadoop. we implemented some parts of the system. We obtained the performance parameters of this system. The parameters can be widely used in future building of the project.

Key words: postal delivery data, Hadoop, big data storage, big data analysis

中图分类号:

TP311

王卫锋, 杨林. 基于Hadoop的邮政寄递大数据分析系统设计与实现[J]. 中国科学院大学学报, 2017, 34(3): 395-400.

WANG Weifeng, YANG Lin. Design and implementation of postal delivery big data analytic system based on Hadoop[J]. , 2017, 34(3): 395-400.

参考文献

[1] 中国产业研究院. 2015-2022年中国电子商务市场全景调研及投资战略咨询报告[EB/OL].(2015)[2016-09-14]. http://www.chyxx.com/research/201510/349060.html.
[2] Pavlo A, Paulson E, Rasin A. A comparison of approaches to large-scale data analysis[C]//ACM. SIGMOD International Conference on Management of Data. Rhode Island: SIGMOD, 2009: 165-178.
[3] Gunther N, Puglia P, Tomasette K. Hadoop superlinear scalability[J]. Communications of the ACM, 2015, 58(4): 46-55.
[4] Apache Foundation. Apache Hadoop [EB/OL]. (2016-01-20) [2016-09-14]. https://wiki.apache.org/hadoop.
[5] Dean J, Ghemawat S. MapReduce: simplified data processing on large clusters[C]//Conference on Symposium on Opearting Systems Design & Implementation. USENIX Association, 2004: 107-113.
[6] Ghemawat S, Gobioff H, Leung S T. The Google file system[J]. Acm Sigops Operating Systems Review, 2003, 37(5): 29-43.
[7] Chang F, Dean J, Ghemawat S, et al. Bigtable: a distributed storage system for structured data[J]. ACM Transactions on Computer Systems, 2008, 26(2):205-218.
[8] Cohen J, Dolan B, Dunlap M, et al. MAD skills: new analysis practices for big data[J]. Proceedings of the Vldb Endowment, 2009, 2(2): 1 481-1 492.
[9] 覃雄派, 王会举, 杜小勇,等. 大数据分析：RDBMS与MapReduce的竞争与共生[J]. 计算机光盘软件与应用, 2013, 23(7): 55-56.
[10] Laurie B, Laurie P. Apache: the defini-tive guide[M]. 3rd ed. O'Reilly & Associates, 2005:14-60.
[11] 蔡斌,陈湘萍. Hadoop 技术内幕: 深入解析Hadoop Common和HDFS架构设计与实现原理[M]. 北京：机械工业出版社, 2013: 151-184.
[12] 董西成. Hadoop技术内幕: 深入解析MapReduce架构设计与实现原理: in-depth study of mapreduce[M]. 北京：机械工业出版社, 2013:228-240.
[13] 魏迪. 基于Hadoop的海量业务数据分析平台的设计与实现[D]. 北京：北京邮电大学, 2013.
[14] Thusoo A, Sarma J S, Jain N, et al. Hive: a warehousing solution over a map-reduce framework[J]. Proceedings of the Vldb Endowment, 2009, 2(2):1 626-1 629.
[15] Hoffman S. Apache flume: distributed log collection for Hadoop[M]. Birmingham: Packt Publishing, 2013: 24-88.
[16] Gupta S, Dutt N, Gupta R, et al. SPARK: a high-level synthesis framework for applying parallelizing compiler trans-formations[C]//International Conference on Vlsi Design. IEEE Computer Society, 2003:461-466.

基于Hadoop的邮政寄递大数据分析系统设计与实现

Design and implementation of postal delivery big data analytic system based on Hadoop

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 1

编辑推荐

Metrics

本文评价

访问统计

联系我们