收藏 分销(赏)

基于hadoop的网络数据分析.pptx

上传人:仙人****88 文档编号:14770661 上传时间:2026-11-18 格式:PPTX 页数:27 大小:2.19MB 下载积分:10 金币
下载 相关
基于hadoop的网络数据分析.pptx_第1页
第1页 / 共27页
基于hadoop的网络数据分析.pptx_第2页
第2页 / 共27页


点击查看更多>>
资源描述
,2012-03-12,#,Hadoop-based Internet Traffic Analysis,2012-03-14,Youngseok Lee,leecnu.ac.kr,Chungnam National University,1,Gartner:10 Key IT Trends for 2012,2,Data Explosion,Digital universe,2005:130 ExaByte,2 time per two years,90,%of the data in the world today was created within the past two years,Motivation,Cheap storage,Mobile computing,3,Big Data,Big,data,data sets whose size is beyond the ability of commonly used software tools to capture,manage,and process the data within a tolerable elapsed,time,Example,Walmart:1 million customer transactions every hour,2.5 PB of data,Facebook:40 billion photos from users,2.5,PB of user data+15 TB/day(4/2009),eBay has 6.5 PB of user data+50 TB/day(5/2009),4,Predicting Flu Trends with Queries,Ginsberg,J.,et al.(2009).Detecting,influenza epidemics using search engine query,data.Nature,457,1012-1014.,5,Cutting Edge of Distributed/Parallel Computing,Distributed:MapReduce,1 PB sorting by Google,2008:6 hours and 2 minutes on 4,000 computers,2011:33 minutes on 8000 computers,Parallel:multi-core CPU,GPU,FPGA,40Gbps IP forwarding capability,S.Han et al.,“PacketShader:A GPU-Accelerated Software Router“ACM SIGCOMM,2010,6,Open-source framework for running applications on large clusters built of commodity hardware,Implementation of MapReduce and HDFS,MapReduce:computational paradigm,HDFS:distributed file system,7,MapReduce,MapReduce is a programming model for large data set,First suggested by Google,J.Dean and S.Ghemawat,“MapReduce:Simplified Data Processing on Large Cluster,”OSDI,2004 8,User only specify a map and a reduce function,Distributed grep,Count of URL access frequency,Reverse web-link graph,Term-vector per host,Inverted index,Distributed sort,8,Data from Internet Measurement,Web,web doc,blog,twitter,IP packets,raw,header,dns,Flow,Cisco NetFlow v5/v9,Log,Server,switch,router,sensor,machine,appliances,Management data,SNMP MIB,Routing table,BGP update,table,Active probing,ping,traceroute,skitter,9,How Large?,Type Packets,4.6TB for 1 hr in 10GE,NetFlow,1.2GB for 1 day for a router in CNU,Routing table,Routeviews in bzip2,RIBS:45MB every 2 hrs-540MB for 1 day-197GB for 1 yr,Updates:1MB for every 15 minutes,10 Gbps,10 Gbps,10 Gbps,4.5 TB,10,CAIDA Data,Data sources,Macroscopic topology data(IPv4/IPv6)by Archipelago,Traffic traces at Internet core backbone links,Traffic traces from UCSD Network Telescope,M.Fomenkov and k.claffy,“Internet measurement data management challenges,in Workshop on Research Data Lifecycle Management.Jul 2011,Workshop on Research Data Lifecycle Management.,11,Tools or Platform for Measurement and Analysis,Open source,CAIDA,Active probing tools,Ping,traceroute,skitter,mper,scamper,Archipelago,Passive tools,CoralReef,flow-tools,tcpdump/wireshark,Bro,snort,12,Our Research,Problem statement,How to handle a large set of packet/flow data?,Input:packet/flow files/streaming,Output:statistics/analytics results,Ex)top 10 heavy users,popular applications,favorite websites,anomaly detection,Proposal,Leverage the distributed parallel computing platform,Googles programming model,MapReduce,Open-source system,Hadoop,13,1 An Internet Traffic Analysis Method with MapReduce,Cloudman workshop,April 2010,2,“,A Hadoop-based Packet Trace Processing Tool,TMA 2011,April,2011,Internet Traffic Measurement&Analysis,Packet,HTTP request packet,DNS packet,Abstract of a set of packets:,flow,(168.188.1.10,61.10.1.1,3000,80)1MB,1000 packets,30 secs,Flow,Data,Routers,High Performance Server,Storage,14,Packet,Data,Fiber splitter,Architecture of Traffic Measurement and Analysis System,Institution C,Institution A,Hadoop-based,Traffic Measurement and Analysis Cloud,Hadoop-based Traffic Measurement and Analysis Cloud,(Public),Institution B,Institution D,flow streaming data,packet streaming data,collected packet file,packet streaming data,Administrator A,Administrator B,Administrator C,collected flow file,Slave,Master,Traffic,Collector,Web Visualizer,Pcap,InputFormat,Text Input/Output Format,Traffic Analysis,Mapper&Reducer,NetFlow,Traffic Analyzer,HDFS,Hadoop,Packets,Binary Input/Output Format,Hadoop I/O Formats,15,Our Tools,Flow processing tool,MapReduce algorithm to summarize flow statistics,Packet processing tool,16,Analysis in Hadoop,17,1 TB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,Reduce Task,Map Task,Map Task,Map Task,Map Task,Reduce Task,Map Task,Map Task,Map Task,Map Task,MapReduce,HDFS,Packet Analysis in Hadoop,18,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,TextInputFormat,1 TB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,Text,MapReduce,HDFS,Packet Analysis in Hadoop,19,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,1 TB,Separator?,Which InputFormat?,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,InputFormat in Hadoop,20,Detect a packet in the HDFS block,00 00 68 2B AD 4C 38 A4 04 00 5C 00 00 00 5C 00 00 00 FF FF 00 21 B5 01 68 2B AD 4C 2B 1C 07 00 3C 00 00 00 3C 00 00 00 01 80,Read Packets from HDFS,21,Block2,Block3,New InputFormat,00 00 68 2B AD 4C 38 A4 04 00 5C 00 00 00 5C 00 00 00 FF FF 01 68 2B AD 4C 2B 1C 07 00 3C 00 00 00 3C 00 00 00 01 80,Detect the first Packet,22,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-01 2010-10-10,block,boundary,92 bytes,Start time:2010-10-01,End time:2010-10-10,time :60 sec,16 Byte,16 Byte,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,Start time:2010-10-01,End time:2010-10-10,time :60 sec,Start time:2010-10-01,End time:2010-10-10,time :60 sec,60 sec,OK!,OK!,OK!,Packet Analysis Architecture,23,P,3,I/O Formats,Hadoop,HDFS,Packet,Collector,Packet Analyzer,(Mapper&Reducer),packet,packet trace file,BinaryInput/,OutputFormat,Pcap,InputFormat,TextInput/,OutputFormat,Our Packet Analysis Tool,24,Traffic,Analysis,P,3,Command,MapReduce,Jobs,CoralReef,Total traffic and host/port count statistics,PcapTotalStats,count up bytes/packets,emit unique IP/port and flow,crl_stats,Csource=pcap file,summarize#of unique,IP/Port and flows,Periodic flow statistics,PcapTotalFlowStats,generate flows and count up,bytes/packets,crl_flow I b Ci=interval,Csource=pcap file,/t2_rate-s,summarize,periodical flow statistics,Periodic simple traffic statistics,PcapRate,compute periodic bytes/packets regarding IPv4/v6/non-IP per interval,crl_rate Ci=interval,Csource=pcap file,Top N,PcapTopN,sort records and emit top N record,command,line/t2_top Sb nn,Total,c,ount grouping by key,PcapCountUp,count,up bytes/packets per key,-,Performance Evaluation,25,Testbed,Packet Trace files,Result,26,Conclusion,Hadoop fits well into big Internet data analysis,Future Internet design and modeling,Hadoop,One size does not fit all!,Hadoop+DB,The,data center,is the computer,!,More data usually beats better algorithm,27,
展开阅读全文

开通  VIP、SVIP  下载更划算
下载10份以上建议开通 VIP 会员
下载20份以上建议开通SVIP会员


开通VIP      成为共赢上传

当前位置:首页 > 学术论文 > 其他

移动网页_全站_页脚广告1

关于我们      便捷服务       自信AI       AI导航        关注我们

©2010-2026 宁波自信网络信息技术有限公司  版权所有

客服电话:0574-28810668  投诉电话:18658249818

gongan.png浙公网安备33021202000488号   

icp.png浙ICP备2021020529号-1  |  浙B2-20240490  

关注我们 :微信公众号    抖音    微博    LOFTER 

客服