资源描述
,2012-03-12,#,Hadoop-based Internet Traffic Analysis,2012-03-14,Youngseok Lee,leecnu.ac.kr,Chungnam National University,1,Gartner:10 Key IT Trends for 2012,2,Data Explosion,Digital universe,2005:130 ExaByte,2 time per two years,90,%of the data in the world today was created within the past two years,Motivation,Cheap storage,Mobile computing,3,Big Data,Big,data,data sets whose size is beyond the ability of commonly used software tools to capture,manage,and process the data within a tolerable elapsed,time,Example,Walmart:1 million customer transactions every hour,2.5 PB of data,Facebook:40 billion photos from users,2.5,PB of user data+15 TB/day(4/2009),eBay has 6.5 PB of user data+50 TB/day(5/2009),4,Predicting Flu Trends with Queries,Ginsberg,J.,et al.(2009).Detecting,influenza epidemics using search engine query,data.Nature,457,1012-1014.,5,Cutting Edge of Distributed/Parallel Computing,Distributed:MapReduce,1 PB sorting by Google,2008:6 hours and 2 minutes on 4,000 computers,2011:33 minutes on 8000 computers,Parallel:multi-core CPU,GPU,FPGA,40Gbps IP forwarding capability,S.Han et al.,“PacketShader:A GPU-Accelerated Software Router“ACM SIGCOMM,2010,6,Open-source framework for running applications on large clusters built of commodity hardware,Implementation of MapReduce and HDFS,MapReduce:computational paradigm,HDFS:distributed file system,7,MapReduce,MapReduce is a programming model for large data set,First suggested by Google,J.Dean and S.Ghemawat,“MapReduce:Simplified Data Processing on Large Cluster,”OSDI,2004 8,User only specify a map and a reduce function,Distributed grep,Count of URL access frequency,Reverse web-link graph,Term-vector per host,Inverted index,Distributed sort,8,Data from Internet Measurement,Web,web doc,blog,twitter,IP packets,raw,header,dns,Flow,Cisco NetFlow v5/v9,Log,Server,switch,router,sensor,machine,appliances,Management data,SNMP MIB,Routing table,BGP update,table,Active probing,ping,traceroute,skitter,9,How Large?,Type Packets,4.6TB for 1 hr in 10GE,NetFlow,1.2GB for 1 day for a router in CNU,Routing table,Routeviews in bzip2,RIBS:45MB every 2 hrs-540MB for 1 day-197GB for 1 yr,Updates:1MB for every 15 minutes,10 Gbps,10 Gbps,10 Gbps,4.5 TB,10,CAIDA Data,Data sources,Macroscopic topology data(IPv4/IPv6)by Archipelago,Traffic traces at Internet core backbone links,Traffic traces from UCSD Network Telescope,M.Fomenkov and k.claffy,“Internet measurement data management challenges,in Workshop on Research Data Lifecycle Management.Jul 2011,Workshop on Research Data Lifecycle Management.,11,Tools or Platform for Measurement and Analysis,Open source,CAIDA,Active probing tools,Ping,traceroute,skitter,mper,scamper,Archipelago,Passive tools,CoralReef,flow-tools,tcpdump/wireshark,Bro,snort,12,Our Research,Problem statement,How to handle a large set of packet/flow data?,Input:packet/flow files/streaming,Output:statistics/analytics results,Ex)top 10 heavy users,popular applications,favorite websites,anomaly detection,Proposal,Leverage the distributed parallel computing platform,Googles programming model,MapReduce,Open-source system,Hadoop,13,1 An Internet Traffic Analysis Method with MapReduce,Cloudman workshop,April 2010,2,“,A Hadoop-based Packet Trace Processing Tool,TMA 2011,April,2011,Internet Traffic Measurement&Analysis,Packet,HTTP request packet,DNS packet,Abstract of a set of packets:,flow,(168.188.1.10,61.10.1.1,3000,80)1MB,1000 packets,30 secs,Flow,Data,Routers,High Performance Server,Storage,14,Packet,Data,Fiber splitter,Architecture of Traffic Measurement and Analysis System,Institution C,Institution A,Hadoop-based,Traffic Measurement and Analysis Cloud,Hadoop-based Traffic Measurement and Analysis Cloud,(Public),Institution B,Institution D,flow streaming data,packet streaming data,collected packet file,packet streaming data,Administrator A,Administrator B,Administrator C,collected flow file,Slave,Master,Traffic,Collector,Web Visualizer,Pcap,InputFormat,Text Input/Output Format,Traffic Analysis,Mapper&Reducer,NetFlow,Traffic Analyzer,HDFS,Hadoop,Packets,Binary Input/Output Format,Hadoop I/O Formats,15,Our Tools,Flow processing tool,MapReduce algorithm to summarize flow statistics,Packet processing tool,16,Analysis in Hadoop,17,1 TB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,Reduce Task,Map Task,Map Task,Map Task,Map Task,Reduce Task,Map Task,Map Task,Map Task,Map Task,MapReduce,HDFS,Packet Analysis in Hadoop,18,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,TextInputFormat,1 TB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,Text,MapReduce,HDFS,Packet Analysis in Hadoop,19,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,1 TB,Separator?,Which InputFormat?,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,64 MB,InputFormat in Hadoop,20,Detect a packet in the HDFS block,00 00 68 2B AD 4C 38 A4 04 00 5C 00 00 00 5C 00 00 00 FF FF 00 21 B5 01 68 2B AD 4C 2B 1C 07 00 3C 00 00 00 3C 00 00 00 01 80,Read Packets from HDFS,21,Block2,Block3,New InputFormat,00 00 68 2B AD 4C 38 A4 04 00 5C 00 00 00 5C 00 00 00 FF FF 01 68 2B AD 4C 2B 1C 07 00 3C 00 00 00 3C 00 00 00 01 80,Detect the first Packet,22,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-01 2010-10-10,block,boundary,92 bytes,Start time:2010-10-01,End time:2010-10-10,time :60 sec,16 Byte,16 Byte,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,92,92,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,2010-10-07,60,60,Timestamp,(sec),Timestamp,(microsec),CapLen,WiredLen,Start time:2010-10-01,End time:2010-10-10,time :60 sec,Start time:2010-10-01,End time:2010-10-10,time :60 sec,60 sec,OK!,OK!,OK!,Packet Analysis Architecture,23,P,3,I/O Formats,Hadoop,HDFS,Packet,Collector,Packet Analyzer,(Mapper&Reducer),packet,packet trace file,BinaryInput/,OutputFormat,Pcap,InputFormat,TextInput/,OutputFormat,Our Packet Analysis Tool,24,Traffic,Analysis,P,3,Command,MapReduce,Jobs,CoralReef,Total traffic and host/port count statistics,PcapTotalStats,count up bytes/packets,emit unique IP/port and flow,crl_stats,Csource=pcap file,summarize#of unique,IP/Port and flows,Periodic flow statistics,PcapTotalFlowStats,generate flows and count up,bytes/packets,crl_flow I b Ci=interval,Csource=pcap file,/t2_rate-s,summarize,periodical flow statistics,Periodic simple traffic statistics,PcapRate,compute periodic bytes/packets regarding IPv4/v6/non-IP per interval,crl_rate Ci=interval,Csource=pcap file,Top N,PcapTopN,sort records and emit top N record,command,line/t2_top Sb nn,Total,c,ount grouping by key,PcapCountUp,count,up bytes/packets per key,-,Performance Evaluation,25,Testbed,Packet Trace files,Result,26,Conclusion,Hadoop fits well into big Internet data analysis,Future Internet design and modeling,Hadoop,One size does not fit all!,Hadoop+DB,The,data center,is the computer,!,More data usually beats better algorithm,27,
展开阅读全文