英语Chapter6.ppt

我们毕业啦其实是答辩的标题地方Taiyuan University of Technology软件工程专业英语计算机科学与技术学院 计算机软件学院SOFTWARE ENGINEERING ESSENTIALSCOMPETENCIESCOMPETENCIESAfteryouhavereadthischapter,youshouldbeableto:Explainwhatbigdatais.Explainthe4Vpropertiesofbigdata:volume,velocity,variety,andveracity.Distinguishthecategoriesofbigdata.Understandthebigvalueofbigdata.DescribeJimGraysFourthParadigm.Understandtheevolutionofbigdata.Discussthefourkindsofchallengesforbigdata.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院2Weareexperiencingunprecedentedgrowthintheamountofdataavailableinnearlyeveryarea,rangingfromthephysicalworld,includingbiology,astronomy,remote sensing etc.,tohumanactivities,includingsocialnetworks,Internet,health,finance,economics,transportation,etc.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院3These data are commonly called big data,and believed tocontaingreatvaluesandimplynewopportunities.Thischapterpresents an overview of big data,including its definition,properties,andcategories,aswellaschallengesitbringstous.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院46.1 BIG DATA AND ITS PROPERTIES6.2 CATEGORIES OF BIG DATA6.3 BIG VALUE FOR BIG DATA6.4 JIM GRAYS FOURTH PARADIGM6.5 EVOLUTION OF DATA MANAGEMENTSUMMARYCHPTER 6 BIG DATA6.6 BIG DATA CHALLENGES2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院56.1 BIG DATA AND ITS PROPERTIESBigdataisabroadtermfordatasetsso large or complex that traditionaldataprocessingapplicationsareinadequate.Mostprofessionalsintheindustryconsidermultipleterabytesorpetabytes to be the current big databenchmark.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院6Others,however,arehesitanttocommittoaspecificquantity,astherapidpaceoftechnologicaldevelopmentmayrendertodaysconceptofbigastomorrowsnorm.Inthemeantime,aconsensusisreachedonthepropertiesofbigdata.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院712WHAT IS BIG DATA?FOUR DIMENSIONS OF BIG DATA2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院8Inadynamicworld,organizationshavebeguntomoreheavilyrelyoninsights derived from their data to uncover new facts andopportunitiesforsciencediscoveryorrevenuegrowth.Intheprocessofdiscoveringanddeterminingtheseinsights,largecomplexdatasetsaregeneratedthatthenmustbemanaged,analyzedandmanipulatedbyskilledprofessionals.Thecompilationofthislargecollectionofdatais collectively known as big data.The definition of big data fromWikipediafollows.”Bigdatausuallyincludesdatasetswithsizesbeyondtheabilityofcommonly used software tools to capture,curate,manage,andprocess data within a tolerable elapsed time.Big data“size”is aconstantly moving target,as of 2012 ranging from a few dozenterabytestomanypetabytesofdata.Bigdataisasetoftechniquesand technologies that require new forms of integration to uncoverlargehiddenvaluesfromlargedatasetsthatarediverse,complex,andofamassivescale.”WHAT IS BIG DATA?2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院9In a 2001 research report and related lectures,METAGroup(now Gartner)analyst Doug Laney defined datagrowth challenges and opportunities as being three-dimensional,i.e.increasing volume(amount of data),velocity(speedofdatainandout),andvariety(rangeofdata types and sources).In 2012,Gartner updated itsdefinition as follows:Big data is high volume,highvelocity,and/or high variety information assets thatrequire new forms of processing to enable enhanceddecisionmaking,insightdiscoveryandprocessoptimization.4Additionally,anewVVeracityisaddedbysomeorganizations.FOUR DIMENSIONS OF BIG DATA2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院10VolumeThesizeofthedatadeterminesthevalueandpotentialofthedataunder consideration,andwhetheritcanactuallybeconsidered big data ornot.Big data doesntsample.It just observesandtrackswhathappens.VelocityVelocityisanindicationofhowquicklythedatacanbe made available foranalysis.Bigdataisoftenavailableinreal-time.VarietyVarietyreferencesthedifferenttypesofstructuredand unstructured data thatorganizationscancollect,such as transaction data,video,audio,text and logfiles.Bigdatadrawsfromalltypesofdata.VeracityVeracityisanindicationofdata integrity and theabilityforanorganizationto trust the data and beabletoconfidently useittomakecrucialdecisions.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院11Different kind of data shows different properties.For example,social network has more requirements on data volume and velocity rather than variety and veracity.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院126.2 CATEGORIES OF BIG DATA2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院1312DATA FROM THE PHYSICAL WORLDDATA FROM HUMAN ACTIVITIESTheamountofdatainourworldhasbeenexplodingwithourabilitytoacquiredataenhancing.Bigdataisubiquitousandcanbepartitionedintotwocategories:2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院14Scientific experiments and observations produce massivescientificdatasetsaboutthephysicalworld.Sensornetworksconsisting of large number of cheap sensors have beenwidely used toobtainednaturaldata.OceanObservatoriesInitiativeuseselectro-opticallycabledobservingsystemstomeasure ocean activities in the northeast Pacific Ocean.Panoramic Survey Telescope and Rapid Response System(Pan-STARRS)produce 2.5PB data each year.Over 5000genomeprojectsin2010producedseveralEBgenomicdata.Remotesensingproducesevenlargernaturaldatasets.DATA FROM THE PHYSICAL WORLD2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院152024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院16Big data sets produced by human activities,such as socialnetworks,Internet,health,finance,economics,transportation,etc.,attracted much attention in recent years.To gain a betterunderstanding on how many data is being generated,considerthefollowingnoteworthyfacts:Facebookcurrentlyholdsmorethan45billionphotosinitsuserdatabase,anumberthatisgrowingdaily.AccordingtoIBM,userscreate2.5quintillionbytesofdataeveryday.In practical terms,this means that 90%of the data in theworldtodayhasbeencreatedinthelasttwoyearsalone.According to FICO,the credit card fraud detection systemcurrentlyinplacehelpsprotectovertwobillionaccountsallovertheglobe.Walmart controls more than 1 million customer transactionsevery hour,which are then transferred into a database workingwithover2.5petabytesofinformation.DATA FROM HUMAN ACTIVITIES2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院176.3 BIG VALUE FOR BIG DATAGinniRometty,PresidentandChiefExecutiveOfficerofIBM,hassaidthat“Dataistheworldsgreatnewnaturalresource.Whatsteampowerwastothe 18th century,electromagnetismto the 19th and fossil fuels to the20th datawillbetothe21st.”Likeother essential factors of productionsuch as hard assets and humancapital,itisincreasinglythecasethatmuch of modern economic activity,innovation,andgrowthsimplycouldnttakeplacewithoutdata.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院18While digital data might once havebeenconcernedbyonlyafewdatageeks,bigdataisnowrelevantforleadersacrosseverysector,andconsumersofproductsandservicestostandtobenefitfromitsapplication.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院19The combination of deepening investments in big data andmanagerialinnovationtocreatecompetitiveadvantageandboostproductivityisverysimilartothewayITdevelopedfromthe1970sonward.TheexperienceofITstronglysuggeststhatwecouldbeonthecuspofanewwaveofproductivitygrowthenabledbytheuseofbigdata.Therearemanywaysthatbigdatacanbeusedtocreatevalue.Large companies across the globe have scored earlysuccessesintheiruseofbigdata.Therearenotableexamplesofcompaniesaroundtheglobethatarewell-knownfortheirextensiveandeffectiveuseofdata.Forinstance,Tescos loyalty program generates a tremendous amount ofcustomer data that the company mines to inform decisionsfrompromotionstostrategicsegmentationofcustomers.Amazon uses customer data to power its recommendationengine“you may also like”based on a type of predictivemodelingtechniquecalledcollaborativefiltering.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院20Bymakingsupplyanddemandsignalsvisiblebetweenretailstoresandsuppliers,Wal-Martwasanearlyadopterofvendor-managedinventorytooptimizethesupplychain.Harrahs,theUShotelsandcasinosgroup,compilesdetailedholistic profiles of its customers and uses them to tailormarketinginawaythathasincreasedcustomerloyalty.Progressive Insurance and Capital One are both known forconductingexperimentstosegmenttheircustomerssystematically and effectively and to tailor product offersaccordingly.Smart,aleadingwirelessplayerinthePhilippines,analyzesitspenetration,retailercoverage,andaveragerevenueperuseratthecityortownlevelinordertofocusonthemicromarketswiththemostpotential.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院21McKinsey&Companyobservedhowbigdatacreatedvaluesafterin-depthresearchontheU.S.healthcare,theEUpublicsectoradministration,theU.S.retail,theglobalmanufacturing,andtheglobalpersonallocationdata.Throughresearchonthefive core industries that represent the global economy,theMcKinseyreportpointedoutthatbigdatamaygiveafullplaytothe economic function,improve the productivity andcompetitivenessofenterprisesandpublicsectors,andcreatehugebenefitsforconsumers.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院22McKinseysummarizedthevaluesthatbigdatacouldcreate:ifbigdatacouldbecreativelyandeffectivelyutilizedtoimproveefficiency and quality,the potential value of the U.S.medicalindustrygainedthroughdatamaysurpassUSD300billion,thusreducingtheU.S.healthcareexpenditurebyover8%;retailersthatfullyutilizebigdatamayimprovetheirprofitbymorethan60%;bigdatamayalsobeutilizedtoimprovetheefficiencyofgovernmentoperations,suchthatthedevelopedeconomiesinEurope could save over EUR 100 billion(which excludes theeffectofreducedfrauds,errors,andtaxdifference).2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院23TheMcKinseyreportisregardedasprospectiveandpredictive,while the following facts may validate the values of big data.During the 2009 flu pandemic,Google obtained timelyinformation by analyzing big data,which even provided morevaluable information than that provided by disease preventioncenters.Nearlyallcountriesrequiredhospitalsinformagencies,such as disease prevention centers,of new type of influenzacases.However,patientsusuallydidnotseedoctorsimmediatelywhen they got infected.It also took some time to sendinformationfromhospitalstodiseasepreventioncenters,andfordisease prevention centers to analyze and summarize suchinformation.Therefore,whenthepublicisawareofthepandemicofanewtypeofinfluenza,thediseasemayhavealreadyspreadforonetotwoweekswithaserioushystereticnature.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院24Googlefoundthatduringthespreadingofinfluenza,entriesfrequentlysoughtatitssearchengineswouldbedifferentfromthoseatordinarytimes,andtheusagefrequenciesoftheentrieswere correlated to the influenza spreading in both time andlocation.Googlefound45searchentrygroupsthatwerecloselyrelevanttotheoutbreakofinfluenzaandincorporatedtheminspecificmathematicmodelstoforecastthespreadingofinfluenzaandeventopredictplaceswhereinfluenzawillspreadfrom.TherelatedresearchresultshavebeenpublishedinNature.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院25In 2008,Microsoft purchased Farecast,a sci-tech venturecompany in the U.S.Forecast has an airline ticket forecastingsystem that predicts the trends and rising/dropping ranges ofairlineticketprices.ThesystemhasbeenincorporatedintotheBingsearchengineofMicrosoft.By2012,thesystemhassavednearly USD 50 per ticket per passenger,with the forecastaccuracyashighas75%.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院262024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院276.4 JIM GRAYS FOURTH PARADIGM2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院28FirstExperimental ScienceOriginally,therewasjustexperimentalsciencedescribingnaturalphenomena.SecondTheoretical ScienceThen in the last fewhundredyears,therewastheoreticalscience,withKeplersLaws,NewtonsLawsofMotion,Maxwellsequations,andsoon.ThirdComputational Science In the last few decades,for manyproblems,the theoretical modelsgrewtoocomplicatedtosolveanalytically,andpeoplehadtostartsimulating.These simulations havecarriedusthroughmuchofthelasthalf of the last millennium.At thispoint,thesesimulationsaregeneratingawholelotofdata,alongwithahugeincreaseindatafromtheexperimental sciences.People nowdonotactuallylookthroughtelescopes.Instead,theyare“looking”throughlarge-scale,complex instruments which relaydata to datacenters,and only thendo they look at the information ontheircomputers.FourthData-intensive Science The world of science has changed,andthereisnoquestionaboutthis.Thenewmodelisforthedatatobecapturedbyinstrumentsorgenerated by simulations beforebeingprocessedbysoftwareandfortheresultinginformationorknowledgetobestoredincomputers.Scientists only get tolook attheir datafairly late in thispipeline.Thetechniquesandtechnologiesforsuchdata-intensivescience are so different that it isworth distinguishing data-intensivesciencefromcomputationalscienceas a new,fourth paradigm forscientificexploration.Jim Gray believes that scientific discovery has experienced fourparadigms.2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院296.5 EVOLUTION OF DATA MANAGEMENT2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院30People proposed“share nothing”a parallel databasesystem,to meet the demand of the increasing datavolume.Thesharenothingsystemarchitectureisbasedonthe use of cluster and every machine has its ownprocessor,storage,anddisk.Teradatasystemwasthefirstsuccessful commercial parallel database system.Suchdatabasebecameverypopularlately.In the 1980sTheconceptof“databasemachine”emerged,whichisatechnologyspeciallyusedforstoringandanalyzingdata.With the increase of data volume,the storage andprocessing capacity of a single mainframe computersystemhasbecomeinadequate.In late 1970s2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院31The advantage of the parallel database was widelyrecognizedinthedatabasefield.In late 1990sOn June 2,1986,a milestone event occurred,whenTeradatadeliveredthefirstparalleldatabasesystemwithastoragecapacityof1TBtoKmarttohelpthelarge-scaleretail company in North America to expand its datawarehouse.On June 2,19862024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院32Anothermilestoneeventoccurred,whenEMC/IDCpublishedaresearch report titled Extracting Values from Chaos,whichintroducedtheconceptandpotentialofbigdataforthefirsttime.This research report aroused great interest in bothindustryandacademiaonbigdata.In June 2011However,many challenges on big data arose.Contentsgenerated by users,sensors,and other ubiquitous datasourcesdrivetheoverwhelmingdataflows,whichrequiredafundamentalchangeonthecomputingarchitectureandlarge-scale data processing mechanism.Jim Gray,a pioneer ofdatabase software,called such transformation“The FourthParadigm”.He thought the only way to cope with such aparadigm was to develop a new generation of computingtoolstomanage,visualize,andanalyzemassivedata.In January 20072024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院33Over the past few years,nearly all majorcompanies,including EMC,Oracle,IBM,Microsoft,Google,Amazon,andFacebook,etc.,havestartedtheirbigdataprojects.TakingIBMasanexample,since2005,IBMhasinvestedUSD16billionon30acquisitionsrelatedtobigdata.Industry2024/11/10 周日太原理工大学.计算机科学与技术学院.计算机软件学院34Inacademia,bigdatawasalsounderthespotlight.In2008,Naturepublishedthebigdataspecialissue.In2011,Sciencealsolaunchedaspecialissueonthekeytechnologiesof“dataprocessing”inbigdata.In2012,EuropeanResearchConso