©2010-2026 宁波自信网络信息技术有限公司 版权所有
客服电话：0574-28810668 投诉电话：18658249818
浙ICP备2021020529号-1 | 浙B2-20240490
关注我们 ：
1、Knowledgediscovery-)Inparticular:JiaweiHAN,SimonFraserUniversity,whoseforthcomingbookDatamining:conceptsandtechniqueshasinfluencedthewholetutorialRajeevRASTOGIandKyuseokSHIM,LucentBellLabsDanielA.KEIM,UniversityofHalleDanielSilver,CogNovaTechnologiesTheEDBT2000boardwhoacceptedourtutorialproposal,Kon
2、stanz,27-28.3.2000,EDBT2000tutorial-Intro,3,Tutorialgoals,IntroduceyoutomajoraspectsoftheKnowledgeDiscoveryProcess,andtheoryandapplicationsofDataMiningtechnologyProvideasystematizationtothemanymanyconceptsaroundthisarea,accordingthefollowinglinestheprocessthemethodsappliedtoparadigmaticcasesthesuppo
3、rtenvironmenttheresearchchallengesImportantissuesthatwillbenotcoveredinthistutorial:methods:timeseries,exceptiondetection,neuralnetssystems:parallelimplementations,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,4,TutorialOutline,IntroductionandbasicconceptsMotivations,applications,theKDDprocess,thetec
4、hniquesDeeperintoDMtechnologyDecisionTreesandFraudDetectionAssociationRulesandMarketBasketAnalysisClusteringandCustomerSegmentationTrendsintechnologyKnowledgeDiscoverySupportEnvironmentTools,LanguagesandSystemsResearchchallenges,,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,5,Introduction-moduleoutl
5、ine,MotivationsApplicationAreasKDDDecisionalContextKDDProcessArchitectureofaKDDsystemTheKDDstepsinshort,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,6,EvolutionofDatabaseTechnology:fromdatamanagementtodataanalysis,1960s:Datacollection,databasecreation,IMSandnetworkDBMS.1970s:Relationaldatamodel,rela
6、tionalDBMSimplementation.1980s:RDBMS,advanceddatamodels(extended-relational,OO,deductive,etc.)andapplication-orientedDBMS(spatial,scientific,engineering,etc.).1990s:Datamininganddatawarehousing,multimediadatabases,andWebtechnology.,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,7,Motivations“Necessity
7、istheMotherofInvention”,Dataexplosionproblem:Automateddatacollectiontools,maturedatabasetechnologyandinternetleadtotremendousamountsofdatastoredindatabases,datawarehousesandotherinformationrepositories.Wearedrowningininformation,butstarvingforknowledge!(JohnNaisbett)Datawarehousinganddatamining:On-l
8、ineanalyticalprocessingExtractionofinterestingknowledge(rules,regularities,patterns,constraints)fromdatainlargedatabases.,,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,8,Alsoreferredtoas:Datadredging,Dataharvesting,DataarcheologyAmultidisciplinaryfield:DatabaseStatisticsArtificialintelligenceMachine
9、learning,ExpertsystemsandKnowledgeAcquisitionVisualizationmethods,Arapidlyemergingfield,Arapidlyemergingfield,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,9,MotivationsforDM,AbundanceofbusinessandindustrydataCompetitivefocus-KnowledgeManagementInexpensive,powerfulcomputingenginesStrongtheoretical/ma
10、thematicalfoundationsmachinelearningstatisticalsummaryinformation(datacentraltendencyandvariation),MarketAnalysisandManagement,MarketAnalysis(2),Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,RiskAnalysis,Financeplanningandassetevaluation:cashflowanalysisandpredictioncontingentclaimanalysistoevaluatea
11、ssetscross-sectionalandtimeseriesanalysis(financial-ratio,trendanalysis,etc.)Resourceplanning:summarizeandcomparetheresourcesandspendingCompetition:monitorcompetitorsandmarketdirections(CI:competitiveintelligence).groupcustomersintoclassesandclass-basedpricingproceduressetpricingstrategyinahighlycom
12、petitivemarket,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,FraudDetection,Applications:widelyusedinhealthcare,retail,creditcardservices,telecommunications(phonecardfraud),etc.Approach:usehistoricaldatatobuildmodelsoffraudulentbehaviorandusedataminingtohelpidentifysimilarinstances.Examples:autoinsur
13、ance:detectagroupofpeoplewhostageaccidentstocollectoninsurancemoneylaundering:detectsuspiciousmoneytransactions(USTreasurysFinancialCrimesEnforcementNetwork)medicalinsurance:detectprofessionalpatientsandringofdoctorsandringofreferences,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,Moreexamples:Detect
14、inginappropriatemedicaltreatment:AustralianHealthInsuranceCommissionidentifiesthatinmanycasesblanketscreeningtestswererequested(saveAustralian$1m/yr).Detectingtelephonefraud:Telephonecallmodel:destinationofthecall,duration,timeofdayorweek.Analyzepatternsthatdeviatefromanexpectednorm.BritishTelecomid
15、entifieddiscretegroupsofcallerswithfrequentintra-groupcalls,especiallymobilephones,andbrokeamultimilliondollarfraud.Retail:Analystsestimatethat38%ofretailshrinkisduetodishonestemployees.,FraudDetection(2),Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,SportsIBMAdvancedScoutanalyzedNBAgamestatistics(sh
16、otsblocked,assists,andfouls)togaincompetitiveadvantageforNewYorkKnicksandMiamiHeat.AstronomyJPLandthePalomarObservatorydiscovered22quasarswiththehelpofdataminingInternetWebSurf-AidIBMSurf-AidappliesdataminingalgorithmstoWebaccesslogsformarket-relatedpagestodiscovercustomerpreferenceandbehaviorpages,
17、analyzingeffectivenessofWebmarketing,improvingWebsiteorganization,etc.WatchforthePRIVACYpitfall!,Otherapplications,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,20,Theselectionandprocessingofdatafor:theidentificationofnovel,accurate,andusefulpatterns,andthemodelingofreal-worldphenomena.Dataminingisam
18、ajorcomponentoftheKDDprocess-automateddiscoveryofpatternsandthedevelopmentofpredictiveandexplanatorymodels.,WhatisKDD?Aprocess!,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,21,TheKDDprocess,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,22,TheKDDProcess,CoreProblems&ApproachesProblems:identificationof
19、relevantdatarepresentationofdatasearchforvalidpatternormodelApproaches:top-downdeductionbyexpertinteractivevisualizationofdata/models*bottom-upinductionfromdata*,DataMining,,OLAP,,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,Learningtheapplicationdomain:relevantpriorknowledgeandgoalsofapplicationDat
20、aconsolidation:CreatingatargetdatasetSelectionandPreprocessingDatacleaning:(maytake60%ofeffort!)Datareductionandprojection:findusefulfeatures,dimensionality/variablereduction,invariantrepresentation.Choosingfunctionsofdataminingsummarization,classification,regression,association,clustering.Choosingt
21、heminingalgorithm(s)Datamining:searchforpatternsofinterestInterpretationandevaluation:analysisofresults.visualization,transformation,removingredundantpatterns,…Useofdiscoveredknowledge,ThestepsoftheKDDprocess,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,24,,IdentifyProblemorOpportunity,Measureeffect
22、ofAction,ActonKnowledge,,,,,,Knowledge,Results,Strategy,Problem,Thevirtuouscycle,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,25,Applications,operations,techniques,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,26,RolesintheKDDprocess,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,27,,,,,,,,,Increasingp
23、otentialtosupportbusinessdecisions,EndUser,BusinessAnalyst,DataAnalyst,DBA,MakingDecisions,DataPresentation,VisualizationTechniques,DataMining,InformationDiscovery,DataExploration,OLAP,MDA,StatisticalAnalysis,QueryingandReporting,DataWarehouses/DataMarts,DataSources,Paper,Files,InformationProviders,
24、DatabaseSystems,OLTP,,Dataminingandbusinessintelligence,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,28,GraphicalUserInterface,DataConsolidation,SelectionandPreprocessing,DataMining,InterpretationandEvaluation,,,,,Warehouse,Knowledge,,,,,,,,,,,,,DataSources,ArchitectureofaKDDsystem,Konstanz,27-28.3.
25、2000,EDBT2000tutorial-Intro,29,Abusinessintelligenceenvironment,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,30,TheKDDprocess,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,31,GarbageinGarbageoutThequalityofresultsrelatesdirectlytoqualityofthedata50%-70%ofKDDprocesseffortisspentondataconsolidationandp
26、reparationMajorjustificationforacorporatedatawarehouse,,Dataconsolidationandpreparation,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,32,Fromdatasourcestoconsolidateddatarepository,,,,,RDBMS,,,,,LegacyDBMS,,,,,FlatFiles,DataConsolidationandCleansing,,,,,Warehouse,,,,,Object/RelationDBMSMultidimension
27、alDBMSDeductiveDatabaseFlatfiles,,,,,External,,Dataconsolidation,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,33,DeterminepreliminarylistofattributesConsolidatedataintoworkingdatabaseInternalandExternalsourcesEliminateorestimatemissingvaluesRemoveoutliers(obviousexceptions)Determinepriorprobabilitie
28、sofcategoriesanddealwithvolumebias,Dataconsolidation,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,34,SelectionandPreprocessing,DataMining,InterpretationandEvaluation,DataConsolidation,,,,,,,,,,,,,,,,,,,,,,,,,Knowledge,,p(x)=0.02,,,,,,,,,,Warehouse,,,,,,,TheKDDprocess,Konstanz,27-28.3.2000,EDBT2000tu
29、torial-Intro,35,GenerateasetofexampleschoosesamplingmethodconsidersamplecomplexitydealwithvolumebiasissuesReduceattributedimensionalityremoveredundantand/orcorrelatingattributescombineattributes(sum,multiply,difference)Reduceattributevaluerangesgroupsymbolicdiscretevaluesquantizecontinuousnumericval
30、uesTransformdatade-correlateandnormalizevaluesmaptime-seriesdatatostaticrepresentationOLAPandvisualizationtoolsplaykeyrole,Dataselectionandpreprocessing,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,36,SelectionandPreprocessing,DataMining,InterpretationandEvaluation,DataConsolidation,,,,,,,,,,,,,,,,,
31、Knowledge,,p(x)=0.02,,,,,,,,,,Warehouse,,,,,,,TheKDDprocess,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,37,Dataminingtasksandmethods,AutomatedExploration/Discoverye.g..discoveringnewmarketsegmentsclusteringanalysisPrediction/Classificatione.g..forecastinggrosssalesgivencurrentfactorsregress
32、ion,neuralnetworks,geneticalgorithms,decisiontreesExplanation/Descriptione.g..characterizingcustomersbydemographicsandpurchasehistorydecisiontrees,associationrules,ifage>35andincome<$35kthen...,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,38,Clustering:partitioningasetofdataintoasetofclasses,calledc
33、lusters,whosememberssharesomeinterestingcommonproperties.Distance-basednumericalclusteringmetricgroupingofexamples(K-NN)graphicalvisualizationcanbeusedBayesianclusteringsearchforthenumberofclasseswhichresultinbestfitofaprobabilitydistributiontothedataAutoClass(NASA)oneofbestexamples,Automatedexplora
34、tionanddiscovery,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,39,LearningapredictivemodelClassificationofanewcase/sampleManymethods:ArtificialneuralnetworksInductivedecisiontreeandrulesystemsGeneticalgorithmsNearestneighborclusteringalgorithmsStatistical(parametric,andnon-parametric),Predictionandcl
35、assification,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,40,Theobjectiveoflearningistoachievegoodgeneralizationtonewunseencases.GeneralizationcanbedefinedasamathematicalinterpolationorregressionoverasetoftrainingpointsModelscanbevalidatedwithapreviouslyunseentestsetorusingcross-validationmethods,,,
36、f(x),x,Generalizationandregression,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,41,Classificationandprediction,Classifydatabasedonthevaluesofatargetattribute,e.g.,classifycountriesbasedonclimate,orclassifycarsbasedongasmileage.Useobtainedmodeltopredictsomeunknownormissingattributevaluesbased
37、onotherinformation.,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,42,Objective:DevelopageneralmodelorhypothesisfromspecificexamplesFunctionapproximation(curvefitting)Classification(conceptlearning,patternrecognition),A,B,,Summarizing:inductivemodeling=learning,Konstanz,27-28.3.2000,EDBT2000tutorial-I
38、ntro,43,Learnageneralizedhypothesis(model)fromselecteddataDescription/InterpretationofmodelprovidesnewknowledgeMethods:InductivedecisiontreeandrulesystemsAssociationrulesystemsLinkAnalysis…,Explanationanddescription,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,44,GenerateamodelofnormalactivityDeviat
39、ionfrommodelcausesalertMethods:ArtificialneuralnetworksInductivedecisiontreeandrulesystemsStatisticalmethodsVisualizationtools,Exception/deviationdetection,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,45,Outlierandexceptiondataanalysis,Time-seriesanalysis(trendanddeviation):Trendanddeviationanalysis
40、regression,sequentialpattern,similarsequences,trendanddeviation,e.g.,stockanalysis.Similarity-basedpattern-directedanalysisFullvs.partialperiodicityanalysisOtherpattern-directedorstatisticalanalysis,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,46,SelectionandPreprocessing,DataMining,Interpretationa
41、ndEvaluation,DataConsolidationandWarehousing,,,,,,,,,,,,,,,,,,,,,,,,,Knowledge,,p(x)=0.02,,,,,,,,,,Warehouse,,,,,,,TheKDDprocess,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,Adataminingsystem/querymaygeneratethousandsofpatterns,notallofthemareinteresting.Interestingnessmeasures:easilyunderstoodbyhum
42、ansvalidonnewortestdatawithsomedegreeofcertainty.potentiallyusefulnovel,orvalidatessomehypothesisthatauserseekstoconfirmObjectivevs.subjectiveinterestingnessmeasuresObjective:basedonstatisticsandstructuresofpatterns,e.g.,support,confidence,etc.Subjective:basedonuser’sbeliefsinthedata,e.g.,unexpected
43、ness,novelty,etc.,Areallthediscoveredpatterninteresting?,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,Findalltheinterestingpatterns:Completeness.Canadataminingsystemfindalltheinterestingpatterns?Searchforonlyinterestingpatterns:Optimization.Canadataminingsystemfindonlytheinterestingpatterns?Approach
44、esFirstgenerateallthepatternsandthenfilterouttheuninterestingones.Generateonlytheinterestingpatterns-miningqueryoptimization.,Completenessvs.optimization,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,49,EvaluationStatisticalvalidationandsignificancetestingQualitativereviewbyexpertsinthefieldPilotsurv
45、eystoevaluatemodelaccuracyInterpretationInductivetreeandrulemodelscanbereaddirectlyClusteringresultscanbegraphedandtabledCodecanbeautomaticallygeneratedbysomesystems(IDTs,Regressionmodels),Interpretationandevaluation,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,50,Visualizationtoolscanbeveryhelpfuls
46、ensitivityanalysis(I/Orelationship)histogramsofvaluedistributiontime-seriesplotsandanimationrequirestrainingandpractice,,,,Response,Velocity,Temp,,,,,,,,,,,,,,,,,,Interpretationandevaluation,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,1989IJCAIWorkshoponKDDKnowledgeDiscoveryinDatabases(G.Piatetsky-
47、ShapiroandW.Frawley,eds.,1991)1991-1994WorkshopsonKDDAdvancesinKnowledgeDiscoveryandDataMining(U.Fayyad,G.Piatetsky-Shapiro,P.Smyth,andR.Uthurusamy,eds.,1996)1995-1998AAAIInt.Conf.onKDDandDM(KDD’95-98)JournalofDataMiningandKnowledgeDiscovery(1997)1998ACMSIGKDD1999SIGKDD’99Conf.,Importantdatesofdatam
48、ining,Konstanz,27-28.3.2000,EDBT2000tutorial-Intro,52,References-general,P.AdriaansandD.Zantinge.DataMining.Addison-Wesley:Harlow,England,1996.M.S.Chen,J.Han,andP.S.Yu.Datamining:Anoverviewfromadatabaseperspective.IEEETrans.KnowledgeandDataEngineering,8:866-883,1996.U.M.Fayyad,G.Piatetsky-Shapiro,P.
49、Smyth,andR.Uthurusamy.AdvancesinKnowledgeDiscoveryandDataMining.AAAI/MITPress,1996.J.HanandM.Kamber.DataMining:ConceptsandTechniques.MorganKaufmann,2000.Toappear.T.ImielinskiandH.Mannila.Adatabaseperspectiveonknowledgediscovery.CommunicationsofACM,39:58-64,1996.G.Piatetsky-Shapiro,U.Fayyad,andP.Smith.Fromdataminingtoknowledgediscovery:Anoverview.InU.M.Fayyad,etal.(eds.),AdvancesinKnowledgeDiscoveryandDataMining,1-35.AAAI/MITPress,1996.G.Piatetsky-ShapiroandW.J.Frawley.KnowledgeDiscoveryinDatabases.AAAI/MITPress,1991.MichaelBerry&GordonLinoff.DataMiningTechniquesforMarketing,SalesandCustomerSu
©2010-2026 宁波自信网络信息技术有限公司 版权所有
客服电话：0574-28810668 投诉电话：18658249818
浙ICP备2021020529号-1 | 浙B2-20240490
关注我们 ：
