人工神经网络.pptx

人工神经网络中国科学院自动化研究所吴高巍2016-11-29联结主义学派 又称仿生学派或生理学派认为人的思维基元是神经元，而不是符号处理过程认为人脑不同于电脑核心：智能的本质是联接机制。原理：神经网络及神经网络间的连接机制和学习算法麦卡洛可（McCulloch）皮茨（Pitts)什么是神经网络所谓的人工神经网络就是基于模仿生物大脑的结构和功能而构成的一种信息处理系统（计算机）。个体单元相互连接形成多种类型结构的图图循环、非循环有向、无向自底向上（Bottom-Up）AI起源于生物神经系统从结构模拟结构模拟到功能模拟功能模拟仿生仿生人工神经网络内容生物学启示多层神经网络Hopfield网络自组织网络生物学启示 神经元组成：细胞体，轴突，树突，突触 神经元之间通过突触两两相连。信息的传递发生在突触。突触记录了神经元间联系的强弱。只有达到一定的兴奋程度，神经元才向外界传输信息。生物神经元神经元神经元特性信息以预知的确定方向传递一个神经元的树突细胞体轴突突触另一个神经元树突时空整合性对不同时间通过同一突触传入的信息具有时间整合功能对同一时间通过不同突触传入的信息具有空间整合功能神经元工作状态兴奋状态，对输入信息整合后使细胞膜电位升高，当高于动作电位的阈值时，产生神经冲动，并由轴突输出。抑制状态，对输入信息整合后使细胞膜电位降低，当低于动作电位的阈值时，无神经冲动产生。结构的可塑性神经元之间的柔性连接：突触的信息传递特性是可变的学习记忆的基础神经元模型从生物学结构到数学模型人工神经元M-P模型x1x2xny12nInputOutputThresholdMcClloch and Pitts,A logical calculus of the ideas immanent in nervous activity,1943f:激活函数激活函数(Activation Function)g:组合函数组合函数(Combination Function)Weighted Sum Radial Distance组合函数 (e)(f)ThresholdLinearSaturating LinearLogistic SigmoidHyperbolic tangent SigmoidGaussian激活函数人工神经网络多个人工神经元按照特定的网络结构联接在一起，就构成了一个人工神经网络。神经网络的目标就是将输入转换成有意义的输出。生物系统中的学习自适应学习适应的目标是基于对环境信息的响应获得更好的状态在神经层面上，通过突触强度的改变实现学习消除某些突触，建立一些新的突触生物系统中的学习Hebb学习律神经元同时激活，突触强度增加异步激活，突触强度减弱学习律符合能量最小原则保持突触强度需要能量，所以在需要的地方保持，在不需要的地方不保持。ANN的学习规则能量最小 ENERGY MINIMIZATION对人工神经网络，需要确定合适的能量定义；可以使用数学上的优化技术来发现如何改变神经元间的联接权重。ENERGY=measure of task performance error两个主要问题结构 How to interconnect individual units?学习方法 How to automatically determine the connection weights or even structure of ANN?Solutions to these two problems leads to a concrete ANN!人工神经网络前馈结构(Feedforward Architecture)-without loops -static 反馈/循环结构(Feedback/Recurrent Architecture)-with loops -dynamic(non-linear dynamical systems)ANN结构General structures of feedforward networksGeneral structures of feedback networks通过神经网络所在环境的模拟过程，调整网络中的自由参数 Learning by data学习模型 Incremental vs.Batch两种类型 Supervised vs.UnsupervisedANN的学习方法若两端的神经元同时激活，增强联接权重Unsupervised Learning学习策略:Hebbrian Learning 最小化实际输出与期望输出之间的误差(Supervised)-Delta Rule(LMS Rule,Widrow-Hoff)-B-P LearningObjective:Solution:学习策略:Error Correction采用随机模式，跳出局部极小 -如果网络性能提高，新参数被接受.-否则，新参数依概率接受Local MinimumGlobal Minimum学习策略:Stochastic Learning“胜者为王”(Winner-take-all)UnsupervisedHow to compete?-Hard competition Only one neuron is activated -Soft competition Neurons neighboring the true winner are activated.学习策略:Competitive Learning重要的人工神经网络模型多层神经网络径向基网络Hopfield网络Boltzmann机自组织网络多层感知机(MLP)感知机实质上是一种神经元模型阈值激活函数Rosenblatt,1957感知机判别规则输入空间中样本是空间中的一个点权向量是一个超平面超平面一边对应 Y=1另一边对应 Y=-1单层感知机学习调整权值，减少训练集上的误差简单的权值更新规则:初始化对每一个训练样本:Classify with current weightsIf correct,no change!If wrong:adjust the weight vector30学习:Binary Perceptron初始化对每一个训练样本:Classify with current weightsIf correct(i.e.,y=y*),no change!If wrong:adjust the weight vector by adding or subtracting the feature vector.Subtract if y*is-1.多类判别情况If we have multiple classes:A weight vector for each class:Score(activation)of a class y:Prediction highest score wins学习:Multiclass Perceptron初始化依次处理每个样本Predict with current weightsIf correct,no change!If wrong:lower score of wrong answer,raise score of right answer感知机特性可分性:true if some parameters get the training set perfectly correctCan represent AND,OR,NOT,etc.,but not XOR收敛性:if the training is separable,perceptron will eventually converge(binary case)SeparableNon-Separable感知机存在的问题噪声（不可分情况）:if the data isnt separable,weights might thrash泛化性:finds a“barely”separating solution改进感知机线性可分情况Which of these linear separators is optimal?Support Vector MachinesMaximizing the margin:good according to intuition,theory,practiceOnly support vectors matter;other training examples are ignorable Support vector machines(SVMs)find the separator with max marginSVM优化学习问题描述训练数据目标：发现最好的权值，使得对每一个样本x的输出都符合类别标签样本xi的标签可等价于标签向量采用不同的激活函数平方损失：单层感知机单层感知机单层感知机单层感知机采用线性激活函数，权值向量具有解析解批处理模式一次性更新权重缺点：收敛慢增量模式逐样本更新权值随机近似，但速度快并能保证收敛多层感知机(MLP)层间神经元全连接MLPs表达能力3 layers:All continuous functions 4 layers:all functionsHow to learn the weights?waiting B-P algorithm until 1986B-P Network结构 A kind of multi-layer perceptron,in which the Sigmoid activation function is used.B-P 算法学习方法 -Input data was put forward from input layer to hidden layer,then to out layer -Error information was propagated backward from out layer to hidder layer,then to input layerRumelhart&Meclelland,Nature,1986B-P 算法Global Error Measuredesired outputgenerated outputsquared errorThe objective is to minimize the squared error,i.e.reach the Minimum Squared Error(MSE)B-P 算法Step1.Select a pattern from the training set and present it to the network.Step2.Compute activation of input,hidden and output neurons in that sequence.Step3.Compute the error over the output neurons by comparing the generated outputs with the desired outputs.Step4.Use the calculated error to update all weights in the network,such that a global error measure gets reduced.Step5.Repeat Step1 through Step4 until the global error falls below a predefined threshold.梯度下降方法Optimization method for finding out the weight vector leading to the MSE learning rategradientvector form:element form:权值更新规则For output layer:权值更新规则For output layer:权值更新规则For hidden layer权值更新规则For hidden layer应用:Handwritten digit recognition3-nearest-neighbor=2.4%error40030010 unit MLP=1.6%errorLeNet:768 192 30 10 unit MLP=0.9%errorCurrent best(SVMs)0.4%errorMLPs：讨论实际应用中Preprocessing is importantNormalize each dimension of data to-1,1 Adapting the learning ratet=1/tMLPs：讨论优点：很强的表达能力容易执行缺点：收敛速度慢过拟合（Over-fitting）局部极小采用Newton法加正则化项，约束权值的平滑性采用更少（但足够数量）的隐层神经元尝试不同的初始化增加扰动 Hopfield 网络反馈 结构可用加权无向图表示Dynamic System两种类型 Discrete(1982)and Continuous(science,1984),by HopfieldHopfield网络Combination function：Weighted SumActivation function：Threshold吸引子与稳定性How do we“program”the solutions of the problem into stable states(attractors)of the network?How do we ensure that the feedback system designed is stable?Lyapunovs modern stability theory allows us to investigate the stability problem by making use of a continuous scalar function of the state vector,called a Lyapunov(Energy)Function.Hopfield网络的能量函数With inputWithout inputHopfield 模型Hopfield证明了异步Hopfield网络是稳定的，其中权值定义为 Whatever be the initial state of the network,the energy decreases continuously with time until the system settles down into any local minimum of the energy surface.Hopfield 网络:联想记忆Hopfield网络的一个主要应用基于与数据部分相似的输入，可以回想起数据本身（attractor state）也称作内容寻址记忆(content-addressable memory).Stored PatternMemory Association虞台文虞台文,Feedback Networksand Associative MemoriesHopfield 网络:Associative MemoriesStored PatternMemory Association虞台文虞台文,Feedback Networksand Associative MemoriesHopfield网络的一个主要应用基于与数据部分相似的输入，可以回想起数据本身（attractor state）也称作内容寻址记忆(content-addressable memory).How to store patterns?=?How to store patterns?=?:Dimension of the stored pattern权值确定:外积(Outer Product)Vector form:Element form:Why?Satisfy the Hopfield modelAn example of Hopfield memory 虞台文虞台文,Feedback Networks and Associative Memories123422123422111111111111StableE=4E=0E=4Recall the first pattern(x1)123422111111111111StableE=4E=0E=4Recall the second pattern(x2)Hopfield 网络:组合优化(Combinatorial Optimization)Hopfield网络的另一个主要应用将优化目标函数转换成能量函数(energy function)网络的稳定状态是优化问题的解例:Solve Traveling Salesman Problem(TSP)Given n cities with distances dij,what is the shortest tour?Illustration of TSP Graph1234567891011Hopfield Network for TSP=?Hopfield Network for TSP=City matrix Constraint 1.Each row can have only one neuron“on”.2.Each column can have only one neuron“on”.3.For a n-city problem,n neurons will be on.Hopfield Network for TSP124351234512345TimeCityThe salesman reaches city 5 at time 3.Weight determination for TSP:Design Energy FunctionConstraint-1Constraint-2Constraint-3能量函数转换为2DHopfield网络形式Network is built!Hopfield网络迭代（TSP）The initial state generated randomly goes to the stable state(solution)with minimum energyA 4-city example 阮晓刚，阮晓刚，神经计算科神经计算科学学，2006自组织特征映射(SOFM)What is SOFM?Neural Network with Unsupervised LearningDimensionality reduction concomitant with preservation of topological information.Three principals -Self-reinforcing -Competition -CooperationStructure of SOFM竞争(Competition)Finding the best matching weight vector for the present input.Criterion for determining the winning neuron:Maximum Inner Product Minimum Euclidean Distance合作(Cooperation)Identify a neighborhood around the winning neuron.Topological neighborhood can be of different shapes such as Square,Hexagonal,or Gaussian.The width of the neighborhood is a function of time:as epochs of training elapse,the neighborhood shrinks.权值自适应(Adaptation)Weights of neurons within the winning cluster are updated.SOFM 算法Repeat Selection:Pick a sample Similarity Matching:Find the winning neuron Adaptation:Update synaptic vectors of ONLY the winning cluster.Update:Update the learning rate and neighborhoodUntil(there is no observable change in the map)小结人工神经网络是人工神经元组成的并行自适应网络，目标是对人类神经系统的某个功能进行抽象和建模。人工神经元基本元素 A set of connecting links A combination function An activation functionANN中的两个关键问题 Architecture and Learning Approach Solutions to these two problems leads to an ANN model两种 ANN 结构 Feedforward vs.Feedback(Recurrent)学习策略 Hebbrain,Error Correction,Stochastic,Winner-take-all人工神经网络发展历程谢谢！