经典深度学习.ppt

单击此处编辑母版标题样式,单击此处编辑母版文本样式,第二级,第三级,第四级,第五级,*,RANLP 2015,Hissar,Bulgaria,Deep Learning in Industry Data Analytics,Junlan,Feng,China Mobile Research,1,人工智能的起点,:,达特茅斯会议,1919-2001,1927-2011,1927-2016,1916-2011,Nathaniel Rochester,人工智能的阶段,1950s 1980s 2000s Future,自动计算机,如何为计算机编程使其能够使用语言,神经网络,计算规模理论,自我提升,抽象随机性与创造性,基于规则的专家系统,通用智能,1,2,3,人工智能的当前技术,:,存在的问题,依赖大量的标注数据,“窄人工智能”训练完成特定的任务,不够稳定，安全,不具备解释能力，模型不透明,人工智能的当前状态,:,应用,人工智能成为热点的原因,:,深度学习,强化学习,大规模的,复杂的,流式的数据,概要,解析白宫人工智能研发战略计划,3.,深度学习及最新进展,2.,解析十家技术公司的的人工智能战略,4.,强化学习及最新进展,5.,深度学习在企业数据分析中的应用,美国人工智能战略规划,美国人工智能研发战略规划,策略,-I:,在人工智能研究领域做长期研发投资,目标：,.,确保美国的世界领导地位,优先投资下一代人工智能技术,推动以数据为中心的知识发现技术,高效的数据清洁技术以，确保用于训练系统的数据的可信性,(varascty),和正确性,(appropriateness),综合考虑 数据，元数据，以及人的反馈或知识,异构数据，多模态数据分析和挖掘，离散数据，连续数据，时间域数据，空间域数据，时空数据，图数据,小数据挖掘，强调小概率事件的重要性,数据和知识尤其领域知识库的融合使用,策略,-I:,在人工智能研究领域做长期研发投资,目标：,.,确保美国的世界领导地位,.,优先投资下一代人工智能技术,推动以数据为中心的知识发现技术,2.,增强系统的感知能力,硬件或算法能提升系统感知能力的稳健性和可靠性,提升在复杂动态环境中对物体的检测，分类，辨别，识别能力,提升传感器或算法对人的感知，以便系统更好地跟人的合作,计算和传播感知系统的不确定性给系统以便更好的判断,策略,-I:,在人工智能研究领域做长期研发投资,目标：,.,确保美国的世界领导地位,.,优先投资下一代人工智能技术,推动以数据为中心的知识发现技术,2.,增强系统的感知能力,当前硬件环境和算法框架下,AI,的理论上限,学习能力,语言能力,感知能力,推理能力,创造力,计划，规划能力,3.,理论能力和上限,策略,-I:,在人工智能研究领域做长期研发投资,目标：,.,确保美国的世界领导地位,.,优先投资下一代人工智能技术,推动以数据为中心的知识发现技术,2.,增强系统的感知能力,目前的,AI,系统均为窄人工智能，,“Narrow AI”,而不是“,General AI”,GAI:,灵活，多任务，有自由意志，在多认知任务中的通用能力（学习能力，语言能力，感知能力，推理能力，创造力，计划，规划能力,迁移学习,3.,理论能力和上限,4.,通用,AI,策略,-I:,在人工智能研究领域做长期研发投资,目标：,.,确保美国的世界领导地位,.,优先投资下一代人工智能技术,推动以数据为中心的知识发现技术,2.,增强系统的感知能力,多,AI,系统的协同,分布式计划和控制技术,3.,理论能力和上限,4.,通用,AI,5.,规模化,AI系统,策略,-I:,在人工智能研究领域做长期研发投资,目标：,.,确保美国的世界领导地位,.,优先投资下一代人工智能技术,推动以数据为中心的知识发现技术,2.,增强系统的感知能力,AI,系统的自我解释能力,目前,AI,系统的学习方法：大数据，黑盒,人的学习方法：小数据，接受正规的指导规则以及各种暗示,仿人的,AI,系统，可以做智能助理，智能辅导,3.,理论能力和上限,4.,通用,AI,5.,规模化,AI,系统,6.,仿人类的,AI,技术,策略,-I:,在人工智能研究领域做长期研发投资,目标：,.,确保美国的世界领导地位,.,优先投资下一代人工智能技术,推动以数据为中心的知识发现技术,2.,增强系统的感知能力,提升机器人的感知能力，更智能的同复杂的物理世界交互,3.,理论能力和上限,4.,通用,AI,5.,规模化,AI,系统,6.,仿人类的,AI,技术,7.,研发实用，可靠，易用的机器人,策略,-I:,在人工智能研究领域做长期研发投资,目标：,.,确保美国的世界领导地位,.,优先投资下一代人工智能技术,推动以数据为中心的知识发现技术,2.,增强系统的感知能力,提升机器人的感知能力，更智能的同复杂的物理世界交互,GPU,：提升的内存，输入输出，时钟,速度，并行能力，节能,“类神经元”处理器,处理基于流式，动态数据,利用,AI,技术提升硬件能力：高性能计算，优化能源消耗，增强计算性能，自我智能配置，优化数据在多核处理器和内存直接移动,3.,理论能力和上限,4.,通用,AI,5.,规模化,AI,系统,6.,仿人类的,AI,技术,7.,研发实用，可靠，易用的机器人,8.AI,和硬件的相互推动,策略,-II,：开发有效的人机合作方法,.,不是替代人，而是跟人合作，强调人和,AI,系统之间的互补作用,辅助人类的人工智能技术,AI,系统的设计很多是为人所用,复制人类计算，决策，认知,策略,-II,：开发有效的人机合作方法,.,不是替代人，而是跟人合作，强调人和,AI,系统之间的互补作用,辅助人类的人工智能技术,2.,开发增强人类的技术,稳态设备,穿戴设备,植入设备,辅助数据理解,策略II,：开发有效的人机合作方法,.,不是替代人，而是跟人合作，强调人和,AI,系统之间的互补作用,辅助人类的人工智能技术,2.,开发增强人类的技术,数据和信息的可视化，以人可以理解的方式展现,提升人和系统通信的效率,3.,可视化，,AI-,人之间的友好界面,策略,-II,：开发有效的人机合作方法,.,不是替代人，而是跟人合作，强调人和,AI,系统之间的互补作用,辅助人类的人工智能技术,2.,开发增强人类的技术,已成功：安静环境下的流畅的语音识,未解决的：噪声环境下的识别，远场语音识别，口音，儿童语音识别，受损语音识别，语言理解，对话能力,3.,可视化，,AI-,人之间的友好界面,4.,研发更有效的语言处理系统,策略,III,：理解并重点关注人工智能可能带来的伦理，法律，社会方面的影响,研究人工智能技术可能带来的伦理，法律，社会方面的影响,期待其符合人的类规范,AI,系统从设计上需要符合人类的道德标准：公平，正义，透明，责任感,策略,III,：理解并重点关注人工智能可能带来的伦理，法律，社会方面的影响,研究人工智能技术可能带来的伦理，法律，社会方面的影响,期待其符合人的类规范,AI,系统从设计上需要符合人类的道德标准：公平，正义，透明，责任感,2.,构建符合道德的,AI,技术,如何将道德量化，由模糊变为精确的系统和算法设计,道德通常是模糊的，随文化，宗教和信仰而不同,策略,III理解并重点关注人工智能可能带来的伦理，法律，社会方面的影响,研究人工智能技术可能带来的伦理，法律，社会方面的影响,期待其符合人的类规范,AI,系统从设计上需要符合人类的道德标准：公平，正义，透明，责任感,2.,构建符合道德的,AI,技术,两层架构：由一层专门负责道德建设,道德标准植入每一个工程,AI,步骤,3.,符合道德标准的,AI,技术的实现框架,策略,-IV:,确保人工智能系统的自身和对周围环境安全性,在人工智能系统广泛使用之前，必须确保系统的安全性,研究创造稳定，可依靠，可信赖，可理解，可控制的人工智能系统所面临的挑战及解决办法,提升,AI,系统 的可解释性和透明度,2.,建立信任,3.,增强,verification,和,validation,4.,自我监控，自我诊断，自我修正,5.,意外处理能力，防攻击能力,策略,-V:,发展人工智能技术所需的共享的数据集和共享的模拟环境,一件重要的公益事业，同时要充分尊重企业和个人在数据中的权利和利益,鼓励开源,策略,-VI:,评价和评测人工智能技术的标准,开发恰当的评级策略和方法,策略,-VII:,更好的理解国家在人工智能研发方面的人力需求,保证足够的人才资源,大数据和人工智能,数据是人工智能的来源,大数据并行计算，流计算等技术是人工智能能实用化的保障,人工智能是大数据，尤其复杂数据分析的主要方法,.Top 10,家技术公司的布局Google:AI-First Strategy,Google,化,4,亿美金购买英国伦敦大学,人工智能,创业公司：,DeepMind,AlphaGo,GNC,WaveNet,Q-Learning,2011,年成立,1.,语音识别，合成；,2.,机器翻译；,3.,无人驾驶车,.4.,谷歌眼镜,.5.Google Now.6.,收购,Api.ui,Facebook,共享深度学习开源代码：,Torch,Facbook M,数字助理,研究和应用：,FAIR&AML,Apple AI,Apple Siri,Apple bought Emotient and Vocal IQ,?,Partnership on AI,It will“conduct research,recommend best practices,and publish research under an open license in areas such as ethics,fairness and inclusivity;transparency,privacy,and interoperability;collaboration between people and AI systems;and the trustworthiness,reliability and robustness of the technology”,2016,年,9,月,29,日,Elon Musk:OpenAI,Paypal,Telsla,SpaceX,SolarCity,四家公司,CEO,，投资十个亿美金成立,OpenAI,Microsoft,小冰,小娜,API,开放,CNTK,微软研究院,IBM,语音,文本,图片,视频,Watson,计算机,百度,国内技术巨头,腾讯，阿里，讯飞在人工智能领域投入巨大,5.,深度学习在企业数据分析中的案例,An example:AI in Data Analytics with Deep Learning -,客户情感分析,INTRODUCTION,EMOTION RECOGNITION IN TEXT,EMOTION RECOGNITION IN SPEECH,EMOTION RECOGNITION IN CONVERSATIONS,INDUSTRIAL APPLICATION,Datasets,Features,Methods,Introduction:Interchangeable Terms,42,Opinion Mining,Sentimental,Analysis,Emotion,Recognition,Polarity,Detection,Review,Mining,Introduction:What emotions are?,43,Introduction:Problem Definition,Positive and Negative;opinions,Target of the opinions;Entity,Related set of components;aspect,Related attributes;aspect,Opinion holder;opinion source,We will only,focus on document level sentiment,Opinion Mining,RANLP 2015,Hissar,Bulgaria,I,ntroduction,:,Text,Examples,6th September 2015,45,a,thriller,without a lot of,thrills,An edgy thriller,that,delivers,a surprising punch,A,flawed,but engrossing,thriller,Its,unlikely,well,see,a,better,thriller,this year,An erotic thriller thats,neither,too erotic nor very thrilling,either,E,motions are expressed artistically with help of,Negation Conjunction Words Sentimental Words,e.g.,RANLP 2015,Hissar,Bulgaria,Introduction,:,Text Examples,DSE,:explicitly express an opinion holders attitude,ESE:indirectly express the attitude of the,writer,6th September 2015,46,Emotions are expressed explicitly and indirectly.,RANLP 2015,Hissar,Bulgaria,Introduction,:,Text Examples,6th September 2015,47,Emotions are expressed,language,that is often obscured by sarcasm,ambiguity,and plays on words,all of which could be very misleading for both humans and computers,A SHARP TONGUE DOES NOT MEAN YOU HAVE A KEEN,MIND,I,DONT KNOW WHAT MAKES YOU SO DUMB BUT IT REALLY,WORKS,PLEASE,KEEP TALKING,.SO GREAT.,I ALWAYS YAWN WHEN I AM INTERESTED.,RANLP 2015,Hissar,Bulgaria,Introduction,:,Speech Conversation Examples,6th September 2015,48,RANLP 2015,Hissar,Bulgaria,Introduction,:,Conversation Examples,6th September 2015,49,RANLP 2015,Hissar,Bulgaria,Typical Approach,:,A Classification Task,6th September 2015,50,A Document,Features:,Ngrams(Uni,bigrams),POS Tags,Term Frequency,Syntactic Dependency,Negation Tags,SVM,Maxent,Nave Bayes,CRF,Random Forest,Pos,Neu,Neg,Supervised Learning,Pos-Tag Patterns+Dictionary+Mutual Info,Rules,Unsupervised Learning,RANLP 2015,Hissar,Bulgaria,Typical Approach,:,A Classification Task,6th September 2015,51,Features:,Prosodic features,:pitch,energy,formants,etc.,Voice,quality features,:harsh,tense,breathy,etc.,Spectral,features,:LPC,MFCC,LPCC,etc.,Teager,Energy Operator(TEO)-based features,:TEO-,FM,-var,TEO-Auto-Env,etc,SVM,GMM,HMM,DBN,KNN,LDA,CART,Pos,Neu,Neg,Supervised Learning,Challenges Remain,TEXT-BASED:,Capture the compositional effects with higher accuracy,Negating Positive sentences,Negating Negative sentences,Conjunction:,SPEECH-BASED:,Effective features unknown.Emotional speech segments tend to be transcribed with lower ASR accuracy,Overview,INTRODUCTION,EMOTION RECOGNITION IN TEXT,Word Embedding for Sentiment Analysis,CNN for Sentiment Classification,RNN,LSTM for sentiment Classification,Prior Knowledge+CNN/LSTM,Parsing+RNN,EMOTION RECOGNITION IN SPEECH,EMOTION RECOGNITION IN CONVERSATIONS,INDUSTRIAL APPLICATION,How deep learning can change the game?,RANLP 2015,Hissar,Bulgaria,6th September 2015,54,Emotion Classification with Deep learning approaches,RANLP 2015,Hissar,Bulgaria,1.Word Embedding as Features,6th September 2015,55,Representation of text is very important for performance,of many,real-world,applications including emotion recognition:,Local representations:,N-grams,Bag-of-words,1-of-N,coding,Continuous Representations:,Latent Semantic Analysis,Latent,Dirichlet Allocation,Distributed Representations:word embedding,Tomas Mikolov,“Learning,Representations of Text using,Neural Networks”,NIPs Deep learning Workshop 2013,(Bengio et al.,2006;Collobert c),RANLP 2015,Hissar,Bulgaria,1.Word Embedding as Features,6th September 2015,56,Representation of text is very important for performance,of many,real-world,applications including emotion recognition:,Local representations:N-grams,Bag-of-words,1-of-N,coding,Continuous Representations:,Latent Semantic Analysis,Latent,Dirichlet Allocation,Distributed Representations:word embedding,Tomas Mikolov,“Learning,Representations of Text using,Neural Networks”,NIPs Deep learning Workshop 2013,RANLP 2015,Hissar,Bulgaria,Word Embedding,6th September 2015,57,Skip-gram Arch,CBOW,The hidden layer vector is the word-embedding vector for w(t),Word Embedding for Sentiment Detection,It has been widely accepted as standard features for,NLP applications including,sentiment analysis since 2013 Mikolov 2013,The word vector space implicitly encodes many linguistic regularities among words:semantic and syntactic,Example:Google Pre-trained word vectors with 1000Billion words,Does it encode polarity similarities?,great0.729151,bad0.719005,terrific0.688912,decent0.683735,nice0.683609,excellent0.644293,fantastic0.640778,better0.612073,solid0.580604,lousy0.576420,wonderful0.572612,terrible0.560204,Good0.558616,Top Relevant Words to“good”,Mostly Yes,but it doesnt separate antonyms well,RANLP 2015,Hissar,Bulgaria,Learning Sentiment-Specific Word Embedding,6th September 2015,59,Tang,et al,“Learning Sentiment Specific Word Embedding for Twitter Sentiment Classification”,ACL 2014,RANLP 2015,Hissar,Bulgaria,Learning Sentiment-Specific Word Embedding,6th September 2015,60,Tang,et al,“Learning Sentiment Specific Word Embedding for Twitter Sentiment Classification”,ACL 2014,In Spirit,it is similar to multi-task learning.It learns the same way as the regular word-embedding with loss function considering both semantic context and sentiment distance to the twitter emotion symbols.,10,million tweets,selected by,positive and negative emoticons as,training data,The Twitter sentiment classification track of SemEval 2013,Learning Sentiment-Specific Word Embedding,Tang,et al,“Learning Sentiment Specific Word Embedding for Twitter Sentiment Classification”,ACL 2014,Paragraph Vectors,Le and Mikolov,“Distributional Representations of Sentences and Documents,ICML 2014,Paragraph vectors are distributional vect,or representation for pieces of text,such as sentences or paragraphs,The paragraph vectors are also asked to contribute to the prediction task of the next word given many contexts sampled from the paragraph,.,Each paragraph corresponds to one column in D,It acts as a memory remembering what is missing from the current context,about the topic of the paragraph,Paragraph Vectors Best Results on MR Data Set,Le and Mikolov,“Distributional Representations of Sentences and Documents,ICML 2014,Overview,INTRODUCTION,EMOTION RECOGNITION IN TEXT,Word Embedding for Sentiment Analysis,CNN for Sentiment Classification,RNN,LSTM for sentiment Classification,Prior Knowledge+CNN/LSTM,Dataset Collection,EMOTION RECOGNITION IN SPEECH,EMOTION RECOGNITION IN CONVERSATIONS,INDUSTRIAL APPLICATION,CNN for Sentiment Classification,Ref:Yoon Kim.Convolutional Neural Networks for Sentence Classification.EMNLP,2014.,CNN for Sentiment Classification,Ref:Yoon Kim.Convolutional Neural Networks for Sentence Classification.EMNLP,2014.,A simple CNN with One Layer of convolution on top of word vectors.Motivated by CNN has been successful on many other NLP tasks,Input Layer:Word,v,ectors are from pre-trained Google-News word2vector,Conv Layer:Window size:3 words,4 words,5 words.Each with 100 feature map.300 features in the penultimate layer,Pooling Layer:Max Over time Pooling at the,Output layer:Fully connected softmax layer,output distribution over labels,Regularization:Drop-out on the penultimate layer with a constrain on the l2 norms of the weight vectors,Fine-train embedding vectors during training,Common Datasets,CNN for Sentiment Classification-Results,CNN-rand:Randomly initialize all word embeddings,CNN-static:word2vec,keep the embeddings fixed,CNN-nonstatic:Fine-tuning embedding vectors,CNN for Sentiment Classification-Results,Why it is successful?,Multiple filters and multiple feature maps,Emotions are expressed in segments,instead of the spanning over the whole sentence,Use pre-trained word2vec vectors as input features.,Embedding word vectors are further improved for non-static training.Antonyms are further separated after training.,Resources for This work,Source Code:https,:/in Tensorflow:,Experiments:,https:/arxiv.org/pdf/1510.03820v4.,pdf,Dynamic CNN for Sentiment,Kalchbrenner et al,“A Convolutional Neural Network for Modeling Sentences”,ACL 2014,Hyper Parameters in Experiments:,K=4,m=5,14 feature maps,m=76 feature maps,d=48,Dynamic CNN for Sentiment,Kalchbrenner et al,“A Convolutional Neural Network for Modeling Sentences”,ACL 2014,One Dimension Convolution,Two Dimension Convolution,48 D word vectors randomly initiated,300 D Initiated with Google word2vector,More complicated model architecture with dynamic pooling,Straight Forward,6,4 feature maps,100-128 feature maps,Johnson and Zhang.,“,Effective Use of Word Order for Text Categorization with Convolutional Neural Networks,”,ACL-2015,Why CNN is effective,A,simple remedy,is to use word bi-grams in addition to unigrams,It has been noted that loss of word order,caused by,bag-of-word vectors(bow vectors)is,particularly problematic,on sentiment classification,Comparing SVM with Tri-gram features with 1,2,3 window filter CNN,Top 100 Features,SVM,CNN,Uni-Grams,68,7,Bi-Grams,28,33,Tri-Grams,4,60,SVMs cant fully take advantage of high-order ngrams,Sentiment Classification Considering Features beyond Text with CNN Models,Tang et al.,“Learning,Semantic Representations of Users and Products for Document Level Sentiment Classification,“”,ACL-2015,Overview,INTRODUCTION,EMOTION RECOGNITION IN TEXT,Word Embedding for Sentiment Analysis,CNN for Sentiment Classification,RNN,LSTM for sentiment Classification,Prior Knowledge+CNN/,LSTM,Dataset Collection,EMOTION RECOGNITION IN SPEECH,EMOTION RECOGNITION IN CONVERSATIONS,INDUSTRIAL APPLICATION,Recursive Neural Tensor Network,Socher et al.,“Recursive Deep Models for Semantic Compositionality over a Sentiment,Treebank”,EMNLP-2013.nlp.stanford.edu/sentiment/,The Stanfor,d Sentiment Treeback is a corpus with fully labeled parse trees,Created to facilitate analysis of the compositional effects of sentiment in language,10,662 sentences from movie reviews.Parsed by stanford parser.215,154 phrases are labeled,A model called Recursive Neural Tensor Networks was proposed,Recursive Neural Tensor Network-Distribution of sentiment values for N-grams,Socher et al.,“Recursive Deep Models for Semantic Compositionality over a Sentiment,Treebank”,EMNLP-2013.nlp.stanford.edu/sentiment/,Stronger,sentime