【课件】MemoryaugmentedNeuralMachineTranslation.ppt

Click to edit Master title style,Click to edit Master text styles,Second level,Third level,Fourth level,Fifth level,*,*,MemoryaugmentedNeuralMachineTranslation,Outline,Introduction,Attention-based,NMT,Memory-augmented,NMT,Experiments,Conclusions,Future,work,Reference,Introduction,Statistical,Machine,Translation,(SMT),Phrase-based,machine,translation,(,Moses,Koehn,et,al.,2007,),Phrase,table,+,language,model,An,example:,什么 是 成人 高考,|,成人 高考 介绍,Phrase,table:,什么 是,=,介绍,成人 高考,=,成人 高考,Language,model,guides,the,order,Neural,Machine,Translation,(NMT),Achieved,significant,success,especially,when,dataset,is,big,enough,NMT,performs,quite,better,than,SMT,Introduction,An,interesting,insight:,Lets,say,we,have,a,zh-en,translation,task,and,the,number,of,Chinese,words,in,training,set,is,150,000.,In,SMT,the,vocabulary,size,is,150,000,OOV,(out,of,vocabulary),words,only,appear,in,test,set.,In,NMT,since,“word,embedding”,is,trained,along,with,the,model,typically,the,vocabulary,size,has,to,be,set,to,30,000,.,The,remained,120,000,words,are,uniformly,labeled,as,one,word,“UNK”.,So,OOV,problem,is,dramatically,aggravated,in,NMT.,But,surprisingly,NMT,is,better,than,SMT.,Why?,NMT,is,very,good,at,reasoning!,Introduction,Overfits,to,frequent,observations,while,overlooks,special,cases.,NMT,gives,a,reasonable,translation,but,the,meaning,drifts,away.,An,experiment:,after,decoding,training,set,30,000,English,vocabulary,shrinks,to,26911.,Introduction,Our,aim:,To,address,rare,and,unknown,word,problems,Our,method:,augment,NMT,with,a,memory,component,which,memorizes,source-target,word,pairs.,Its,like,equipping,a,translator,with,a,dictionary.,Outline,Introduction,Attention-based,NMT,Memory-augmented,NMT,Experiments,Conclusions,Future,work,Attention-based,NMT,Outline,Introduction,Attention-based,NMT,Memory-augmented,NMT,Experiments,Conclusions,Future,work,Memory-augmented,NMT,Memory-augmented,NMT,Memory-augmented,NMT,OOV,treatment,Main,idea:,Represent,an,OOV,word,by,its,similar,word,in,vocabulary,An,example:,Src:,目前 没有 治愈,阿尔兹海默症,的 方法,Word,mapping:,UNK,Not,UNK,Res:,Currently there is no cure for alzheimer s disease,Note,that,similar,words,can,either,be,defined,by,human,or,selected,based,on,word,vector,similarity.,Outline,Introduction,Attention-based,NMT,Memory-augmented,NMT,Experiments,Conclusions,Future,work,Experiments,(zh-en),Data:,IWSLT:,44K,sentence,pairs,in,training,set,13,000,zh,words,9,500,en,words.,NIST:,1M,sentence,pairs,in,training,set,190,000,zh,words,100,000,en,words.,Systems:,SMT:,Moses,NMT,NMT-L,(,Arthur,P.,et,al.,2016,),NMT-PL,(,Minh-Thang Luong,et,al.,2014,),M-NMT,Evaluation,metrics:,BLEU:,the,average,of,1-4,grams,bleu,multiplied,by,a,brevity,penalty,Translation,baseline,OOV,baseline,Experiments,(zh-en),Two,observations:,M-NMT,performs,best,M-NMT,brings,more,improvement,on,IWSLT,corpus,Two,conclusions:,M-NMT,is,effective,M-NMT,is,robust,Experiments,(zh-en),M-NMT,recalls,more,OOV,words.,Experiments,(zh-en),Experiments,(zh-uy),Data:,180k,sentence,pairs,170,000,Uyghur,words,130,000,Chinese,words,Performance:,Systems,SMT,NMT,M-NMT,1-gram BLEU,54.5,57.7,58.8,2-gram BLEU,34.6,39.8,40.8,3-gram BLEU,26.6,31.9,32.4,4-gram BLEU,22.1,27.0,27.1,Brevity,penalty,1.000,0.939,0.968,BLEU,32.44,35.24,36.88,Systems,Recalled,words,in,test,SMT,3680/6666,NMT,3509/6666,M-NMT,3560/6666,*,6666,is,the,number,of,words,in,reference,Outline,Introduction,Attention-based,NMT,Memory-augmented,NMT,Experiments,Conclusions,Future,work,Conclusions,M-NMT,alleviates,rare,word,and,under-translation,problems,in,NMT.,M-NMT,provides,a,way,to,address,OOV,problem.,So,far,M-NMT,brings,at,least,1.6,BLEU,improvement,on,different,datasets.,Outline,Introduction,Attention-based,NMT,Memory-augmented,NMT,Experiments,Conclusions,Future,work,Future,work,Better,OOV,treatment?,No,need,to,do,similar,word,replacement,Implement,to,the,whole,dataset,Phrase-based,memory?,Reference,Koehn,Philipp,Hoang,Hieu,Alexandra,&CallisonBurch,et al.(2007).Moses:open source toolkit for statistical machine translation.,in Proceedings of the Association for Computational Linguistics(ACL07,9,(1),177-180.,Bahdanau,D.,Cho,K.,&Bengio,Y.(2014).Neural machine translation by jointly learning to align and translate.,Computer Science,.,Arthur,P.,Neubig,G.,&Nakamura,S.(2016).Incorporating discrete translation lexicons into neural machine translation.,Minh-Thang Luong,Ilya Sutskever,Quoc V Le,Oriol,Vinyals,and Wojciech Zaremba.2014.Addressing the rare word problem in neural machine translation.,arXiv preprint arXiv:1410.8206,.,Thanks!,Q&A,