Modeling word and morpheme order in natural language as an efficient trade-off of memory and surprisal学位论文（英文版）.pdf

Mo del in g w o r d a n d mo r pheme o r der in n a t u r a l l a n gu a ge a s a n ef f icien t t r a deo f f o f memo r y a n d su r pr isa lMicha el Ha hn Ju dit h Degen Richa r d Fu t r el lSt a n f o r d Un iv er sit y St a n f o r d Un iv er sit y UC Ir v in e mha hn 2 st a n f o r d.edu jdegen st a n f o r d.edu r f u t r el l u ci.eduJa n u a r y 9,2021AbstractMemo r y l imit a t io n s a r e k n o w n t o co n st r a in l a n gu a ge co mpr ehen sio n a n d pr o du ct io n,a n d ha v e been a r gu ed t o a cco u n t f o r cr o ssl in gu ist ic w o r d o r der r egu l a r it ies.Ho w ev er,a syst ema t ic a ssessmen t o f t he r o l e o f memo r y l imit a t io n s in l a n gu a ge st r u ct u r e ha s pr o v en el u siv e,in pa r t beca u se it is ha r d t o ext r a ct pr ecise l a r ge-sca l e qu a n t it a t iv e gen er a l iza t io n s a bo u t l a n gu a ge f r o m ex ist in g mecha n ist ic mo del s o f memo r y u se in sen t en ce pr o cessin g.We pr o v ide a n a r chit ect u r e-in depen den t in f o r ma t io n-t heo r et ic f o r ma l iza t io n o f memo r y l imit a t io n s w hich en a bl es a simpl e ca l cu l a t io n o f t he memo r y ef f icien cy o f l a n gu a ges.Ou r n o t io n o f memo r y ef f icien cy is ba sed o n t he idea o f a memory-surprisal t radeoff:a cer t a in l ev el o f a v er a ge su r pr isa l per w o r d ca n o n l y be a chiev ed a t t he co st o f st o r in g so me a mo u n t o f in f o r ma t io n a bo u t pa st co n t ex t.Ba sed o n t his n o t io n o f memo r y u sa ge,w e a dv a n ce t he Efficient Tradeoff Hypot hesis:t he o r der o f el emen t s in n a t u r a l l a n gu a ge is u n der pr essu r e t o en a bl e f a v o r a bl e memo r y-su r pr isa l t r a deo f f s.We der iv e t ha t l a n gu a ges en a bl e mo r e ef f icien t t r a deo f f s w hen t hey ex hibit informat ion l ocal it y:w hen pr edict iv e in f o r ma t io n a bo u t a n el emen t is co n cen t r a t ed in it s r ecen t pa st.We pr o v ide empir ica l ev iden ce f r o m t hr ee t est do ma in s in su ppo r t o f t he Ef f icien t Tr a deo f f Hypo t hesis:a r ea n a l ysis o f a min ia t u r e a r t if icia l l a n gu a ge l ea r n in g ex per imen t,a l a r ge-sca l e st u dy o f w o r d o r der in co r po r a o f 54 l a n gu a ges,a n d a n a n a l ysis o f mo r pheme o r der in t w o a ggl u t in a t iv e l a n gu a ges.These r esu l t s su ggest t ha t pr in cipl es o f o r der in n a t u r a l l a n gu a ge ca n be ex pl a in ed v ia highl y gen er ic co gn it iv el y mo t iv a t ed pr in cipl es a n d l en d su ppo r t t o ef f icien cy-ba sed mo del s o f t he st r u ct u r e o f hu ma n l a n gu a ge.1 IntroductionNa t u r a l l a n gu a ge is a po w er f u l t o o l t ha t a l l o w s hu ma n s t o co mmu n ica t e,a l beit u n der in her en t co gn it iv e r eso u r ce l imit a t io n s.Her e,w e in v est iga t e w het her hu ma n l a n gu a ges a r e gr a mma t ica l l y st r u ct u r ed in a w a y t ha t r edu ces t he co gn it iv e r eso u r ce r equ ir emen t s f o r co mpr ehen sio n,co mpa r ed t o co u n t er f a ct u a l l a n gu a ges t ha t dif f er in gr a mma t ica l st r u ct u r e.The su ggest io n t ha t t he st r u ct u r e o f hu ma n l a n gu a ge r ef l ect s a n eed f o r ef f icien t pr o cessin g u n der r eso u r ce l imit a t io n s ha s been pr esen t in t he l in gu ist ics a n d co gn it iv e scien ce l it er a t u r e f o r deca des(Yn gv e,1960;Ber w ick a n d Wein ber g,1984;Ha w k in s,1994;Cho msk y,2005;Ja eger a n d Til y,2011;Gibso n et a l.,*is This is a pr epr in t o f a pa per in pr ess a t Psychol ogical Review.A v er sio n in cl u din g Su ppl emen t a r y In f o r ma t io n(SI Appen dix)is a v a il a bl e a t:ht t ps:/psya r x iv.co m/n u 4qz 2021,Amer ica n Psycho l o gica l Asso cia t io n.This pa per is n o t t he co py o f r eco r d a n d ma y n o t ex a ct l y r epl ica t e t he f in a l,a ut ho r it a t iv e v er sio n o f t he a r t icl e.Pl ea se do n o t co py o r cit e w it ho u t a u t ho r s9 per missio n.The f in a l a r t icl e w il l be a v a il a bl e,u po n pu bl ica t io n,v ia it s DOI:l().1037/r ev()00026912019;Ha hn et a l.,2020).The idea ha s been su mmed u p in Ha w k in ss(2004)Performance-Grammar Correspondence Hypothesis(PGCH),w hich ho l ds t ha t gr a mma r s a r e st r u ct u r ed so t ha t t he t ypica l u t t er a n ce is ea sy t o pr o du ce a n d co mpr ehen d u n der per f o r ma n ce co n st r a in t s.On e ma jo r so u r ce o f r eso u r ce l imit a t io n in l a n gu a ge pr o cessin g is in cr emen t a l memo r y u se.When pr odu cin g a n d co mpr ehen din g l a n gu a ge in r ea l t ime,a l a n gu a ge u ser mu st k eep t r a ck o f w ha t t hey ha v e a l r ea dy pr o du ced o r hea r d in so me k in d o f in cr emen t a l memo r y st o r e,w hich is su bject t o r eso u r ce co n st r a in t s.These memo r y co n st r a in t s ha v e been a r gu ed t o u n der l ie v a r io u s locality principles w hich l in gu ist s ha v e u sed t o pr edict t he o r der s o f w o r ds w it hin sen t en ces a n d mo r phemes w it hin w o r ds(e.g.Beha ghel,1932;Giv o n,1985;Bybee,1985;Rijk ho f f,1990;Ha w k in s,1994,2004,2014;Temper l ey a n d Gil dea,2018).The idea is t ha t l a n gu a ge sho u l d be st r u ct u r ed t o r edu ce l o n g-t er m depen den cies o f v a r io u s k in ds,by pl a cin g el emen t s t ha t depen d o n ea ch o t her cl o se t o ea ch o t her in l in ea r o r der.Tha t is,el emen t s o f u t t er a n ces w hich a r e mo r e r el ev a n t5 o r men t a l l y co n n ect ed,t o ea ch o t her a r e cl o ser t o ea ch o t her.Ou r co n t r ibu t io n is t o pr esen t a n ew,highl y gen er a l f o r ma l iza t io n o f t he r el a t io n ship bet w een sequ en t ia l o r der a n d in cr emen t a l memo r y in l a n gu a ge pr o cessin g,f r o m w hich w e ca n der iv e a pr ecise a n d empir ica l l y t est a bl e v er sio n o f t he idea t ha t u t t er a n ce el emen t s w hich depen d o n ea ch o t her sho u l d be cl o se t o ea ch o t her.Ou r f o r ma l iza t io n a l l o w s u s t o pr edict t he o r der o f w o r ds w it hin sen t en ces,a n d mo r phemes w it hin w o r ds dir ect l y by t he min imiza t io n o f memo r y u sa ge.We f o r ma l ize t he n o t io n o f memo r y co n st r a in t s in t er ms o f w ha t w e ca l l t he memory-surprisal tradeoff:t he idea t ha t t he ea se o f co mpr ehen sio n depen ds o n t he a mo u n t o f co mpu t a t io n a l r eso u r ces in v est ed in t o r emember in g pr ev io u s l in gu ist ic el emen t s,e.g.,w o r ds.Ther ef o r e,t her e ex ist s a t r a deo f f bet w een t he qu a nt it y o f memo r y r eso u r ces in v est ed,a n d t he ea se o f l a n gu a ge pr o cessin g.The sha pe o f t his t r a deo f f depen ds o n t he gr a mma r o f a l a n gu a ge,a n d in pa r t icu l a r t he w a y t ha t it st r u ct u r es in f o r ma t io n in t ime.We cha ra ct er ize memo r y r eso u r ces u sin g t he t heo r y o f l o ssy da t a co mpr essio n(Co v er a n d Tho ma s,2006;Ber ger,2003).Wit hin o u r f r a mew o r k,w e pr o v e a t heo r em sho w in g t ha t l o w er memo r y r equ ir emen t s r esu l t w hen u tt er a n ce el emen t s t ha t depen d o n ea ch o t her st a t ist ica l l y a r e pl a ced cl o se t o ea ch o t her.This t heo r em do es n o t r equ ir e a n y a ssu mpt io n s a bo u t t he a r chit ect u r e o r f u n ct io n in g o f memo r y,ex cept t ha t it ha s a bo u n ded ca pa cit y.Usin g t his co n cept,w e in t r o du ce t he Efficient Tradeoff Hypothesis:Or der in n a t u r a l l a n gu a ge is st r u ct u r ed so a s t o pr o v ide ef f icien t memo r y-su r pr isa l t r a deo f f cu r v es.We pr o v ide ev iden ce f o r t his hypo t hesis in t hr ee st u dies.We demo n st r a t e t ha t w o r d o r der s w it h sho r t depen den cies do in deed en gen der l o w er w o r k in g memo r y r eso u r ce r equ ir emen t s in t o y l a n gu a ges st u died in t he pr ev io u s l it er a t u r e,a n d w e sho w t ha t r ea l w o r d o r der s in co r po r a o f 54 l a n gu a ges ha v e l o w er memo r y r equ ir emen t s t ha n w o u l d be ex pect ed u n der a r t if icia l ba sel in e co mpa r iso n gr a mma r s.Fin a l l y,w e sho w t ha t w e ca n pr edict t he o r der o f mo r phemes w it hin w o r ds in t w o l a n gu a ges u sin g o u r pr in cipl e o f t he min imiza t io n o f memo r y u sa ge.Ou r w o r k n o t o n l y f o r ma l izes a n d t est s a n o l d idea in f u n ct io n a l l in gu ist ics a n d psycho l in gu ist ics,it a l so o pen s u p co n n ect io n s bet w een t ho se f iel ds a n d t he st a t ist ica l a n a l ysis o f n a t u r a l l a n gu a ge(Debo w sk i,2011;Ben t z et a l.,2017;Lin a n d Tegma r k,2017),a n d mo r e br o a dl y,bet w een l in gu ist ics a n d f iel ds t ha t ha v e st u died in f o r ma t io n-pr o cessin g co st s a n d r eso u r ce r equ ir emen t s in br a in s(e.g.,Fr ist o n,2010)a n d gen er a l physica l syst ems(e.g.,St il l et a l.,2012).2 BackgroundA w ide r a n ge o f w o r k ha s a r gu ed t ha t in f o r ma t io n in n a t u r a l l a n gu a ge u t t er a n ces is o r der ed in w a ys t ha t r edu ce memo r y ef f o r t,by pl a cin g el emen t s cl o se t o get her w hen t hey depen d o n ea ch o t her in so me w a y.Her e,w e r ev iew t hese a r gu men t s f r o m l in gu ist ic a n d co gn it iv e per spect iv es.22.1 Dependency locality and memory constraints in psycholinguisticsWhen pr o du cin g a n d co mpr ehen din g l a n gu a ge in r ea l t ime,a l a n gu a ge u ser mu st k eep t r a ck o f w ha t she ha s a l r ea dy pr o du ced o r hea r d in so me k in d o f in cr emen t a l memo r y st o r e,w hich is su bject t o r eso u r ce co nst r a in t s.An ea r l y ex a mpl e o f t his idea is Mil l er a n d Cho msk y(1963),w ho a t t r ibu t ed t he u n a ccept a bil it y o f mu l t ipl e cen t er embeddin gs in En gl ish t o l imit a t io n s o f hu ma n w o r k in g memo r y.Co n cu r r en t a n d su bsequ en t w o r k st u died ho w dif f er en t gr a mma r s in du ce dif f er en t memo r y r equ ir emen t s in t er ms o f t he n u mber o f symbo l s t ha t mu st be st o r ed a t ea ch po in t t o pr o du ce o r pa r se a sen t en ce(Yn gv e,1960;Abn ey a n d Jo hnso n,1991;Gibso n,1991;Resn ik,1992).In psycho l in gu ist ic st u dies,memo r y co n st r a in t s t ypica l l y ma n if est in t he f o r m o f pr o cessin g dif f icu l t y a sso cia t ed w it h l o n g-t er m depen den cies.Fo r ex a mpl e,a t t he l ev el o f w o r d-by-w o r d o n l in e l a n gu a ge co mpr ehen sio n,t her e is o bser v a bl e pr o cessin g dif f icu l t y a t mo men t s w hen it seems t ha t in f o r ma t io n a bo u t a w o r d mu st be r et r iev ed f r o m w o r k in g memo r y.This dif f icu l t y in cr ea ses w hen t her e is a gr ea t dea l o f t ime o r in t er v en in g ma t er ia l bet w een t he po in t w hen a w o r d is f ir st en co u n t er ed a n d t he po in t w hen it mu st be r et r iev ed f r o m memo r y(Gibso n,1998;Gibso n a n d Tho ma s,1999;Gibso n,2000;McEl r ee,2000;Lew is a n d Va sisht h,2005;Ba r t ek et a l.,2011;Nicen bo im et a l.,2015;Ba l l in g a n d Kiza ch,2017).Tha t is,l a n gu a ge co mpr ehen sio n is ha r der f o r hu ma n s w hen w o r ds w hich depen d o n ea ch o t her f o r t heir mea n in g a r e sepa r a t ed by ma n y in t er v en in g w o r ds.This idea is mo st pr o min en t l y a sso cia t ed w it h t he Dependency Locality Theory o f hu ma n sen t en ce pr o cessin g(Gibso n,2000).Fo r ex a mpl e,Gr o dn er a n d Gibso n(2005)st u died w o r d-by-w o r d r ea din g t imes in a ser ies o f sen t en ces su ch a s(1)bel o w.(1)a.The a dmin ist r a t o r w ho t he n u r se su per v ised.b.The a dmin ist r a t o r w ho t he n u r se f r o m t he cl in ic su per v ised.c.The a dmin ist r a t o r w ho t he n u r se w ho w a s f r o m t he cl in ic su per v ised.In t hese sen t en ces,t he dist a n ce bet w een t he n o u n administ rat or a n d t he v er b supervised is su ccessiv el y in cr ea sed.Gr o dn er a n d Gibso n(2005)f o u n d t ha t a s t his dist a n ce in cr ea ses,t her e is a co n co mit a n t in cr ea se in r ea din g t ime a t t he v er b supervised a n d f o l l o w in g w o r ds.The hypo t hesized r ea so n f o r t his r ea din g t ime pa t t er n is ba sed o n memo r y co n st r a in t s.The idea go es:a t t he w o r d supervised,a co mpr ehen der w ho is t r yin g t o co mpu t e t he mea n in g o f t he sen t en ce mu st in t egr a t e a r epr esen t a t io n o f t he v er b supervised w it h a r epr esen t a t io n o f t he n o u n administ rat or,w hich is a dir ect o bject o f t he v er b.This in t egr a t io n r equ ir es r et r iev in g t he r epr esen t a t io n o f administ rat or f r o m w o r k in g memo r y.If t his r epr esen t a t io n ha s been in w o r k in g memo r y f o r a l o n g t imef o r ex a mpl e a s in Sen t en ce 1c a s o ppo sed t o l a-t hen t he r et r iev a l is dif f icu l t o r in a ccu r a t e,in a w a y t ha t ma n if est s a s in cr ea sed r ea din g t ime.Essen t ia l l y,t her e ex ist s a depen den cy bet w een t he w o r ds administ rat or a n d supervised,a n d mo r e ex cess pr o cessin g dif f icu l t y is in cu r r ed t he mo r e t he t w o w o r ds a r e sepa r a t ed;t his ex cess dif f icu l t y is ca l l ed a dependency locality effect.The ex ist en ce o f depen den cy l o ca l it y ef f ect s in hu ma n l a n gu a ge pr o cessin g,a n d t heir co n n ect io n w it h w o r k in g memo r y,a r e w el l-est a bl ished(Fedo r en k o et a l.,2013).These l o ca l it y ef f ect s in o n l in e pr o cessin g mir r o r l o ca l it y ef f ect s in w o r d o r der,descr ibed bel o w.2.2 Locality and cross-linguistic universals of orderDepen den cy l o ca l it y in w o r d o r der mea n s t ha t t her e is a pr essu r e f o r w o r ds w hich depen d o n ea ch o t her syn t a ct ica l l y t o be cl o se t o ea ch o t her in l in ea r o r der.Ther e is a mpl e ev iden ce f r o m co r pu s st a t ist ics in dica tin g t ha t depen den cy l o ca l it y is a r ea l pr o per t y o f w o r d o r der a cr o ss ma n y l a n gu a ges(Fer r er-i-Ca n cho,2004;3Gil dea a n d Temper l ey,2007;Liu,2008;Gil dea a n d Temper l ey,2010;Fu t r el l et a l.,2015a;Liu et a l.,2017;Temper l ey a n d Gil dea,2018).Ha w k in s(1994,2003)f o r mu l a t es depen den cy l o ca l it y a s t he Pr in cipl e o f Doma in Min imiza t io n,a n d ha s sho w n t ha t t his pr in cipl e ca n ex pl a in cr o ss-l in gu ist ic u n iv er sa l s o f w o r d o r der t ha t ha v e been do cu men t ed by l in gu ist ic t ypo l o gist s f o r deca des(Gr een ber g,1963).Su ch a pr essu r e ca n be mo t iv a t ed in t er ms o f t he do cu men t ed o n l in e pr o cessin g dif f icu l t y a sso cia t ed w it h l o n g-t er m depen den cies a mo n g w o r ds:depen den cy l o ca l it y in w o r d o r d