QGAE：用于生成问答对的端到端无答案问题生成模型.pdf

QGAE:an end-to-end answer-agnostic question generationmodel for generating question-answer pairsLinfengLi1,LichengZhang2,ChiweiZhu1,andZhendongMao11School of Cyber Science and Technology,University of Science and Technology of China,Hefei 230027,China;2School of Information Science and Technology,University of Science and Technology of China,Hefei 230027,ChinaCorrespondence:ZhendongMao,E-mail:2024TheAuthor(s).ThisisanopenaccessarticleundertheCCBY-NC-ND4.0license(http:/creativecommons.org/licenses/by-nc-nd/4.0/).Cite This:JUSTC,2024,54(1):0102(8pp)ReadOnlineAbstract:Questiongenerationaimstogeneratemeaningfulandfluentquestions,whichcanaddressthelackofaquestion-answertypeannotatedcorpusbyaugmentingtheavailabledata.Usingunannotatedtextwithoptionalanswersasinputcontents,questiongenerationcanbedividedintotwotypesbasedonwhetheranswersareprovided:answer-awareandanswer-agnostic.While generating questions by providing answers is challenging,generating high-quality questionswithoutprovidinganswersisevenmoredifficultforbothhumansandmachines.Toaddressthisissue,weproposedanovelend-to-endmodelcalledquestiongenerationwithanswerextractor(QGAE),whichisabletotransformanswer-agnosticquestiongenerationintoanswer-awarequestiongenerationbydirectlyextractingcandidateanswers.Thisap-proacheffectivelyutilizesunlabeleddataforgeneratinghigh-qualityquestion-answerpairs,anditsend-to-enddesignmakesitmoreconvenientthanamulti-stagemethodthatrequiresatleasttwopre-trainedmodels.Moreover,ourmodelachievesbetteraveragescoresandgreaterdiversity.OurexperimentsshowthatQGAEachievessignificantimprovementsingeneratingquestion-answerpairs,makingitapromisingapproachforquestiongeneration.Keywords:deeplearning;naturallanguageprocessing;answer-agnosticquestiongeneration;answerextractionCLC number:TP391.1Document code:A1 IntroductionQuestiongeneration1,2(QG)isdefinedasthetaskofgenerat-ing fluent,meaningful questions automatically from textswithoptionalanswers,soitcanbemainlydividedintotwostreams:answer-aware QG3 that requires answers,andanswer-agnosticQG4thatdoesnot.QGisthereversetaskofquestionanswering(QA),whichisalong-standingandvalu-abletaskhelpingcomputersachievemachinereadingcom-prehension5,6,dating back to the 1960s7.As with manyothersupervisedlearning8,9tasks,QAwillalsoencounterthelackofannotateddatainspiteofthefactthatannotateddatasometimesmakethemostessentialpartofthewholework.QGisapopularchoicefordataaugmentationforQAtoal-leviateinsufficientlabeleddata.Withthecontinuousdevelop-ment of Internet technology,it is becoming increasinglyeasier to obtain valuable data from the Internet.However,question-answerpairs(asshowninTable1)arestillsuchex-pensivecorporathattypicallyrequiremanualannotationbycrowdsourcingbeforebeingusedforsupervisedlearningonQAandQGtasks.Toalleviatethehigh-costproblemofgen-eratingquestion-answerpairs,itisnaturaltoconsideranswer-agnosticQG,sinceitsonlyinputisrawtext.Although labeled answers are not necessary,answer-agnosticQGisstillfacingagreatchallenge.Mostpreviousworks focused on providing additional information to theirmodelsbyleveragingnamedentityrecognition(NER)10toobtain extra linguistic features,adding answer positionfeatures11,usingknowledgegraphs12,andsomeothermeth-odstoimprovethegenerationeffect.Thesemethodseffect-ivelyimprovethefluencyandaccuracyofgeneratedtexts,butanswer-agnosticQGstillperformsworsethananswer-awareQG.Thus,answer-awareQGmayplayanirreplaceablerole,andchanginganswer-agnosticQGtoanswer-awareQGisagoodchoice.Apartfromthis,thereisstillanobstacleingen-eratingquestion-answerpairsthatanswer-agnosticQGcantgenerateanswers.Toaddressthisissue,researchersoftenaddan additional measure for question-answer pair generation:answerextraction.Comparedwithgeneratingananswer,ex-tractinganexactspaninthecontextismuchsimpler.Explicitlyextracting candidate answers will not only re-solvethedemandforthelackofanswersbutalsocantrans-formanswer-agnosticQGintoanswer-awareQG.AsshowninFig.1,someworkssuchasRGF13(retrieve-generatefilter)proposedamulti-stagepipelinemethodtohandletheprob-lem.Amulti-stagepipelinemethodisoftendesignedincom-plexity,includingseveralparts,andeachpartmayneeddif-ferent inputs.Some early RNN-based1417 works optimizedpipeline methods in an end-to-end way,which makes theoverallstructurelighterandfaster.Thoughpre-trainedlan-guagemodels(PLMs)haveoccupieddominanceinbothnat-urallanguagegenerationandunderstanding,thereisstillnoend-to-end work using pre-trained models to generatequestion-answerpairs.WearesurethereisenoughpotentialforPLMstoachievethetask.Articlehttp:/Received:January 08,2023;Accepted:April 14,202301021DOI:10.52396/JUSTC-2023-0002JUSTC,2024,54(1):0102Inthisstudy,wearemotivatedbytheweakperformanceofanswer-agnosticQGcomparedtoanswer-awareQG,inspiredbythecombinationofQGandAEtasks,tryingtoproposeananswer-agnostic question generation model called questiongeneration with answer extractor(QGAE)to alleviate thehighdemandforlarge-scaleQApairs.QGAEisamulti-taskmodelthatrequiresonlyrawtextsasinputandcanachievethedualtasks:answerextractionandquestiongeneration.WedesignourmodelbasedonthePLMmodelBART18,whichhasdualencodersandadecodertogeneratequestionsandex-tractanswersinparallel.Inourstudy,questiongenerationisthemaintask,whichisthemostchallengingpartsimilartoallothergenerationtasksforgeneratedtextshighsyntacticdi-versityandsemanticsubstitutability,sowepaymoreatten-tionandassignahigherweighttothecorrespondingmodule.Thereforeanswerextractionisconsideredanauxiliarytask.Thedesignnotonlymakesitfeasibletoturnanswer-agnosticquestion generation into answer-aware question generationbutalsoenablesthemodeltobeconsideredcapableofgener-atingquestion-answerpairs.Thecontributionsofthispaperaresummarizedasfollows:Wearethefirsttoproposeanewend-to-endmodelus-ingPLMs,whichiscalledQGAEforanswer-agnosticques-tiongeneration.TheQGAEmodelgeneratesquestion-answerpairsfromunannotatedtextswithoutrequiringanyadditionalinformation.Ourmodelachievesstate-of-the-artperformanceingen-eratinghigh-qualityquestion-answerpairs,outperformingex-istingmethodsbyasignificantmargin.Therestofthispaperisorganizedasfollows.InSection2,wereviewtherelatedworksofquestiongenerationandan-swerextraction.InSection3,weformulatetheQGtaskandAEtask.InSection4,wedescribeeachmoduleofourQGAEmodel.InSection5,weintroduceourexperimentindetail.InthelastSection6,weconcludethisworkandgiveadetailedanalysis.2 Related works2.1 Question generationTheQGfieldwasdevotedgreatinterestbyresearchersforitsgreatpotentialbenefits;therefore,ithasmadegreatprogressinapplicationscenariossuchasdataaugmentation19,chat-bots20,machine reading comprehension21,and intelligenttutors22.Intheneuralmodelage,Duetal.4proposedthefirstneur-alQGmodelfocusedonanswer-agnosticQG.Theyinvestig-atedtheeffectofencodingsentence-vs.paragraph-levelin-formationbyusinganattention-basedmodelandfoundthatTable 1.AcaseofQA-pairsgeneratedbyourQGAEmodel:themodelaccepts unannotated texts as input,extracts the highlighted phrase“Lorentzslaw”asananswer,thenusesthisanswertomakequestiongeneration.Input context:Throughcombiningthedefinitionofelectriccurrentasthetimerateofchangeofelectriccharge,aruleofvectormultiplicationcalledLorentzs lawdescribestheforceonachargemovinginamagneticfield.Theconnectionbetweenelectricityandmagnetismallowsforthedescriptionofaunifiedelectromagneticforcethatactsonacharge.Thisforcecanbewrittenasasumoftheelectrostaticforce(duetotheelectricfield)andthemagneticforce(duetothemagneticfield).Extracted answer:LorentzslawGenerated question:Whatdescribestheforceonachargemovinginamagneticfield?Answer extractionQuestiongenerationAnswer extractionmoduleQuestiongenerationmoduleQAQAInput:Unlabeled texts(context)Input:Extracted context span as candidate answer and its corresponding contextOutput:Question-answer pairsStage 1Stage 2Multi stagesEnd to endFig.1.Thedifferencebetweenmulti-stagemethodsandend-to-endmodelsisthatamulti-stagemethodusuallyhasmorethanonemodelinthewholeworkflow.Ineverystage,amulti-stagemethodmayneedtodealwithdifferentinputsandoutputs,whileonthecontrary,anend-to-endmodelonlyneedsadefinitekindofinput.QGAE:anend-to-endanswer-agnosticquestiongenerationmodelforgeneratingquestion-answerpairsLietal.01022DOI:10.52396/JUSTC-2023-0002JUSTC,2024,54(1):0102asthesizeoftheinputtextincreased,theevaluationscoreoftheoutputdecreased.Todealwiththerareorunknownwordproblem,Gulcehreetal.23proposedacopymechanismthatwasfirstusedintheneuralmachinetranslation24tosolvetheout-of-vocabularyproblem.ThismechanismwasabsorbedintheQGtaskandwidelyused.Followingtheoldexperienceofrule-basedQG25,Wuetal.26suggestedtwonewstrategiestodealwiththistask:questiontypepredictionandacopylossmechanism.Duetal.15combinedanswerextractionandques-tiongenerationinanLSTM27modelincludinganswerfea-ture embedding,denoting answer span with the usual BIOtaggingscheme28.In the transformer-based29 PLM era,compared to auto-encodermodels,auto-regressive30modelsarewidelypickedas baselines for the QG task.Laban et al.20 fine-tuned aGPT231asthebasepartofaquestion-drivennewschatbot.Wangetal.32leveragedBARTtoproposeQAGS(questionanswering and generation for summarization)to evaluateautomaticsummarization.Bhambhoriaetal.33leveragedT534togenerateQApairsforCOVID-19literature.Paranjapeetal.13developedaretrieve-generatefilter(RGF)techniquetocreatecounterfactualevaluationandtrainingdatawithminimalhumansupervision,whichisamulti-stagejob.Thetraditionalworksabovehavemotivatedustoexpli-citly infer the candidate answer to transform the answer-agnosticQGintotheanswer-awareQG.Meanwhile,PLMswithfine-tuningachievedSOTAinmanyNLPfields,becom-ingbenchmarks hard to bypass.In multi-stage work,re-searcherswillchoosedifferentPLMsfordifferentstagesinquestion-answerpairgeneration,whichiseffectivebutheavy.Theresstillnoend-to-endworktohandlethewholetask.Therefore,wecombineanswerextractionandquestiongener-ationusingPLMsandproposeanend-to-endmodelthatex-tractsanswersandgeneratesquestionsinparallel.2.2 Answer extractionInformationextraction35,36(IE)isbasicallydefinedasthetaskofturningtheunstructuredinformationexpressedinnaturallanguagetextintoastructured3-tuplerepresentationas(NE1;R;NE2).Thus,answerextractioncanbeseenasasub-fieldofIE,expectingtopickthemostvaluablephrasefromtuples,regardlessofwhetheritisanamedentity,arelation,ortheircombination:anevent.ManyIEsystemshavebeenproposedforopendomains.Yahyaetal.37describeReNoun,anopeninformationextractionsystemthatcomplementspreviousef-fortsthatrelyonbigknowledgebasesbyfocusingonnomin-alattributesandonthelongtail.DelCorroandGemulla38proposedClausIE,anovel,clause-basedapproachtoopenin-formationextraction,whichextractsrelationsandtheirargu-ments from natural language text.Additionally,some rule-based systems using man-made extraction rules have beenproposed,including verb-based39,semantic role labeling40,anddependencyparsetrees41.Intheeraofpre-trainedmodels,auto-encoder42models,such as BERT43 havemade great progress in natural lan-guageunderstanding(NLU)tasks.BERTachievesSOTAintheGLUE44scorewhichisamulti-taskbenchmarkincludingnamedentityrecognition.ItisadeclarationthatlargePLMsare blossoming in the IE field and will take the place oftraditionalmethods.3 Task definitionQ=q1,q2,qnC=c1,c2,cmnmQAnswer-agnostic question generation.Itaimstogeneratefluent,meaningful questions from un-labeledinputcontextwithoutaspecifican-swer.Supposethelengthofthequestionsequenceis whilethelengthofthecontextsequenceis.Duringtraining,thistaskaimstomaximizetheconditionalprobabilityof.Allrelevantparametersinthemodelaredenotedby:p(Q|C;)=nt=1p(qt|C,qit;),(1)qtqitCwheretheprobabilityofeachispredictedbasedonallthewordsgeneratedpreviously(i.e.,),andinputsentence.Inourwork,wesplittraditionalanswer-agnosticquestiongeneration into 2 sub-tasks:answer extraction and answer-awarequestiongeneration,asinearlyworks.C=c1,c2,cmA=ai,ai+1,ajAC1 i j mAnswer extraction.It supposes there is at least onequestion-worthy candidate answer in the input contextandthenreturnsitsanswer,where sspanislimitedby,therefore,.A=a1,a2,allAnswer-aware question generation.It is similar toanswer-agnosticquestiongenerationwhileitprovidesanad-ditional answer,is the length of theanswer:p(Q|C,A;)=nt=1p(qt|C,A,qit;).(2)4 Model4.1 Foundation modelWechoose BART(bidirectional and auto-regressive trans-former)as our foundation model.BART is a sequence-to-sequence model that uses a standard transformer-basedencoder-decoder architecture,inheriting its encoder fromBERTsbidirectionalencoderanditsdecoderfromGPTsleft-to-rightdecoder,andisparticularlyeffectivefortextgenera-tionaswellasreadingcomprehensiontasks.OnelimitationofBARTisthatitcannotsimultaneouslyperformNLUandNLG(naturallanguagegeneration)tasks.Itexcelsattaskssuchastextgenerationandreadingcomprehensionindividu-ally,butintegratingthesetasksinasinglemodelremainsachallenge.However,withitsstrongfoundation,webelievethatBARThasthepotentialtobefurtherimprovedtohandlesuchtaskseffectively.4.2 QGAEACA,CA,C,QQGAEisasequence-to-sequencemodelasshowninFig.2whichmainlyadoptsBARTsarchitecturewhileaddinganadditionalencoder,sotherearetwoencodersandadecoder.Themodelfirstextractsthephrasewithhighprobabilityasandrebuildsinputto.Themodelwillreturnthere-buildinputand.4.2.1AnswerextractorencoderAnswerextractorencoderisthefirstencoderinheritedfromBARTsimilartoBERTandisusedtounderstandtheinputLietal.01023DOI:10.52396/JUSTC-2023-0002JUSTC,2024,54(1):0102context and extract the most valuable phrase.We leveragethisencoderbyappendinganextralinearasaclassifiertopredict the high probability answer span position.BecauseBARTonlysupports,atmost,apairofsequencesasinput,wechoosethehighestscoreanswerofallpredictionsasthecandidateanswer.Thismodulewillfocusonthefirsttaskan-swerextraction(AE).KK0,m1mxi,kikpqWeselectcrossentropytocalculatethelossoftheAEtask.isthenumberofclasses.Inthistask,classistheposi-tionoftheinputparagraphspanintherange,andistheinputcontextlength.indicatesthatthe thsampleisthe thcategory.istheprobabilitydistributionofannot-ateddatawhile istheprobabilitydistributionofpredictiondata:H(p,q)=1NNi=1Kk=1p(xi,k)log(q(xi,k).(3)Concretely,weputthespecificanswerintoEq.(3),andtheequationcanbechangedas:LAE=(a,a)=1NNi=1Li,(4)Li=Kk=1ti,kloge ai,kKj=1e ai,j,(5)a aNti,kikwhere isthelabeledanswerspanasground-truth,isthetargetcandidateanswerspan,andisthedatasize.indic-atesthatthetruelabelofthe thansweristhe thcategory,whichcanonlytake0or1.4.2.2Questiongenerationencoder-decoderQuestiongenerationencoder-decoderismainlyderivedfromsC/ssA/s/sC/sBARTbutaddsauniquefunctionleveragingthecandidateanswer extracted from the first encoder to rebuild input to traditional QG inputs as.Then,themoduleusesrebuiltinputtogeneratetextasBARTdoes.Thismodulewillfocusonthesecondtaskquestiongen-eration(QG).q qKThelossoftheQGtaskisalsocrossentropywiththeonlydifferencebeingthatweusethelabeledquestions asground-truthandpredictionquestions,andclassisthevocabu-larysizeofthemodel:LQG=(q,q)=1NNi=1Li,(6)Li=Kk=1ti,kloge qi,kKj=1e qi,j.(7)4.2.3QGAElossTheQGAElossisthelossofthemulti-taskmodel,inthiswork,itisthesumoftheanswerextractionlossandquestiongenerationloss:L=LAE+(1)LQ