自动化系统评估PPT课件.ppt

自动化系统评估自动化系统评估Evaluation of the performance of an automated system can provide several useful types of management information on(1)whether new or updated systems meet contract requirements;(2)whether the system is living up to the performance and output standards of its user community;(3)the point at which a new system or system refinements are needed;(4)possible future resource consumption.評估自動化系統的效益可以由下列幾種有用的管理資訊予以獲得：(1)新系統或更新系統是否符合合約需求;(2)系統產出或效益是否有達到使用者(群)的要求標準;(3)明確指出新系統或系統強化的需求;(4)未來可能的資源消耗。1-1.第四項資訊即考慮到未來潛藏性的成本支出，與TCO的觀念相符。It is obvious that a computer system can be evaluated according to different types of criteria ease of use,cost,reliability,integratibility,and so on.電腦系統的評估可以依據不同的標準，如容易使用、成本、穩定性以及整合性等等。In her survey of 54 major research libraries in North America,Johnson(1991)discovered that ease of use by patrons was a major consideration in the selection of a new system.above cost and,perhaps surprisingly,ease of use by staff.Her found that these same libraries considered improvement of user services as the major objective of automation and improvement service to users as the major accomplishment of automation.經過評估北美54個主要的圖書館，Johnson(1991)發現使用者感覺容易使用是選擇系統的主要考量。2而她亦發現改善使用者服務是自動化系統的主要目標，亦是自動化的使命。Peters(1988)identifies three types of systems evaluation:(1)functional to determine whether a systems features meet the librarys needs;(2)economic to determine the affordability of a system;and(3)performance to reveal whether the system capacity can meet present or anticipated future demands.Peters定義系統評估三個型式：功能性決定系統特性是否符合圖書館需求；經濟決定系統的支出(時間與金錢等)；效益系統是否能夠符合現今或未來的需求。CriteriaValue123Ease of use by patrons(顧客感覺好用)77.814.87.4Availability of application modules and subsystems(應用模組或子系統之可用性符合使用需求)77.818.57.4Completeness of modules and subsystems(模組或子系統完整性)68.522.29.3Cost of system(系統成本)68.529.61.9Cost of hardware(硬體成本)61.133.35.6Need for local programming stall(MIS人員的需求)59.329.611.1Service reputation of vendor(軟體廠商的保證-e.g.售後服務)53.737.09.3Easy of use by staff(職員或幕僚感覺好用)48.151.90.0Comparable installed sited(軟體的客戶參考)44.540.714.8Previous experience with vendor(軟體廠商的導入經驗)25.929.644.5Training and documentation provided(教育訓練及文件提供)22.266.711.1Bias against vendor(對軟體廠商的徧見)5.625.968.5Key:1.=Seriously considered(審慎考慮)2.=Considered to some extent(僅考慮某些範圍)3.=Not considered at all(從不考慮)There are obviously many possible ways in which approaches to the evaluation of automated systems can be categorized.For the purposes of this chapter,two major approaches are identified:Evaluation without user involvement or with less than full user involvement.(user free)Evaluation with full user involvement.(user involved)評估自動化系統有許多方法，但可歸納成兩類：1.没有使用者參與的評估方法；2.使用者完全參與的評估方法。3-ThinkingTAM模型屬於上述兩種評估方法中第2種型式。User-Free Evaluation:This category of evaluation focuses on system features rather than on how these are exploited by a particular group of users.無使用者參與評估：這類型的評估著重於系統特性而不是特定使用者的試用經驗。這類評估可用於系統的選擇、系統的接受性評估以及系統強化或替換的決策評估。One useful tool that can be used in the selection of systems is a checklist to determine the features present in a particular system or,more particularly,to compare the characteristics of two or more systems.這類的評估通常會使用Checklist 工具，以呈現特定系統的特性或比較兩個系統之間的特色。A point value may be assigned to each feature,and a differential weighting scheme may be established to place emphasis on features that are considered more important than others.In other cases,features are assigned an equal rating of 1 or 0.system scores can be derived from the grids,with subtotals to indicate system strengths in particular areas,and total scores to indicate overall performance.在Checklist中,可針對不同的特色指定權值以強調重要的系統特性。系統評分可以由垂直加總導出，子項加總分數為該系統某特定區項的系統強度，整體加總可以顯示出整體的效益。QuestionsABCDEFGHIJKLDOBISGeacNOTISPALSDRAHOMeHOMe1.Is there adequate logon instruction(i.e.explain which terminal types are supported)2.Are the contents and coverage of the OPAC ckearkt exokaubed?3.Are the key equivalencies explained for remote users keyboard?NANANANANANANA4.Is there adequated logoff instruction?5.Is the screen display always clean?(i.e.,no garbage characters)6.(a)Is remote access unrestricted in terms of time of day?(b)Does the system tell the user if there is a time limit to remote sessions?(c)Does the system give a warning message of automatic logoff if there is no user input?7.Does the remote user have access to the some OPAC as those who use dedicated terminals in the library?8.Does the sytem indicate where the remote user can get additional help?Score(Maximum 10)654755666887Note:“NA”means“not applicable”The checklist method of evaluation is useful for several reasons.In the case of a single system review,it helps one to arrive at a list of desirable features,and to identify the strengths and weaknesses of a particular system.In the case of a multiple system review,a comparative checklist can help to verify the existence of features across systems and thus to identify comparative strengths and weaknesses.在進行系統評估時Checklist是相當有用的工具，對單一系統評估，它可以協助整理需求清單，同時指出特定系統的優缺與弱點。對多個系統的比較上，它協助定義不同系統間的特色差異，並找出強勢與弱點。The use of a checklist ensures that the same questions about system features are posed consistently across systems.Checklist的使用確保問題在不同系統間可以在一致性的標準下進行比較。Cherry et al.(1994)employ4ed a checklist to survey features in the OPACs of twelve Canadian academic libraries.Data on each system were collected twice,by two different researchers,and the two datasets were checked a third time against the systems to resolve any disagreements.One hundred seventy features were included in the checklist,grouped into ten functional categories:1.Database characteristics;2.Operational control;3.Searching;4.Subject search aids;5.Access pints;6.Screen display;7.Output control;8.Commands;9.User assistance;10.OPAC usability via remote access.Cherry等(1994)使用checklist對加拿大十二所學院圖書館進行評估，首先他對兩個研究群組進行系統的特性資料收集，針對此兩個資料集再進行第三次查核以除去誤差以求數據之公正。在他的研究中checklist總共收集了120因素，並予以歸類聚集成十大類別之中。此十大類別為：1.資料庫特性;2.操作性控制;3.搜尋;4.主題搜尋輔助；5.存取點;6.螢幕畫面顯示;7.輸出控制;8.命令輸入;9.使用者輔助;10.經由OPAC遠端存取的使用率。Acceptance testing or benchmarking is a process often used by libraries to verify that the new or upgraded system meets the contract requirements.Often the conditions of acceptance in a contract indicate clearly what type of performance is expected,and the acceptable level of performance,to determine whether a system works in the manner agreed upon in the contract.接受度測試及標竿法經常用於檢驗新系統或系統更新是否符合合約的需求。合約接受性(驗收標準)清楚地指出何種效益必須達到要求，以及效益水準為何、檢驗系統是否在合約所認定的規範下執行。At times,public or staff users identify problems or make suggestions for system changes designed to refine its operation or its interaction with users.The feedback for making these changes can come from word of mouth or from the periodic review of performance logs generated by the system.職員或使用者所找出的問題或建議可於系統功能或設計上予以進行修正或加強，而這些回饋意見的收集可以透過口頭、文字或系統記錄分析獲得。Stress tests are commonly used to test implementations of new features.著重測試(具時迫性測試)是為了對新上線的系統特性予以測試。Capacity planning is another important element in overall evaluation.By tracking the size of the database,and estimating its growth rate,projections can be made about when to increase capacity,and whether this increase in size will degrade or otherwise affect response time and other performance factors.Precise capacity planning is difficult because it involves projection and prediction based on numerous complex performance factors.系統容量規劃亦是整體性評估的重要項目，資料庫容量是否足夠符合未來成長需求。資料存取算是否會影響系統回應時間，而確地規劃是相當困難，因為它包含大量效率相關因素的預測與其相互影響的評估。User-involved Evaluation:For over twenty years,a growing body of research based on information science and cognitive psychology has been performed to gain a better understanding of how users interact with systems,and how the results of that interaction can be evaluated.One practical goal of this type of research is to collect and analyze information that can be fed back into better system design.使用者參與的評估：資訊科學與認知心理學人機互動的研究於近廿年來有長足的進步，甚至對於互動結果亦可進一步進行評估。這類研究的目標在於收集並分析回饋資訊以求更好的系統設計。The interaction between the user and the system can be the subject of study for a number of purposes.The studies discussed in this chapter are carried out to learn more about how a system is used and to improve its performance.Many possible methods are applicable.Unobtrusive measures gather data while library patrons are actually using the system.User may or may not be aware that their keystrokes,or other actions,are being recorded or observed.The methods are unobtrusive in the sense that users are not being asked any questions and are not required to do anything they would not otherwise be doing.基於許多目的，使用者與系統的互動可以做為研究主題，而本章主要是探討如何提昇效益。非強制性觀察法是以使用者實際使用系統的過程收集資料，而使用者可能在不知情的情況下鍵入動作，均被記錄下來並以觀察。這種觀察法使用者不會被問及有關使用系統的任何問題，而這主要的目的在於讓使用者處於自然的環境下，評估使用者的活動。Obtrusive measures are used primarily to obtain feedback on user preferences for various system features and their opinions on system performance.強迫性觀察的評量主要是可以獲得使用者徧好的回饋，以及他們對系統效益的看法。諸如，以訪談方式或在研究者督下進行系統測試。Data thus collected can be useful in revealing how specific system features are exploited and in identifying features that appear to be giving users significant problems.At least three types of approach are applicable:review of transaction logs,direct observation of users operating at terminals,and video and/or audio taping of user performance.特定的系統特質可以依據使用者所提供的重大問題予以發掘。至少有三種方式是可行的：直接檢視交易記錄檔、直接觀察使用者在終端機操作的情形、或者以影音錄製使用者使用系統績效的情形。Transaction log analysis(TLA)has been defined as the“studey of electronically recorded interactions between online information retrieval systems and the persons who search for the information found in those systems.(Peters et al.,1993a)交易記錄分析被定義研究”使用者存取系統與資訊檢索間電子記錄的研究”。交易記錄分析在1970年代被視為分析使用者與線上型錄(選單)間互動的工具。Many TLA studies gather information on how frequently system features are used:choice of search type,use of help screens,how many hits users are willing to review,how often a search results in zero hits,the number and type of error messages that users receive,and so on.多數TLA研究在於收集系統特性、使用的頻率、搜尋型態的選擇、help螢幕以及功能的使用、使用者重覆點選的hits數、使用者看到錯誤訊息的數量。An annotated bibliography by Peters et al.(1993b)and a review article by Simpson(1989)serve as two excellent sources of further information about TLA.使用TLA的著名文獻有Peters et al.(1993b)以及 Simpson(1989)是兩篇相當優秀參考來源，可以做為未來的TLA研究。Despite all of its potential benefits,transaction log analysis does have limitations.In many systems with transaction log monitoring facilities,it is either difficult or impossible to delineate individual user searching sessions.除了上述所提及的優點外,TLA方法亦有其限制，在許多系統中TLA要找到別使用者的搜尋Session是相當因難，甚至是不可能的。另一個限制是在跨系統的比較上，TLA並不適合且無法顯示相同的特性。Another problem is that of cost.A comprehensive monitoring module can add a significant overhead to the cost of operating the system.另一個問題是成本考量，持續的觀察對系統的運作是不個明顯的負擔。甚至對圖書館人員及管理者而言缺乏時間去分析這項的訊息。Transaction log analysis collects data about system use in the aggregate and deals on with the quantitative-which commands are used how often,which heading are consulted,how much time is spent per session,and so on.The most obvious example is the monitoring and analysis of use of a help command.Knowing what types of help are requested by users,especially in the case of a new system or one that has recently added new features,can be of great value in identifying problem areas that may not have been anticipated in the system design but may in fact,be rather easy to correct.TLA 在系統使用分析上以量為分析對象包含對使用頻率、上線的時間以及對系統項目的協助查詢等，在量的收集包含總數及其分配。最明顯的例子是觀察並分析help指念的使用狀況。如果知道何種型態的help command是使用者最常發送的動作，則能清楚對新系統或新增功能找出其問題所在，而這些問題剛好是在系統設計時未曾被考量但卻需予以修正。Although it is rarely acknowledge,direct observation is perhaps one of the most commonly employed techniques for collecting information about online system users.Critics often suggest that observation is an unscientific way of gathering only the information needed to support ones own views.The technique need not be flawed;it is the degree of consistency in what is observed,and at what intervals it is observed,that determines the reliability of the data collected.雖然較少受到研究者的認同，直接觀察法可能是收集線上系統使用者資訊中最普遍的技術。評論家認為這類方式是不符科學方式(僅收集符合研究者論點的資訊)。然而這類技術並不完全有缺陷，重要的是所觀察事件現象的一致性程度如何，且觀察時段區間如何，而這些均是決定所收集到資訊的可靠性。It is important to employ valid sampling techniques in conjunction with observation in order to obtain reliable data on which management decisions can be based.For examples,if one wants to know how many times users have to wait in line to use terminals in the reference room,one obviously cannot rely solely on the observations of a single librarian who only staffs the reference desk fifteen hours per week,between 8 a.m.and 5 p.m.,Monday through Friday.Direct observation can be useful,only its own or to supplement other methods,when appropriate sampling methods are employed and input is received from more than one observer.使用直接觀察法必須結合其他的抽樣技術以得到可靠性的資料，這樣方能提供管理決策使用。例如，如果想要得知使用者在閱覽室中需多少時間排隊方能使用終端機，若僅依據某一圖書館人員在一週內上班時間中十五小時的觀察根本不夠。直接觀察法必需有多人觀察收集資訊且結合其他的抽樣等收集方法才能具備有用性。A number of studies have analyzed the results of video and/or audio taping of the speech and actions of users during search sessions.The technique has been used to examine whether the cognitive,affective,or attitudinal behavior of users affects their performance and the outcomes of searching.Methods likes protocol analysis,which employ a pre-determined framework for analyzing user comments(asking a user to“think-aloud”while searching,then recording the resulting behavior and comments),can be used to classify and evaluate the relative effect of user decision-making and behavior on the success or failure of searching.有些研究利用錄影帶或錄音帶的方式將使用者在搜尋的連線過程中所做的動作及關鍵字(併字)予以記錄並分析。這樣的技術已被使用為評估是否使用者的認知或態度會影響搜尋結果其效率與否。上述方法類似protocol analysis(要求使用者在搜尋時自言自語，同時記錄其行為及使用評語)，可以進一步對使用者的決策過程及搜尋行為的結果予以評估。Survey questionnaires enable the collection of data about user satisfaction with a system or specific aspects of it,searching preferences and attitudes,demographic data about users,and the level of skills or knowledge that users possess.從問卷亦可以收集到使用者對系統的滿意度及使用觀點，如搜尋偏好及搜尋態度、使用者的人數統計以及使用者的資訊素養等。The published literature includes several studies in which online questionnaires were applied to record user attitudes and preferences for system features.The advantage of online questionnaires is the ability to collect critical incident data about session immediately following the session.應用線上問卷可以記錄使用者對系統特性的態度及偏好研究，同時可以對使用者在搜尋的連線狀態下收集到重要事件的資料。In comparison with questionnaires and transaction log analysis,interviews can provide a more intimate view of the users perspective on the system under examination.相較於問卷調查及線上交易記錄分析，訪談法可以提供使用者對系統更詳細更深入的觀點。Another type of interview the focus group interview may be conducted with a small group and one or more interview,with video and/or audio taping,and/or assistants transcribing notes and statements during the group discussion.Focus group interviews are regularly used in marketing research to gather information from a particular,pre-selected group of users about products or potential products.1另一種訪談法是特定群組訪談，它可是和一個小群組或多個小群組進行訪談(可以錄音或錄影、或者是以小組討論記錄方式)。特定群組的訪談通常使用於市場研究或收集特定群組對產品及潛在產品的看法。1:亦可以以線上群組聊天方式收集群組意見，進行網路虛擬社群相關主題研究。Some online systems offer the option for users to send unsolicited commen