Evaluation of information retrieval system

Chapter 10 Evaluation of information retrieval system Introdution An evaluation is basically a judgment of worth. In other words, we evaluate a system in order in escertain the level of its performance or its value. Lancaster states that we can evaluate an information retrieval system by considering the following three issues: 1. How well the system is satisfying its objectives, that is, how well is satisfying the demands placed upon it 2. How efficiently it is satisfying its objectives, and finally 3. Whether the system justifies its existence. In the information retrieval environment we may want to assess which of to existing system performs better, or we may try to assess how the level of performance of a given system can be improved. It is thus quite evident that during an evaluation study we measure the performance of the system under study against some sort of scale. There are two basic parameters for measuring the performance of a system effectiveness and afficiency. By effectiveness we mean the level up to which the given system attains its stated objectives. In an informaton retrieval system, the effectiveness may be a measure of how far it can retrieve relevant information while with holding non-relevant information. By efficiency we mean how economically the system is achieving its objective. In an information retrieval system efficiency can be measured by such factors as at what minimum cost does the system function effectively. The cost factors are be calculated indirectly. The include such factors as response time, that is time taken by the system to provide an answer: user effort, i.c.the amount of time and effort needed by a user to interact with the system and analyse the output retrieved in order to get the correct information: the financial expenditure involved per search, and so on. This chapter discusses the major issues related to the evaluation of information rretrieval systems. Factors to be taken into consideration for measuring to performance of information retrieval systems are identified and followed by a discussion on the major steps to be followed in designing a programme for evaluation of such a system. The purpose of evaluation Evaluation studies investigate the degree to which the stated goals or expectations have been achieved or the degree to which these can be achieved. Keen gives three major purposes of evaluating an information retrieval system: 1. The need for measures with which to make merit comparisons within a single test situation. In other words, evaluation studies are conducted to compare the merits (or demerits) of two or more systems. 2. The need for measures with which to make comparisons between results obtained in different test situations, and 3. The need for assessing the merit of a real-life system. Swanson states that evaluation studies have one or more of the following purposes: 1. To assess a set of goals, a programme plan, or design prior to implementation 2. To determine whether and how well goals or peformance expactations are being fulfilled 3. To determine specific reasons for successes and failures 4. To uncover principles underlying a successful programme 5. To establish a foundation of further research on yhe reasons for the relative success of alternative techniques, and 6. To improve the means employed for attaining objectives or to redefine subgoals or goals in view of research findings. Evaluation criteria An evaluation study can be conducted from two different points of view. When it is conducted from managerial point of view, the evaluation study is called management-oriented; conducted from the users’ point of view it is called a user-oriented evaluation study. Many information scientists advocate that evaluation of an information retrieval system should always be user-criented, i.e.evaluators should pay more attention to those factors that can provide improved service to the users. Cleverdon says that a user-orientation evaluation should try to answer the following questions. 1. To what extent does the system meet both the expressed and latent needs of its users’ community? 2. What are the reasons for the failure of the system to meet the users’ needs? 3. What is the cost-effectiveness of the searches made by the users themselves as against those made by the intermediaries? 4. What basic changes are required to improve the output? 5. Can the cost be reduced while maintaining the same level of performance? 6. What would be the possible effect if some new services were introduced or an existing service were withdrawn?