The Guidelines project : Providing instruments for evaluating development research

Things to be examined and agreed upon prior to start writing a specific guide

General

This preliminary and necessary phase concerns equally the evaluation of research, of researchers, and of scientific publications. It is dedicated to :

The output of this preliminary phase will be a set of written documents, each fitting adequately the specific situation and needs, such as :

The present page provides insights on such points. The steps to be followed when actually writing the instruments and procedure in final form are described in "evaluation of research projects", the "evaluation of researchers", and of the "evaluation of scientific publications".

When critically reviewing or amending an existing evaluation procedure, the same phase may be applied.

Steps

In this preliminary phase it might be useful to flexibly take the following sequence :

Step 1. Define the objectives of evaluation

The evaluation objectives need to be clearly defined early in the process of writing a guide. Basically the purpose of evaluation is to help making decisions about :

Obviously the methodology for evaluating development research will depend on the purpose of evaluation. Each of these aims asks for a different and specific approach. Let us remember here that another possible objective ( a very general one, anyhow) is to contribute to maintain and increase the quality of development research. The authors of the specific evaluation guide will therefore insist that the evaluators (the users of the specific guide) would examine the questions below, and explicitly decide upon the answers. The questions are: what is it that the evaluator wants to know in this specific case ? What will be done with the answers ? In other words one needs to :

In most cases, obviously, a single evaluation will meet more than one objective. Examples of possible objectives are given in "Objectives of evaluation of development research"

Step 2. Which is (or are) the disciplines(s) or categories of disciplines involved ?

It must be clear within which discipline or category of disciplines the tools and procedures will be used, and one most take into consideration the specificity of such discipline(s). A convenient beginning may be to use the three broad areas of the Academy’s operations:

• social and human sciences,
• natural and medical sciences,
• engineering and exact sciences
  

Other categories are listed in "Disciplines or groups of disciplines involved"

Step 3. Who needs evaluation ? Actors and stakeholders

The identification of actors and stakeholders is the third logical step in the building of an evaluation guide or of evaluation tools. The future users of the specific guide, in particular, need to be precisely identified.

They are people or organizations who have a stake, an interest in good development research and/or need to make decisions about research – or researchers - in this area. Their diversity is remarkable and their needs and situations also vary widely. The present chapter should therefore be interpreted and used in a flexible manner.

If necessary the responsibilities of each actor will be defined, together with appropriate terms of reference.

Step 4. Nature and type of evaluation

The nature and type of evaluation described in the specific guide will of course depend on decisions made on earlier points. The guide should in particular specify whether the evaluation will be :

- external, participatory, or both
- national or international.

In a few cases it may be appropriate to use instruments such as the “Logical Framework” or CPPE “Comprehensive and Participatory Planning and Evaluation” (both used successfully in a somewhat different context : the evaluation of development projects and programmes). That issue , if necessary, may be touched at this point.

In this latter case the authors of the specific evaluation guide may use the references provided in "dimensions of evaluation; criteria", and decide whether such an approach is appropriate with their needs.

Step 5. Dimensions of evaluation

One of the main and earliest operation in evaluation is to choose criteria. But before making such a choice the specific guide’s author needs to determine which dimensions he wishes to assess. A distinction needs indeed to be made between the dimensions of evaluation and the criteria used to express such dimensions.

The Webster Dictionary provides a long list of definitions of the term “dimension”. According to the specific situation one of the following definitions may more conveniently be applied :

A dimension can thus be expressed using one or more criteria. Good examples of how a variety of criteria can be used to express a single dimension are “scientific quality” or “valorisation”.

In the case of evaluation of development research, besides the two examples just given, dimensions such as “relevance for development”, “performance”, and “impact” are widely used. More dimensions are listed in "dimensions of evaluation; criteria".

Step 6. Distribute responsibilities among the specific guide’s authors

A guide will seldom be built by an individual, and it more generally will be the work of a team. Various decisions will be made about the sharing of responsibilities and tasks among the authors of the evaluation guide : committee work, individual contributions, call to external referees, etc. Also, at this stage, the bases should be established for reaching a consensus. The stage is then set for starting to write the first version of the specific guide.