Document(s) 4 of 4

International Development Research Centre
February, 2000

Table of Contents

1. Overview

The Centre maintains a decentralized evaluation system. It places the locus of responsibility for different evaluation activities where the information is most relevant. While the Centre carries out some evaluations to document corporate performance for accountability purposes, the primary purpose of evaluation is for corporate learning and to contribute to the body of knowledge on what works in development research.

At the project and program levels, responsibility for evaluation rests with the programs. The Regional Offices are responsible for regional level evaluation work; they also provide input to project, program, corporate and strategic evaluations which affect their regions. At the corporate and strategic levels, responsibility for evaluation rests with the Evaluation Unit. In addition to this evaluation work, the Unit is responsible for

• the development of tools and methods appropriate to the evaluation of development research;
• providing advice and guidance in evaluation to IDRC programs and projects;
• evaluation capacity building within IDRC and recipient institutions; and
• the maintenance of evaluation information systems.

Annex I to this report presents an overview of the evaluation system at the Centre.

As part of developing its programming strategy for the next five years, IDRC's Evaluation Unit has reviewed past performance, surveyed evaluation users, and conducted strategic planning sessions. The resulting strategy, presented in this paper, retains the original four objectives of IDRC's evaluation system, updates its guiding principles based on lessons learned, and responds to needs and opportunities which have emerged from its four immediate client groups: senior managers, IDRC programs, Secretariats and special projects, Southern partners, and other donors. It also reflects views expressed by its ultimate client, the Board of Governors, during discussions of the Annual Corporate Evaluation Report.

In the new strategy, evaluation continues as an integral part of good program management. Corporate learning and improvement will continue to drive all evaluation activities with collegial participation by stakeholders as a key ingredient. The decentralization of evaluation will be maintained, with the programs and Secretariats responsible for reporting on project and program results, and the Evaluation Unit responsible for overall coordination, technical support, strategic evaluations and reporting on corporate performance. New elements in the 2000-2005 strategy include: strengthening incentives for doing and using evaluations in IDRC; implementation of a framework for Secretariats to use in assessing and reporting on their performance to multiple donors; and increasing ownership of and benefits from evaluation for Southern institutions. A more clearly defined approach to resource expansion is also included. As with any strategy, oversight and interest by the Board, support by senior management, staff commitment and the resources allocated will be the primary determinants of its success.

2. Review of Performance: 1992 - 1999

In December 1992, the Evaluation Unit's objectives and strategy were approved by senior management and the Board of Governors. Listed below are examples of the outputs achieved, over the past seven years, grouped by objective.

Objective 1: develop and coordinate IDRC's evaluation activities and systems in order to improve the efficiency and effectiveness of the Centre.

• Strategic studies: some of the topics covered include project size/efficiency, cooperative projects, participatory research, development networks, project leader capacity building, institutional capacity building at the Navrongo Health Research Centre, ICTs, the specifics of IDRC's funding approach, and the Secretariat modality.
• The Unit provided assistance to Program Initiatives with their evaluation planning and reporting requirements through advice, technical support and a program monitoring and evaluation manual. This assistance included testing innovative approaches, commissioning evaluations, and conducting or participating in project evaluations.
• The Annual Corporate Evaluation Report (ACE) has been presented to the Board of Governors each year since 1994, making it an instrument for raising awareness of, and fostering action on, key evaluation findings at all levels of the organization.

Objective 2: conduct and support research to improve evaluation tools and methods in order to strengthen the use of evaluation for development research.

• The Unit has kept abreast of state-of-the-art evaluation approaches and methodology development focussed on four relevant methodological areas: assessing progress towards sustainability⁽¹⁾; institutional assessment; promoting evaluation as a learning mechanism; and, assessing the impact of development research. All of these initiatives have resulted in widely distributed products, and partnerships with other organizations including: the World Conservation Union (IUCN); the International Service for National Agricultural Research (ISNAR); Performance Measurement Network; Universalia Management Group; the Pacific Institute for Research and Evaluation; the Centro Internacional de Agricultura Tropical (CIAT); and the International Centre for Research in Agroforestry (ICRAF).

Objective 3: strengthen the capacity for evaluation in order to improve the quality and management of research.

• Secondment placements for HO staff were organized in 1993-1994 and for regional staff in 1998-1999. A variety of training activities and technical support have promoted the development of evaluation expertise within the Centre.
• IDRC's Southern partners have been drawn into consultations and co-learning approaches. By building evaluation into different social, cultural, and economic contexts and advocating with other agencies we have begun to make evaluations more relevant and useful to recipient organizations.

Objective 4: collect, analyse, and disseminate evaluation information in order to promote its use for decision making by the Centre, its recipient institutions, and other agencies concerned with research for development.

• The Unit maintains and updates the Centre's Evaluation Information System (EVIS). The Project Completion Reporting (PCRs) system was reorganized in order to make the reports more useful and the system more "user friendly" for Centre staff. The Evaluation Inventory currently contains 420 evaluation reports and the Evaluation Unit launched its web-page in June 1998 to publicize its work and make reports more widely available.

Details on all the activities and achievements within each of these four areas are available from the Evaluation Unit.

3. Guiding Principles for Evaluation

The four objectives above cover the core services required by our clients: the Board of Governors, senior management, program and Secretariat staff, and IDRC partner institutions and researchers. Inherently, evaluation has a dual function; it serves both learning and accountability. IDRC has chosen to use evaluation first as a corporate learning tool, believing this to be the best way to strengthen its accountability function. The following principles guide evaluation at IDRC:

• Evaluation is intended to improve program planning and delivery. It contributes to decision-making and strategy formulation at all levels in the Centre. To aim evaluations at providing useful findings, projects and programs are assessed strategically, based on the client's purpose and information needs.
• Evaluations are designed to lead to action. To be useful, evaluations need to produce relevant, action-oriented findings and this is fostered by sustained involvement and ownership by the client and stakeholders throughout the process.
• No single, best, generic evaluation methodology exists. Each case usually requires tools and methods appropriate to the data that is to be gathered and analysed, and appropriate to the client's needs. Credible evaluations interlace quantitative and qualitative data from several sources.
• Evaluations should enlist the participation of relevant stakeholders. Those affected by the outcome of an evaluation have a right to be involved in the process. Their participation will make the evaluation's purpose and process better understood, promote stakeholder contribution to, and acceptance of, the evaluation results, and will increase the likelihood that the evaluation findings will be utilized.
• Monitoring and evaluation planning add value at the design stage of a project or program. They can make the project or program more efficient and effective by helping clarify the results to be achieved. Also, knowing what information will be necessary to evaluate the outcomes will allow for it to be collected as it becomes available. This will reduce the financial and human resources required and improve the project or program team's ability to report on, and learn from, its experiences.
• Evaluation should be an asset for those being evaluated. Evaluation can impose a considerable time and resource burden on recipient institutions. IDRC evaluations should also generate information that can benefit the recipient institution.
• Evaluation is both science and art. The art of identifying critical issues to be evaluated, organizing them conceptually, and bringing the appropriate people to participate in the collection, interpretation, and utilisation of the evaluation information is as important as the systematic collection and analysis of reliable data.
• Evaluations are a means of negotiating different realities. Evaluations provide opportunities for project or program stakeholders to reconcile their various perspectives and/or versions of reality.
• Evaluations should leave behind an increased capacity for using evaluation findings. Organizations need some level of internal evaluation capacity in order to be able to devise, participate in, or utilize evaluations effectively. Exclusive reliance on external expertise can limit an organization's ability to be clear and specific about its goals and to learn and apply lessons. Specific strategies can be built into evaluations which are explicitly aimed at fostering these organizational characteristics.

4. Developing a New Strategy

4.1. Client Survey

In July and August 1998, as the Evaluation Unit began to formulate a new strategy, the team conducted interviews with 21 IDRC staff members from various levels within the organization in Ottawa and in Regional Offices to assess past performance and obtain guidance for future directions. The results of this survey are presented under the following three headings.

4.1.1 Quality, Availability, and Relevance of Past Performance

The majority of the comments on the Evaluation Unit's past performance were positive. According to the interviewees, the Evaluation Unit's good reputation is based on its responsiveness and flexibility, facilitation of an honest evaluation culture within the Centre, and promotion of IDRC's international reputation. There were, however, several comments that the Evaluation Unit needs to get more involved in expanding the resources available for evaluation activities.

4.1.2 Current Needs and Improvements to Services Provided

Helping Program Initiatives with their evaluation planning and facilitating group sessions were identified as useful services the Evaluation Unit provides, but the interviewees also identified a few areas that need improvement. These include: being more pro-active in disseminating information and publicising evaluation activities and results; providing Program Initiatives more evaluation support; greater cooperation with Internal Audit, Publications, the Executive Office, and Policy and Planning; and, assisting Secretariats in developing and implementing monitoring and evaluation plans for their multiple donors. Also, few of the interviewees knew the variety of services and products the Evaluation Unit offers and some interviewees were unsure about how to approach the Unit for assistance.

4.1.3 Future Roles and Influencing Factors.

Rather than imagining entirely new roles for the Evaluation Unit, the interviewees recommended that current activities be maintained and expanded. They suggested staff seminars, training opportunities, and information sessions that the Evaluation Unit could arrange. A variety of issues were also identified for future study: the impact of development research; inter/multi-disciplinarity; Program Initiatives as a program delivery mechanism; and, the Centre's progress in achieving its global development objectives.

4.2. Strategic Planning Workshop

The information gathered in the interviews was used during a two-day strategic planning workshop held in October 1998. This participatory process provided the Evaluation Unit and a staff member from each of Programs Branch, a Secretariat, and Policy and Planning Group with an opportunity to create a common vision for evaluation in IDRC to guide future work and to examine the obstacles impeding the achievement of that vision. The workshop and survey results along with lessons learned over the past eight years form the basis for the Unit's new evaluation strategy.

5. Evaluation Strategy: Mapping Outcomes

In response to the problems in measuring impact, the Evaluation Unit has been developing a new approach to planning, monitoring and evaluating development research, Outcome Mapping (see the Annual Corporate Evaluation Report 1999, p.14 - Annex II). Outcome Mapping is an approach to think holistically about how a program intends to advance its goals. It focuses on measuring the changes in behaviours, relationships, or activities that the program has helped to bring about. Rather than attempting to measure ultimate development impact (which is beyond the sphere of influence of the program), it characterizes and assesses the contributions a program has made to the problématique it is addressing. The Unit has defined its new strategy using Outcome Mapping. This approach is based on the definition of vision and mission statements and outcome challenge statements against which progress can be measured. Outcome Mapping requires a focus on the individuals or organizations with whom one interacts directly, and assesses contributions to changes in their behaviours. These "boundary partners" are the groups one can influence directly, not necessarily the ultimate beneficiaries of the activities.

The vision statement below characterizes how evaluation practice in the Centre should ideally look in five years. The mission statement indicates the broad activity areas in which the Unit will work in order to operationalize its contribution to the vision. The outcome challenges and progress markers indicate the type and level of change the Unit will encourage in its partners to contribute to the vision.

5.1. Vision

Development research supported by IDRC is highly valued by its Southern partners for its diverse contributions to critical local understanding and capacities, shared insights, timely actions, and desired local, regional, and global impacts. A recognized key to this success is the integration of informative and provocative feedback mechanisms within all IDRC-supported programs and projects. These mechanisms ensure that researchers and research users remain focused on outcomes of consequence, devise and revise strategies and action plans that maximize the likelihood of these outcomes being reached, act with fidelity in accordance with their plans, and freely share their experiences and discoveries with IDRC and others who might benefit. Further, parallel feedback mechanisms integrated at all levels within the decision-making processes of IDRC ensure that funding and technical assistance are directed where they are best aligned with corporate objectives, most needed, and can achieve the most good.

5.2. Mission

In support of this vision, the Evaluation Unit promotes the use of state-of-the-art evaluation and management methodologies and practices throughout IDRC and with its Southern partners. The Unit fosters and helps plan the flow of performance-based knowledge to inform and contribute to strategic decisions, strong project designs, effective research utilization, cohesive programs, and significant advances in development theory and practice. We promote the compilation, interpretation, and dissemination of outcome-oriented performance data critical to project- and program-level understanding and refinement. We are a respected, oft-cited voice for change and excellence in donor evaluation policies and practices and a champion for the effective use of evaluation and management tools in the South. In sum, the Evaluation Unit is a potent and frequently tapped IDRC resource that is highly valued throughout the organization and by its key partners around the globe

5.3. Outcome Challenges and Progress Markers

Although the Evaluation Unit will define and guide the process, our vision requires involvement, support, and change in our four primary boundary partners (our immediate clients):

• senior managers;
• IDRC Program, Secretariat, and Special Project staff;
• Southern partner institutions and affiliated researchers; and
• other donors.

In order to understand the implications of our mission and establish means of monitoring progress, the Evaluation Unit has set the following outcomes challenges and graduated progress markers for each of our four primary boundary partners. The outcome challenges describe the change we will try to help bring about in the behaviour, relationships, activities, and/or actions of our boundary partners. The outcome challenges are specifically formulated to challenge the Unit to encourage the deepest level of change possible so that our partners can contribute to the broader vision. The progress markers are graduated and their degree of difficulty moves from what the Unit would expect to see its boundary partner doing if it carries out its basic activities well (expect-to-see markers), to achieving deeper and more difficult changes (like-to-see and love-to-see markers). They represent the information the Unit will gather in order to improve our performance and be able to monitor achievements in the direction of the outcomes by our partners.

5.3.1 Outcome Challenges and Progress Markers: #1 IDRC Senior Managers

Outcome Challenge 1: The Evaluation Unit intends to see IDRC senior managers who routinely demand and wholeheartedly support the adoption of evaluation processes and the generation and use of outcome-based performance data, and who themselves use these data and related feedback with confidence to expand their viewpoints and inform their decisions.

Graduated Progress Markers

Expect to see senior managers who:

• Support continued existence of the Evaluation Unit
• Hold staff accountable for setting evaluation targets
• Understand and share the principles underlying the Unit's approach to evaluation
• Integrate monitoring and evaluation requirements into programming cycle

Like to see senior managers who:

• Suggest strategic evaluation topics/issues to be studied
• Respond favourably to the Unit's requests for resources to meet special demands
• Quickly recognize and act to remove IDRC barrier(s) to evaluation
• Demand high-quality, outcome-focussed evaluation results from programs
• Foster the viewpoint that learning derives from both successes and failures
• Use evaluation data to inform a key decision or decisions
• Set their own outcome-based performance criteria
• Endorse a set of corporate performance targets

Love to see senior managers who:

• Regularly measure corporate performance targets
• Establish a culture that rewards staff for treating evaluation as integral to their programs
• Systematically use evaluation data to formulate strategies and guide programming decisions

5.3.2 Outcome Challenges and Progress Markers: #2 IDRC Staff

Outcome Challenge 2: The Evaluation Unit intends to see IDRC staff (e.g., those from Program Initiatives, Secretariats, and special projects) who design, launch, and implement high-quality evaluations, who generate and make effective use of outcome-based performance data in self-assessments and in their work with program participants, who take actions to further increase their understanding of performance assessment and the role it can play in support and promotion of their work, and who share their results and learning with others in IDRC and beyond.

Graduated Progress Markers

Expect to see program staff who:

• Are regularly exposed to what the Unit is and does
• Are exposed to information about how to make contact with the Unit
• Are made aware of the Unit's evaluation guidelines
• Provide the Unit with evaluation reports on a routine basis

Like to see program staff who:

• Contact the Unit for evaluation resource materials and/or advice
• Participate in capacity building events
• Use the Unit's guidelines to inform their evaluation approach and design
• Consult with the Unit on an evaluation design and/or implementation
• Draw in the Unit to participate in an evaluation activity
• Indicate that evaluation feedback improves their work
• Integrate evaluation resources into projects for learning
• Support their partners' use of evaluation

Love to see program staff who:

• Use evaluation to transform the approach and content of programs
• Make important suggestions and/or contributions to IDRC's evaluation approach
• Mentor other staff and/or partners in evaluation

5.3.3 Outcome Challenges and Progress Markers: #3 Southern Partners

Outcome Challenge 3: The Evaluation Unit intends to see Southern partner institutions and affiliated researchers that share IDRC's enthusiasm and commitment to outcome-oriented, participatory, and learning-based evaluation, that exhibit mastery in their own use of monitoring and evaluation, which ensure that external evaluations add value to their work, that can serve with competence as external evaluators and as providers of training and technical assistance in monitoring and evaluation, and that join with IDRC and others to support and promote advances in the science and art of performance assessment.

Graduated Progress Markers

Expect to see Southern partners who:

• Are exposed to the Unit's evaluation approach, publications, or resources
• Contact IDRC staff or the Unit to obtain evaluation information or resource materials
• Meet with or contact IDRC staff or the Unit to discuss evaluation issues
• Contact IDRC staff or the Unit to express an interest in collaborating on evaluation

Like to see Southern partners who:

• Participate in capacity building events
• Invite IDRC staff or the Unit to participate in events focused on evaluation
• Participate in events to increase relevance of evaluation to Southern partners
• Develop internal evaluation talent and expertise
• Demonstrate an ability to design and implement a quality evaluation
• Routinely compile evaluation findings to inform and improve their work
• Indicate that evaluation feedback improves their work
• Jointly publish articles or books on evaluation with IDRC

Love to see Southern partners who:

• Become regional nodes for professional development in evaluation
• Develop long-term, creative partnerships around evaluation with IDRC
• Provide critical and insightful feedback on donor performance

5.3.4 Outcome Challenges and Progress Markers: #4 Other Donor Agencies

Outcome Challenge 4: The Evaluation Unit intends to see other donor agencies that share IDRC's enthusiasm and commitment to outcome-oriented, participatory, and learning-based evaluation, that routinely compile and draw upon evaluation findings based on these principles to inform program improvement efforts, and that join with IDRC and others to support and promote advances in the science and art of performance assessment.

Graduated Progress Markers

Expect to see other donor agencies who:

• Are exposed to IDRC's Evaluation Unit and its approach
• Contact the Unit to request information
• Meet with the Unit to discuss evaluation issues (virtual or face-to-face)
• Invite the Unit staff to engage in events focused on evaluation

Like to see donor agencies who:

• Seek the Unit's advice and guidance on evaluation issues in areas of IDRC expertise
• Invite the Unit to conduct evaluations for them in areas with potential for new learning for IDRC
• Jointly publish articles or books on evaluation with IDRC
• Participate in discussions on ways to increase relevance of evaluation to Southern partners
• Sponsor evaluations that meet information needs of both donors and partners
• Rely heavily on Southern evaluation talent and expertise
• Jointly support professional development opportunities in evaluation for Southern partners
• Seek out feedback on performance from current or previous IDRC Southern partners

Love to see donor agencies who:

• Develop long-term, creative partnerships with the Unit
• Place trust in Southern partners' assessments of their own performance
• Place high value on Southern partners' assessments of donor performance

6. Activities

The work envisaged in each of these areas is outlined below.

6.1. Senior Managers

Senior management is supportive of evaluation and has asked the Evaluation Unit to analyse the management system itself and to advise action on any elements which impede evaluation. In order to adjust IDRC's management system so that it better promotes the use of evaluation at all levels in the organization and permits the Centre to report on its corporate performance, the Evaluation Unit will focus its work with senior managers in two areas: 1) the development of corporate performance targets; 2) and the establishment of a system of rewards and incentives for good evaluation work by Centre staff.

6.2. Program Initiative, Secretariat, and Special Project Staff

The path to increasing Program Initiatives' expertise in, and use of, monitoring and evaluation in managing and reporting on their performance is obstructed by a few factors that must be addressed. These include the time and financial constraints of Program Initiatives for evaluation; the lack of incentives and rewards for evaluation within the Centre; an incomplete understanding of evaluation concepts, processes, and usefulness in their work among program staff; and the limited human resources in the Evaluation Unit to address all these, compared with the number of Program Initiatives.

In order to overcome these obstacles, the Evaluation Unit will undertake two types of activities: some will be aimed generically at all program staff; others will partner the Evaluation Unit with individual Program Initiatives directly. Work with Program Initiatives will build on work carried out over the course of CPF 2.

Building on the information gathered during the preparation of Learning Partnerships: A Review of IDRC Secretariats, and the IDRC response, The Strategic Framework for Secretariats at IDRC, the Evaluation Unit will focus more attention on developing evaluation capacity within Secretariats. The objective will be to develop and implement a system by which Secretariats can evaluate and report on their performance to their multiple donors. As with Program Initiatives, some of the Evaluation Unit's activities will be geared generally towards all Secretariats while others will involve partnering with individual Secretariats.

To compensate for limited staff resources, the Unit will build a pool of external consultants to draw on. It will also focus on efforts to train evaluation resource people in teams and Secretariats.

6.3. Southern Partner Institutions

Efforts over the past five years to strengthen the evaluation culture in the Centre have not resulted in a stronger evaluation capacity in our partner institutions. In its work to strengthen the research organizations and institutions with which the Centre works, the Evaluation Unit has experienced strong demand for strategies which create ownership of the evaluation process by our partners, and which foster its use as an organization building and management tool. The long term purpose is to extend the value gained from evaluation beyond the donor community to include our Southern partners.

This is an expanded area of activity for the Evaluation Unit over the next five years. Activities will focus on increasing Southern partner institutions' involvement in, and ownership of, monitoring and evaluation. Realistically, IDRC will be a small player in this area because activities of this type are expensive, and we can offer limited incentives to participate. Nonetheless, it is an important endeavour and the Evaluation Unit will focus on working with a small number of IDRC partners to develop regional expertise in monitoring and evaluation. We will also try to act as a catalyst among other donor organizations to build interest in pursuing this goal.

6.4. Other Donors

Over the past eight years the Unit has maintained contact with a wide range of donors and monitors the evaluation priorities of many of these. Over the next five years emphasis will be placed on working with those donors active in co-funding Secretariats and other Centre activities.

In addition, in support of our intent to build evaluation capacity with the Centre's partners in the South, we will give priority to building working relationships with donors who are building capacity for learning-based evaluation with partners in the South.

7. Resources

7.1. Resources Required

If successful, this strategy will put IDRC in the forefront among organizations having integrated systematic assessment within their management and reporting mechanisms. This will enable well-documented learning and performance improvement at the program and corporate levels, enhancing both program effectiveness and accountability. IDRC would also be modelling the behaviour it desires among its partners and recipients. The strategy has the potential to begin a radical shift in the evaluation practices of development agencies in two ways: first, in the way external donors define and measure results; and, second, in the degree of control and benefit gained by Southern institutions through the evaluation of development aid.

While the financial resources set out in the revised 2000-2001 Program of Work & Budget are adequate to initiate the 2000-2005 strategy, the complement of permanent staff is not adequate to fully carry it through to 2005. Over the past 6 years the benchmark percentage of program budget allocated to evaluation averages around 2.5%. Of this amount, 40% is spent by the Evaluation Unit and 60% by the Programs themselves. To carry out its functions of technical support, strategic evaluation, system maintenance and tool development under the Centre's established evaluation approach, the Evaluation Unit has successfully worked with the equivalent of about .5 percent of total appropriations in permanent staff costs - or approximately one person year per 20 million in appropriation funds. While, for the most part, the 2000-2005 strategy would continue with the same resource requirements, the increased Southern focus and the expansion of work to include Secretariats will create demands on staff time and travel which may require revisions to this formula.

Given our limited experience in working in the complex multi-donor context of the Secretariats, our established benchmarks in this area may have to be revised.

7.2. Resource Expansion

Two paths to resource expansion appear promising and we will pursue both of these:

1. partnerships addressing interests held jointly with other organizations. Our work with other donors will be based on the principles of partnership and co-funding in areas of mutual interest, such as evaluation capacity building, outcome assessment and development research evaluation; and
   
2. the sale of publications and training materials based on work done to serve IDRC's needs. For the organizational assessment materials we have experienced continued demand both within and outside IDRC (1,050 copies of Enhancing Organizational Performance sold to date). Continued work in the organizational assessment area will be complemented by publications and training materials in Outcome Mapping over the next several years.

Apart from these two foci, we will limit efforts in other areas. Experience over the past 4 years suggests that contracting out evaluation services drains resources from the Centre, rather than draws resources to us. The level of effort required to produce high quality evaluation findings in unfamiliar contexts, combined with the resources needed to seek and respond to contractual opportunities (many of which do not bear fruit), add up to a level of expenditure which we have not been able to recoup through competitive contract pricing. The net effect tends to be a reduction in resources available for IDRC-centred work. Therefore, over the next 5 years, the Evaluation Unit will engage in contract evaluations only under those exceptional arrangements which guarantee to avoid the above risks.

ANNEX I

ANNEX II

The Evaluation Unit and several PIs have been working with Dr. Barry Kibel, Pacific Institute for Research and Evaluation, to adapt his Outcome Engineering approach to the development research context. The result is a methodology that characterizes and assesses the contributions a project/program makes to the achievement of outcomes. Outcomes are defined as changes in behaviour, relationships, activities, and/or actions that the project/program has influenced. These changes should contribute to improvements in people or societies' well-being.

Figure 1 shows the categories of actors among whom IDRC tries to encourage outcomes. In seeking to change the way certain development actors act, IDRC and its partners enter into a wide variety of activities and relationships. The resulting changes are dialogical and non-linear. Actions and reactions go in both directions, since in each situation there may be cooperation, resistance or negotiation, resulting in mutual influence and learning. Outcomes can be measured at all points where changes in actors occur or were intended to occur.

Development impacts are shown separately in the diagram as they act as the beacon which guides action, and against which the outcomes can be assessed, but for which no development agency can be accountable. Impacts cannot be achieved by any single actor; they result when a sufficient quantity and quality of outcomes and other intervening variables come together.

This approach provides a method to help a project/program identify and evaluate the specific mix of strategies it uses to achieve its desired outcomes. It involves a sequence of steps which include: a) identifying the key actors; b) characterizing the strategies aimed at each actor; and, c) applying the data collection instruments appropriate to each strategy. Focussing on the strategies allows a project/program to measure the results it achieves within its sphere of influence and to take credit for its contributions. At the same time, however, it recognises that it is ultimately the combined activities, thoughts, and behaviours of individuals, groups, and organizations that will create and sustain development impacts. Outcome Mapping, as we are tentatively calling this approach, also provides a project/program with a system to think holistically and strategically about how it intends to achieve its goals. Activities are underway to test and refine this evaluation approach by applying it in IDRC projects, programs, networks, and with partner institutions.

From IDRC Support to System-Level Impacts by Barry Kibel (1999).

Source: Annual Corporate Evaluation Report 1999, page 14.

1. Sustainability refers here to the preservation and enhancement of resources for future generations. The assessment work was carried out through IUCN, the World Conservation Union, in a multi-year initiative first presented to the Board in Montreal in October, 1996. Phase II of this initiative was managed by Programs Branch and was completed in January 2000.

Document(s) 4 of 4