Document(s) 4 of 10

As early as possible in the evaluation process, each study should include a clearly defined exercise to “scope” the dimensions of the evaluation. This scoping exercise should result in evaluation objectives and criteria agreed on by the stakeholders, an identification of the necessary activities, and a workplan and budget. These constitute the initial evaluation plan. The evaluation plan will include both the analytic framework for the evaluation and an implementation workplan. One important component of the evaluation planning process is a multistakeholder process designed to enable telecentre stakeholders and those responsible for the evaluation to arrive at a shared understanding of the overall objectives of the evaluation and how they will be achieved in the different evaluation and monitoring activities. The multistakeholder process may necessitate meetings and other forms of discussion at local, national, regional, and international levels, depending on the context of the telecentre projects. Most important, at the outset of the evaluation study, the various stakeholders will discuss the procedures for reporting, interpreting, and disseminating the results.

2.1 Identifying the research questions

The logical place to begin an evaluation plan is with the questions you want to answer. You can pose two types of question (the following questions are given as examples):

Questions immediately related to the evaluation — What size of population is needed to make the telecentre financially feasible?

More fundamental research questions — What is the role of information and communication in development? How do the impacts of information technologies differ from those of other technological innovations introduced into African communities? Under what models of social change and economic development is the evaluation being conducted? What are our assumptions about how information technology and content change political power relationships, social learning, or economic benefit?

These questions may seem too theoretical and far removed from the immediate concerns of the program manager, who wants to know how many people a telecentre adequately serves and how often the equipment is breaking down. However, these research questions are fundamental to the evaluator’s efforts to frame hypotheses for an evaluation study. Too often, projects are designed and implemented without adequate attention being given to the scientific models or theories underlying them. Consideration of these underlying models early on in the evaluation planning process is important for two reasons:

Evaluators bring their own biases — The evaluation team’s assumptions about how community telecentres change people’s lives will influence the questions the team asks and how it asks them. The team members should identify their own assumptions about the role of information in social change and how community development takes place, not only to identify the key research questions for the study but also to better recognize their own biases.

Research methods are not value free — Evaluation designs and methods are each embedded in particular research paradigms, and the evaluation approach should reflect a scientific model of the phenomenon under study (in this case, the impact of information technology on individuals and communities). The broader research questions are fundamental to IDRC’s rationale for launching the Acacia Initiative and thus become doubly important in evaluating the program itself, in addition to its individual projects.

What social-research questions might frame an evaluation of a telecentre project?

Some of the most important concern the social role of information. Communication can be defined as “the exchange of information and the transmission of meaning (which lead to action).” Katz and Kahn (1978) argued that it is the essence of any social system or organization. Communication is a social process fundamental to any group’s functions: it is the means of motivating, influencing, and restricting social interaction, such as cooperation, conflict, and leadership. Communication is organized information flow, ordered as much by limitations and restrictions as by openness.

The proper understanding of communication situates it in the context of a social system. Ashby (1952) conceptualized social systems as restricted information networks. Thus, the introduction of a community-access telecentre, if it is successful, is going to have a major impact on the community — its culture, communication patterns, economy, social structure, and future development — and the community will, in turn, determine the telecentre’s sustainability.

Models of learning and innovation assume a pivotal role for information transmission and exchange. Such models stress that information without context is so much useless noise, and the lesson here is not only that community telecentres are social forces impacting on the community but also that they cannot succeed unless implementers give due attention to the types, sources, and quality of their information and the relevant applications, such as health care and education. Experience suggests that if the telecentres are seen as technology providers, rather than as social and cultural community centres, they will be less sustainable and provide fewer benefits.

Information has been described by some telecommunications enthusiasts as intrinsically unlike other resources, such as energy or water: rather than being depleted, information increases in value when people use and share it. This might indicate that information has no cost, but of course, access to information does have a cost, both for the individual and for the community, as does the provision of other basic human needs. Is greater access to information always a benefit? Probably not. When does increased information bring negative impacts, as well as positive ones? These are the kinds of question that research projects like the Acacia Initiative are designed to address (Table 1).

Information can come from within or outside the community, and the balance of the two sources of information creates community structure and development. Telecentres affect this balance between endogenous and exogenous information and the relationship between its various suppliers. Control of information has long been central to maintaining political power and economic advantage in many societies. Western ideas about patenting knowledge clash head on in Africa and elsewhere with traditions of free reciprocal exchange of valuable knowledge, such as knowledge about the use of plants. How does the social construction of information (and the Internet) affect the ways people view telecentres and the ways telecentres will change the social exchange of information within a community and between it and the outside world?

Are information technologies, such as telecentres, intrinsically different from other types of technological interventions that we associate with development, such as water pumps or roads? Can we apply our long experience with these interventions to the telecentre programs? Is it more useful to look at the experience of community schools and libraries, on grounds that the relevant lessons for telecentres are more related to social programs than technological innovation?

How should research and evaluation of telecentres distinguish information, communication, and knowledge in practical terms? Knowledge is information meaningfully structured into concepts and facts to achieve some end. This implies the need to examine the use and impact of information within a specific knowledge context to judge whether it is relevant, timely, understandable, and of practical benefit to the user. At one level, the assessment of telecentres is inseparable from the information and knowledge that flow through the telecentres and the resulting behaviour and effects. At another level, telecentres are service centres and places of social interaction. Because telecentres generate new knowledge, learning, and patterns of behaviour, we must evaluate them not only as a new technology, but also as a set of social processes mediated through a technology.

2.2 Systems approaches to evaluation

Program evaluation as a distinct field of professional practice was born of two lessons. … First, the realization that there is not enough money to do all the things that need doing; and second, even if there were enough money, it takes more than money to solve complex human and social problems. As not everything can be done, there must be a basis for deciding which things are worth doing. Enter evaluation.

— Patton (1997, p. 11)

Program evaluation developed in the 1960s to meet the need to prove the value of publicly funded social projects. The 1960s also saw the rise of the scientific method in the social sciences and an emphasis on experimental design, statistical significance, and identification of causes. Although evaluation methods have evolved considerably since then, they are still heavily influenced by the need to measure performance for accountability purposes. Concern with accountability focuses partly on returns on financial and other investments in a program and partly on its initial objectives and the success of program management in obtaining these objectives. Evaluations undertaken in this mould do not deal very well with a program that learns as it goes along and changes its objectives and activities midstream. Nor are they particularly appropriate for programs that are so successful that they result in major transformations in the project communities. Most evaluations are designed to measure incremental changes along predicted (outcome) trajectories. In other words, traditional evaluation models do not necessarily deal well with adaptive, complex systems, which is what human communities and social-information systems are.

An important lesson from general systems theory is that one of the first steps in designing an evaluation project is to map out, or model, the system under evaluation: its components, connectivity, and feedback loops; boundaries; inputs, throughputs, and outputs; behaviour; and critical thresholds. The next lesson, this one provided by complex systems theory, is that the general systems model is a gross oversimplification of reality, as human systems are never linear but develop with a good measure of surprises and uncertainty. A human system is an adaptive, or learning, system. When the process of change reaches some critical state, an apparently linear change can suddenly lead to a dramatic development or reversal of a characteristic (as described in catastrophe theory, or “the straw that broke the camel’s back”). The third lesson for evaluation, this one taken from the history of science, is that science, and thus the evaluation, is contextual and value laden, carrying the burden of the values embedded in the theories, methods, and value systems of the evaluation team.

What are the implications for the telecentre evaluation plan?

First, the evaluation should make its assumptions explicit, and the evaluation team should be prepared to have these assumptions challenged. In fact, the team should even encourage this. Second, although limitations and biases are unavoidable, evaluators can compensate for them by involving as many diverse stakeholders as possible. The evaluation should include a stakeholder consultation process. Third, because each method and instrument adopted in an evaluation study originally stems from a particular scientific paradigm, the evaluation team needs to reflect on these assumptions to determine whether they are consistent with its overall approach. Fourth, the design of the evaluation study should not be overly rigid but should be open to new discoveries and pose new questions, even after the project is under way. In other words, the evaluation study itself should be an adaptive learning system. Fifth, the evaluation study should include diverse methods and research instruments to capture diverse types of information.

Discussions such as these are particularly important in the evaluation of telecentre projects because telecentres change the patterns of information and communication in communities, and these communities are inherently adaptive, complex systems; projects implemented in these systems are likely to have unexpected and decidedly stochastic outcomes. It will be surprising if there are not surprises in the assessment of community telecentres. But the large financial and political investments in community-telecentre programs will tend to drive the focus of the evaluation toward the shorter term objective of accountability and away from “fuzzy” research questions. Research and evaluation teams should take the time to achieve a balance between the two and include some discussion with stakeholders about longer term research questions and alternative models of community development.

The key elements of the Acacia adaptive-systems approach are as follows:

Develop a systems model for the program or project to be evaluated to scope the evaluation;

Identify the stakeholders in the project and their information needs;

Test your scoping model with various stakeholders to develop a working model for the evaluation;

Design the evaluation or research study to be adaptive to new information and diverse information needs, and adapt your working model;

Include a variety of methods and research instruments to obtain diverse types of qualitative and quantitative data;

Develop a workplan to achieve the goals of the evaluation within the available budget and time frame;

Be as participatory as possible in conducting the research and evaluation by including representatives of the various stakeholder groups in data collection and interpretation and using self-assessment as part of the evaluation tool kit;

Select an evaluation and research team with credibility among the various stakeholders and with both internal and external expertise; and

Ensure that the evaluation team and stakeholders interact throughout the evaluation or research study and that the evaluation itself contributes to learning and adaptation at all levels.

2.3 Identifying the stakeholders

Projects and their evaluations have multiple stakeholders. The evaluation plan should identify who they are and, if possible, “map” how they are either separate or clustered together in groups by their interests (stakes) in the project. This may simplify the otherwise daunting task of dealing with large numbers of stakeholders, some of whom may be relatively peripheral to the evaluation. Table 2 gives examples of potential stakeholder groups in a community-telecentre project. The evaluation team can identify the stakeholders as part of its exercise in mapping the project system. Once the team has identified the main stakeholders, it can use a “snowball” strategy, asking each stakeholder to identify others potentially interested in the project or evaluation, until the evaluators are reasonably confident of having included all key stakeholders.

What might the stakeholders want to know?

The snowball strategy also enables the evaluation team to identify what the stakeholders expect from the evaluation study. This information will provide an input to the study design and allow the team to map the various stakeholders’ interests. Those involved in operating the telecentre are likely to be particularly interested in its financial sustainability, and they will need detailed quantitative information on revenues and earnings in the context of the population served (see Table 2). Stakeholders at the national level, such as ministries, may want information about the impacts of telecentres on their programs, such as youth employment or improved medical service, or about the demand that telecentres create for government information online or on call. Stakeholders in the international-donor group may have a specific interest in information relating to the goals of their own programs, such as the impact of telecentres on women’s political participation or the operations of local nongovernmental organizations (NGOs). Stakeholders at the national, regional, and international levels will likely be more interested in comparisons across telecentre projects and more generally valid indicators of financial sustainability and community impact. These indicators may only be determined in the light of comparisons across various telecentre evaluation studies.

The evaluation plan should include a consultative process with a representative and manageable subset of stakeholders. This process may include multistakeholder consultations or smaller meetings with individual stakeholder groups or leaders. The evaluation team should be confident that it has identified the stakeholders, their information needs, and their expectations for the evaluation and that the stakeholders are regularly informed about the progress of the evaluation and its findings. Some stakeholders are likely to have unrealistic expectations for the evaluation study, and it is better to work this through with them before the final report is in their hands. An effective way to achieve several of these objectives is to allow appropriate individuals from some stakeholder groups to participate on the evaluation team and in data collection.

Stakeholders sometimes have conflicting interests in a telecentre project. Managing multistakeholder processes may demand mediation, negotiation, and conflict-resolution skills and almost always requires patience. The evaluation team will have to decide how best to design the stakeholder consultative process, keeping in mind the particular circumstances and personalities involved in the process. Sometimes a multilevel approach is the most cost-effective and practical. Using this approach, the evaluators consult local, national, and international stakeholders in different forums, although some exchange of views between these groups can also be revealing for the evaluation team and mutually enlightening for the team and the stakeholders. Because an evaluation study should be responsive to new events and information, its design should be open to modification in the light of feedback from stakeholders.

2.4 Evaluation design

The evaluation design includes several interrelated components: the purpose of the evaluation, the research design, the sampling frame, the selection of indicators and primary and secondary data to be collected, the selection of research methods and instruments, and the type of analysis and reporting. There is no one ideal design for an evaluation or research study. All studies involve compromises in the light of on-the-ground circumstances and the realities of resource constraints. To obtain results as accurate as possible, given the available time and funding, many trade-offs are made between survey design, sample sizes, and the types of data collected.

These guidelines are no substitute for the many excellent manuals dealing with various strategies for evaluation surveys. The purpose here is to highlight some of the questions that need to be addressed to help researchers arrive at more comparative studies of telecentres across the Acacia Initiative. It should be noted that evaluation exercises are useful as business-planning and marketing tools, as they can produce crucial information on what services should be offered and will be profitable and what the customers need and are willing to pay for. Evaluation design questions include the following:

What will the research design be? Will the survey be a single survey, a longitudinal survey, or a cross-sectional one?

A key question for the research design will be whether to operate with single or repeated data collection, and whether the latter will use the same respondents or different samples of the population on different occasions. Longitudinal surveys collect data from the same respondents on different occasions and likely supply the most accurate information. But they are the most expensive and suffer from their own problems: respondent attrition and bias through repeated interviews.

How will baseline data be collected?

The evaluation of telecentre impacts requires some baseline (pretelecentre) data. These data can be obtained in several ways: a baseline survey before the telecentre is operational; a retrospective survey after the telecentre is operational, in which respondents answer questions about the pretelecentre situation; or a cross-sectional survey, in which some communities without telecentres are used as controls. A macroresearch design could be constructed across various telecentre programs to provide longitudinal and cross-sectional data for the Acacia Initiative. This approach may provide surrogate baseline data for telecentre projects already in operation, at least on some key indicators.

How is the population for each telecentre to be defined?

What is the definition of telecentre community? It could be the population living within a specified geographic area, an administrative unit, or some measure of the population served. Evaluators will have to establish a clear definition.

What subsamples will be selected for study?

An important design decision will be which subgroups of the population to survey. These may include the early adopters, leaders of community organizations, telecentre users, or particular economic or social groups. Several subsamples of the community are likely to be chosen within any evaluation study.

How will community-level variables be measured?

A key decision relates to the way the general population or households within the community will be surveyed. For a number of reasons, statistically robust samples using random sampling of total populations is impracticable in developing countries, where face-to face interviews are necessary, up-to-date and complete records of residents are unavailable, and cost is a major factor. Alternative strategies are to select households according to location and to interview in other locations (telecentres, schools, community meetings). Each of these strategies will have costs and benefits in terms of sample bias and ease of conducting the survey. They are discussed further in section 4 on data collection.

What primary and secondary data are to be collected?

The evaluation team will pay most attention to questions related to primary-data collection, such as performance reports, interview surveys, and focus-group discussions. But it may also examine secondary sources, such as government statistics, census data, and telecommunication records because, where available, they are quick, cost-effective, and useful for cross-checking primary data.

For household surveys, who will be interviewed in each household?

The usual choices for respondents in household surveys are the head of household, the adult who comes to the door, or all household members who are available to answer the questions. Although these are the most practical choices, they can introduce systematic bias, which the interviewers can be overcome only with extra effort, such as by revisiting the household or carefully selecting a time to come back. For example, people at home during the day are less likely to be the employed members of the household, and the researchers may specifically wish to survey employed people.

Sometimes the questions relate to the respondent herself or himself, and sometimes the researcher asks one member of the household to answer on behalf of the others. In the case of telecentres, the views of young people who are potential users would be of particular relevance.

How will ongoing monitoring be undertaken? How will these data feed into the evaluation?

The evaluation study should incorporate routine monitoring of use, users, and equipment performance as part of the business management of the telecentre, including data-entry and phone logs.

2.5 Resource planning

Whether the telecentre evaluation is part of a research project or a separate evaluation study, it will require considerable investment of resources, including money, time, and people. It is worth planning the allocation of these resources carefully, especially if some can substitute for others. With participatory methods, for example, it takes longer to collect and analyze data but may cost less in expert salaries and travel. Evaluation studies generally cost around 5–7% of a project’s total budget, depending on the evaluation design, the number of years needed to complete the research, and the type of data required by the stakeholders.

Resource planning should include the following:

Evaluation budget — The evaluation budget includes staff salaries and benefits, consultants, travel and per diems, supplies and equipment, research materials, communications, reports, and other office costs. It is not uncommon for evaluation budgets to be underestimated, especially if the initial budget is set by the national telecentre program managers. So be prepared to ask for more or make the stakeholders aware that budget restrictions will limit the scope or quality of the evaluation. Try to avoid the twin problems of inadequate resources and program managers with unrealistic expectations.
Workplan — The workplan should identify all the activities to be undertaken, the time they will take, their schedule, and who is responsible for which activity. As evaluations never go exactly as planned, the name of the game is flexibility in scheduling and in the choice of who does what. Provide for specific points in the schedule to discuss the workplan with the relevant stakeholders and, if necessary, modify the workplan, with their agreement. Particularly challenging is a situation in which new information requires a mid-course decision to change the data to be collected. The workplan has to accommodate such changes, sometimes without the benefit of additional resources.
The evaluation team — The evaluation team is a group of people with complementary skills and experience, including some who are external to the community, along with others actively involved in the community or the telecentre’s operations. Each evaluator brings a unique set of skills and perspectives to the evaluation and strengthens the end result. The internal people will certainly know the community better and may play the role of liaison or interpreter within the evaluation team. They may or may not have the technical skills of an external evaluator, but they will certainly bring an internal perspective and in-depth understanding to the group. These are the people who should lead the community in self-assessments (see section 5). Those who are involved in managing or operating the telecentre will also have a role to play, such as in surveys of telecentre users. Clearly, the more participatory the approach, the more the local people will be directly involved in data collection and interpretation.

In the planning phase of the evaluation, the team should carefully consider its composition, with a view to meeting the needs for inside and outside perspectives and technical expertise. The personal qualities of outside evaluators are as important as their technical skills. They must respect the local people and be willing to work closely with them, be prepared to solve problems, and be patient in dealing with personality conflicts or when just explaining what the study is about. In some cases, they may have to resist pressure to make a positive report; in others, they may have their own credibility attacked. Evaluation studies can engender many tensions within a project. Mistakes in the selection of the evaluation team, especially in choosing its leader or leaders, are costly in terms of the success of the assessment exercise and are hard to undo.

2.6 Developing an evaluation framework

A key part of evaluation planning is developing a framework to understand how the project works. What are its objectives? How are they related to inputs, activities, and outputs? In other words, how would you describe the telecentre project as a system? A system is usually described in terms of its components and how they are linked together to achieve its purpose; it inputs, throughputs, and outputs; its purpose; and its external environment.

A description of the telecentre as a system would include the following:

Components — Equipment, staff, owners, users, nonusers, and information suppliers;
Environment — The local-level environment, including community population, local economic activities, family income, educational levels, infrastructure, and services like schools and clinics; and the national-level and international-level environments, including national telecommunication policy, availability of Internet service providers [ISPs], and attitudes of international donors;
Objectives — Economic objectives, like obtaining financial sustainability; and program objectives, like providing universal access to telecommunications or fulfilling national-program policy objectives; and
Activities — Providing telephone and fax services, Internet access, training sessions, business-support services, promotion, etc.

Scoping the telecentre system

At the beginning of any assessment or evaluation process, the evaluation team and key stakeholders should engage in an explicit and deliberate exercise to

Map out the telecentre system;
Describe the team’s understanding of its structure and how it operates within its environment;
Identify critical conditions and limitations;
Agree on the objectives of the project;
Define its inputs and outputs; and
Agree on what indicators to use to measure its behaviour, characteristics, and impacts.

The advantage of this approach is twofold: it ensures that the evaluation team has an overall and systematic understanding of the project (which should be updated regularly throughout the assessment process); and it is a group exercise that should involve the whole evaluation team and as many key stakeholders as possible, so that everyone has a negotiated, shared vision of the evaluation.

In practice, the scoping exercise can range from an informal process involving only the evaluation team to a major multistakeholder workshop held over several days, when participants and a facilitator work through a series of steps together. A group exercise also acts as a buying-in process, in which ambiguities and differences in understanding are brought into the open and at least partially resolved for the purposes of the evaluation project. The group exercise will also ensure that the model of the project system that is constructed is as complete and accurate as possible.

Some development agencies recommend a type of systems analysis for evaluation called the logical-framework approach (LFA), or logframe. LFA has a formal methodology that is sometimes criticized for being too rigid, especially when applied to complex social systems requiring a more flexible, adaptive-systems approach. A community system can change radically and unexpectedly when its information and communication patterns are altered. Table 3 lists the main tasks of a scoping exercise, based on the LFA approach.

One result of undergoing a systems-framework exercise may be to choose a different level of analysis for the evaluation. For example, an initial focus on a single telecentre may be expanded to a wider evaluation of a telecentre program after discussions with national-program authorities and the realization that some common indicators are already available or could be easily collected across various communities. Sometimes an early focus on the operation of a single service centre, such as a telecentre, may be increased in scope to include the community itself as the system under study, and the telecentre is then reduced to just one component. In this case, the evaluation may collect additional data on the community and on other service facilities to better understand their relationships. Alternative ideas about the best level of analysis will emerge from discussions with stakeholders and within the evaluation team in the planning phase of the evaluation.

Document(s) 4 of 10