Document(s) 2 of 10

These guidelines in this handbook are designed to support research and evaluation studies of community telecentres, particularly in Africa, where the Acacia Initiative of the International Development Research Centre (IDRC) and its partners will be undertaking such studies. The guidelines identify the key questions facing the research and evaluation team, propose alternative solutions and best practices based on experience from similar field situations, and facilitate comparability of pilot projects by providing a common reference and starting point. The structure of this handbook mirrors decisions on a variety of issues, from the initial development of hypotheses, through to the research design and sampling strategies, identification of variables and indicators, collection of data, and their ultimate analysis and interpretation.¹

Section 1 discusses the rationale for the guidelines:

The evaluation studies are urgently needed to provide an assessment of the role and impact of community telecentres, as organizations and donors are implementing these facilities in many parts of Africa without an adequate understanding of how well they respond to the communication and information needs of African communities (particularly, the rural communities) or of their impacts on social equity and economic development.

Evaluation studies should include the participation of both local communities and national organizations, so an introductory volume on some of the key research issues might be useful in this regard.

The guidelines may encourage the research teams to share ideas, instruments, and methods, so that each of their case studies can contribute to a larger multinational research design — that of the Acacia Initiative itself — on community telecentres, their national policy environments, and the role of information and communication technologies (ICTs) more generally in sustainable development in Africa.

Section 2 emphasizes the importance of having an evaluation plan, with both an analytical framework for the evaluation and an implementation workplan setting out the necessary resources. The section recommends that the evaluation planning process include a multistakeholder process designed to enable telecentre stakeholders and the evaluation team to arrive at a common understanding of the objectives of the evaluation and the key dimensions of its implementation. It sets out the major research questions for community telecentre projects (Table 1) and the key information needs of the main telecentre stakeholders.

Section 3 concerns indicators, which are essential for achieving comparability across Acacia telecentre projects. Indicators for assessing telecentre performance and impacts are the conceptual link between the projects’ objectives, key concepts, and data collection — the more overlap there is in indicators used, the greater the comparability. Section 3 discusses how indicators are developed for an evaluation and the criteria for assessing the indicators (Table 5). Based on other studies and some of the initial work on community telecentres, a series of tables present the indicators proposed for background information on the telecentres (Table 6) and the communities (Table 14); the communities’ demand for telecentre services (Table 7) and for various types of information (Table 13); service performance (Table 8); and user behaviour and perceptions (Table 10). Some of the most important information for investors — whether local entrepreneurs, private-sector investors, government agencies, or international donors — concerns the financial feasibility of telecentres and the likely items to consider in their budgets (Table 11). The key objective of most evaluation studies will be to measure the telecentres’ impacts on individuals, organizations, and the community as a whole. Section 3 proposes indicators for measuring the economic (Table 15), social (Table 16), and organizational impacts (Table 17).

Section 4 focuses on data collection, especially decisions on sampling methods and techniques for conducting surveys. It sets out four guiding principles for the Acacia Initiative:

The information needs of the various telecentre stakeholders should be built into the decisions on data collection;

Stakeholders should be provided with feedback from the study;

Data collection should aim for comparability across projects, wherever possible; and

Data sets should be shared in a common data archive or repository.

These guiding principles have implications for data collection, data disaggregation, and the use of multiple methods and multiple samples within any one study. Section 4 raises a number of issues concerning sampling — including the sampling frame, unit of analysis, types of samples, stratification and multistage sampling, and sample size and frequency — and undertaking surveys. It discusses what types of survey to use for various purposes, when surveys might not be useful, the alternatives to community surveys, and the need for community-level data. Section 4 briefly introduces two practical challenges: dealing with nonresponse in any survey and the problem of attrition in longitudinal surveys. The section concludes with a discussion of the theory and ethics of interviewing.

Section 5 reviews research methods and techniques to use in the study and evaluation of community telecentres. Some of these methods, such as questionnaire surveys, are more widely known (although their potential pitfalls may not be), whereas others may be less familiar, such as projective techniques, household budgets and diaries, and observation techniques. Other techniques, such as attitude scales and participatory approaches, may be familiar but not thought of in relation to telecentres. Section 5 covers routine monitoring of telecentre operations, as well as group techniques, such as focus-group, nominal-group, and Delphi techniques.

Section 6 provides an overview of the stages of data analysis and reporting. It again emphasizes that the needs of the various stakeholders must be taken into account in the data analysis and that the data sets are valuable, not only in their own right but also as components of a larger research design. Some of the implications for analysis are that researchers need to

Document the analysis with great care, as other teams may be sharing the data;

Agree on some common coding protocols in general and specific coding categories for certain variables;

Review the theoretical models and assumptions underlying the collection of the data; and

Know which are the explanatory variables and which are the variables to be explained.

This approach encourages the research teams to discuss the issues relating to analysis, including what statistical programs to use early on in the process, and, above all, to temper the common enthusiasm for throwing all the variables into a computer-program “mixing pot” and, instead, to use common sense in interpreting the variables and their relationships. The evaluation teams can share problems and solutions through mechanisms such as the Acacia Telecentre Research Network (ATRN), an electronic discussion group currently hosted by IDRC. Section 6 also deals with the Acacia Stakeholder Information System, which has several components, including ATRN; the research-data archives proposed for Acacia; government stakeholders; telecentre managers and operators; international partners; and, last, but not least, local community stakeholders. Each of these groups of stakeholders should be connected to the Evaluation and Learning System for Acacia through the coordinator and regional staff.

It is hoped that the researchers evaluating the first wave of community telecentres in Africa will find the approach and suggestions in these guidelines useful and will build on its recommendations to establish a common corpus of knowledge on the role of ICTs in development and to create and sustain an active research network.

¹ In this handbook, we do not provide an exhaustive treatment of any one of the many aspects of designing and implementing evaluation studies, as this would be impossible within one volume. However, at the end of this report, the reader will find many good references on specific aspects of research design, methodology, and analysis. Return

Document(s) 2 of 10