Document(s) 7 of 10

6.1 General considerations

In section 4, some guiding principles were proposed for data collection within the Acacia Initiative. You need to consider the same principles when establishing a system for analyzing data and reporting back to stakeholders. These principles can be summarized as follows:

• The information needs of the different telecentre stakeholders should guide the data analysis.

• An effective stakeholder information system should be established to ensure that stakeholders, from the local to the national and international levels, receive information in a form that is understandable and useful to them and facilitates their participation in interpreting the results.

• Each telecentre study has significance for local decision-making and is important in its own right, but together all the telecentre studies form an international research framework for comparative analysis. Therefore, analysis should also facilitate meaningful comparisons across telecentre projects.

• Data sets are valuable, not only for analysis of data in the short term but also for future research questions. Acacia data sets should be deposited in a common database to make the data available to bona fide researchers in the future.

6.2 Implications for analysis

The suggestions for analysis discussed here include good practice for any research or evaluation study. They are doubly important when researchers plan to make comparisons between case studies, as the ability to meaningfully compare data depends on the confidence one has that the data have been derive from equivalent phenomena. The preparation and analysis of data, especially survey data, are skilled tasks, for which researchers require an understanding of the assumptions they are testing, the meaning of the variables in the real world, and the statistical tests appropriate to each. You can find a number of suggestions for further reading in the bibliography on analysis, but it is strongly recommended that the research team include analytical expertise or find a local expert to help with this part of the study.

6.2.1 Documentation

In the haste and excitement of getting the analysis under way, it is easy to forget to keep adequate records of the process. At the time, everyone knows what variable labels and shorthands mean and which variables they will manipulate in which ways to create dummy variables and indices. In the next round or in the next case study, their meanings are forgotten or opaque. Errors creep in, which are hard to trace. If the Acacia Initiative is to test its own hypotheses, then it is vital that each telecentre study include careful documentation of the definitions of variables, creation of new variables and indices, and manipulation of the data and samples. In general, it is good practice to provide enough documentation in technical reports to allow readers and other researchers to judge the weaknesses and strengths of the data and their analyses for themselves.

6.2.2 Coding categories

For the Acacia telecentre studies, an additional advantage of agreeing on some common coding protocols and even some common coding categories for a core set of questions and data is that they enormously simplify the task of comparative analyses and reduce the risk of error. Time spent getting the teams to agree at the outset on coding protocols will benefit everyone later on. Standardization can range from the use of the same ranges for asking and coding respondents’ ages (20–29, 30–39, etc.) to using the same number codes in data processing. For example, it is common to use the code “0” for an item that is inappropriate or inapplicable to a respondent and to use “9” for missing information or a nonresponse. Likert scales are commonly coded using 5 points, where 1 is the most favourable or positive response; 5 is the most negative response; 9 is used to record an uncodable or nonresponse; and 0 means the question was inapplicable to the respondent. Standardizing these codes within any questionnaire has also been found to reduce coder error.

It is also useful to hammer out some of the more complex issues in coding before starting the analysis, such as how to code complex responses if the respondent can give as many answers as they wish; whether to use two-digit codes if more than nine response codes are allowed or to break the codes down into two smaller categories; when to collapse codes because too few cases are given for some items; and when to code variables, such as household income or money spent on communications, as individual amounts or as ranges, even when the question may have been asked as individual amounts.

6.2.3 Questions to review before analysis begins

Even if the research and evaluation team started out with some initial hypotheses, it is useful to review the project and the research design, as a prelude to deciding on an approach to the analysis in light of the data-collection experience. Some questions for the research team to discuss at this stage are the following:

1. What is our theoretical model, and what are our assumptions? What alternative hypotheses can we test? How will we treat causal sequences?

2. What kind of sample do we have as a result of our surveys and our use of other research methods? Does it include a probability sample from a reasonably adequate frame that allows us to make some statistical inferences for a known population?

3. Given the ways we treated missing information, was the response rate adequate to reduce biases to acceptable levels?

4. Did our questions in the survey or group discussions elicit the right kind of information, or should we treat some data with caution or even discard them?

5. Who, in practice, were the respondents, and did they satisfy our initial assumptions about the unit of analysis?

6. Which phenomena are we trying to explain in the study (behaviours, attitudes, or situations) and which are our dependent variables?

7. What problem data do we have, such as extreme cases or differential errors that skew our variables, intercorrelations between predictors, or interaction effects; and how will we treat these — statistically or in qualitative interpretation?

8. What conclusions can we draw from the data, and what other information do we need to add?

6.2.4 Variables and indices

It is important to recognize that measured variables are rarely, if ever, exactly the same as theoretical ones and that these limitations in the data will constrain the validity of the theoretical model. For example, measured household income is not the same as actual household income, not only because the cash amount given in an interview may be inaccurate but also because other noncash income that may affect the disposable cash available for telecentre services may not be included. Researchers can also manipulate variables to create various types of scales, particularly ordinal and interval scales, which enable them to apply various types of statistical tests. Variables are sometimes a response to a single question, and they are sometimes constructed out of a combination of answers to several questions or observations. Researchers can combine these initial variables to create indices, either by simply adding them or by some other means.

6.2.5 Computer statistical programs and common sense

Several good spreadsheet programs and statistical packages are available to use in manipulating and analyzing data, in addition to programs to enable the evaluators to enter questionnaire survey data directly into the computer. These are available for desktop computers. The most widely available spreadsheet programs are probably Microsoft Excel and Corel Quattro Pro, and the most widely available statistical packages are SPSSX and SAS. Microsoft Access is a powerful relational database program that allows the researcher to manipulate, group, and compare data.⁷ Each of these programs has some particular strengths in performing the type of analysis required in a telecentre evaluation, and the evaluation team may be able to convert between some programs, although this is not always easy. The same rationale urged throughout these guidelines for using comparable methods, research instruments, and coding categories also applies to the analysis of data: researchers will facilitate comparison across the Acacia telecentre case studies by sharing common approaches to data analysis.

Another important point to make about statistical analysis is that the power of computers seduces us into trying everything and running every statistical test, without imposing much previous judgment or theoretical structure, with the result that we have an embarrassment of riches when it comes to interpreting the results. The hidden costs of easy statistical computer programs include the considerable time spent in trying to figure out what all the tables generated actually mean. An effort at the outset to think through the key variables and relationships deserving of analysis is very worthwhile, and disciplining oneself not to throw everything into the statistical-software “pot” is a wise practice. It is recommended that the researcher not only clearly label distribution tables, so that the results are easily read, but also spend time reviewing the tables for patterns and possible problems in the distributions for later analysis.

In the end, the most important test of all is common sense in the search for the structure of relations among variables.

6.3 Acacia Stakeholder Information System

These guidelines have throughout stressed the importance of a stakeholder information system. The research and evaluation projects of pilot community telecentres are themselves pioneering studies, with a significance for community leaders, local entrepreneurs, private investors, government policymakers, and international donors. Each of the stakeholder groups needs information on the telecentres’ performance and wider economic and social impacts on the community, as such information is relevant to their decision-making. It has been emphasized that these considerations should influence the research and evaluation project from the initial design stage through to data collection and analysis.

The research teams will develop systems to report to their various stakeholders and make these systems both appropriate to the stakeholders’ needs and manageable in terms of the resources available to the project. These reporting systems should not be the only ways evaluators obtain feedback from stakeholders and periodic interaction with them. In some cases, specific agreements with stakeholders, such as donors and government departments, will mandate the timing of reports. In the case of local authorities and community groups, the project timetable and the schedule of visits to the community will guide the timing of the reports. The following are proposed as complementary elements in a multifaceted stakeholder information system for Acacia.

6.3.1 Acacia Telecentre Research Network

ATRN is an electronic discussion group (atrn-cl@lyris.idrc.ca), established early in 1999. It includes the research teams of the Acacia projects, plus other interested researchers who ask to join. It is available for posing problems and suggesting solutions, as well as sharing ideas and research results. Eventually, its archives will become a valuable source of information on the evolution of ideas in the group. It will be important for Acacia researchers to lead the discussion group, perhaps by rotating the moderator’s role between research teams, so that ATRN primarily serves the need for collaboration among Acacia pilot telecentre projects. Later, as new projects and networks come on stream, the focus of ATRN will likely shift to other research issues. ATRN is a subset of a wider open electronic discussion group on telecentres, also hosted by IDRC (telecentres-l@lyris.idrc.ca), which addresses issues in the operation of community telecentres and shares experiences across regions.

6.3.2 Acacia research-data archives

The Acacia Initiative is considering how to establish a repository for research instruments and data to support collaboration among its projects and provide a facility for comparative research across projects. These guidelines provide a starting point for collaboration in research design, sampling strategy, and the design of research instruments. Sharing research instruments, such as questionnaires, will strengthen the basis for comparative data analysis at a later stage. Eventually, research results and data sets will also be available from the Acacia archives.

One alternative for hosting and managing the data archives is to identify a research institution, preferably in Africa, with the technical and staff resources to provide the necessary archiving services and respond to requests for information and data while limiting access to data to legitimate institutions with appropriate safeguards in place. Another possibility is an institution in a developed country or one of the donor organizations. Guidelines will need to be in place to determine who can access what level of disaggregated or aggregated data, after what time period, and for what purpose. These issues are presently under discussion.

6.3.3 Telecentre operators’ forum

USASA, the agency responsible for telecentres in South Africa, has recommended that an electronic forum be established for telecentre operators to share ideas and experiences. This would seem to be a useful mechanism, and, if possible, it should be extended beyond South Africa. It might also allow researchers to share some basic monitoring information on equipment use and performance, as long as this does not compromise business competitiveness. Another important role for an operators’ forum would be in the implementation of training and distance-learning programs.

6.3.4 National stakeholders

In the four pilot-project countries for Acacia (Mozambique, Senegal, South Africa, and Uganda), specific mechanisms have been established to bring the various national stakeholders together in a National Steering Committees for Acacia. These committees include representatives of key sectors, such as telecommunications, universities, other research institutions, government departments, and the private sector. These structures facilitate the involvement of national stakeholders in the community telecentre projects, and they are a key audience for reporting, as well as key participants in the process of learning through discussion.

One important objective of the projects for government departments (beyond those directly responsible for telecommunications) will be to learn about the communities’ needs for government information, which the local people look to the telecentres to provide. This need has implications for how government makes information accessible to the public, how rapidly it will put its services online, and how open it will be with information and access to assistance from its staff. All departments that should have this information are not likely to be part of the Acacia National Steering Committees, so Acacia should consider a wider dissemination of results across federal departments.

Where possible and appropriate, results and summary reports from Acacia projects should be posted on certain government websites, including Acacia’s and those of participating organizations. This will make information more accessible to interested members of the public and may make the government departments, themselves, more aware of the role and potential of community telecentres and how they are affecting the local and national situations.

6.3.5 International partners

For some time, it has been felt that the community telecentre initiatives under way in various parts of Africa represent an important learning opportunity, not only at local and national levels, but also at that of international organizations examining their programing with respect to ICTs. This consideration led to the establishment of both the Partnership for ICTs in Africa group, which includes the World bank, United Nations Development Programme, World Health Organization, USAID, the Carnegie Foundation, Kellogg Foundation, and IDRC, and the funding partnership between UNESCO–ITU and IDRC, which covers some of the Acacia projects.

The Acacia Initiative has also developed inside of a regional framework policy approved by African governments: the Economic Commission for Africa’s African Information Society Initiative. This facilitates the ministerial and technical links from Acacia and its projects to other ICT initiatives in Africa and provides another forum for exchange of information and learning.

As an initiative of IDRC, Acacia is also well placed to play a key role in enabling Canadians to learn how best to increase access to ICTs in sub-Saharan Africa. Clearly, the Acacia Initiative is part of several important networks in Africa and the world that are interested to learn about ICTs and will influence future investment and policy responses. Acacia will take every appropriate opportunity to enable the research findings to reach these key organizations, in addition to ensuring that its electronically posted reports are linked to the appropriate websites and get maximum visibility in the organizations and networks most important to its goals.

6.3.6 Local stakeholders

Perhaps the most important stakeholders for Acacia are local, and each project will need to consider with local leaders how best to provide both feedback and learning for the communities. To some extent, the feedback can be structured along lines of the strata selected for study. Thus, the various groups, such as telecentre staff, users, students, key organizations, women’s groups, and chambers of commerce, can be called together to hear the results of relevance to them and discuss their implications for change or future action. It is also a good strategy to sometimes mix these stakeholders up in later focus groups or stakeholder meetings, to generate more interactive discussion across the community.

Key stakeholders, such as political leaders and leaders of institutions and organizations in the community, should receive more personal and detailed feedback, both within and across groups. Feedback meetings should take place before the evaluation team makes its final decisions about the next round of data collection, so that it has genuine opportunities to modify the research plan to take the stakeholder feedback into account. This also prepares the local stakeholders and their leaders to cooperate in the next round of data collection and reminds them of the goals of the study and how the evaluation can be useful to them.

It is also valuable to post the key issues and perhaps the decisions identified in the stakeholder feedback process so that everyone can learn from them. This can be done in various ways: by writing articles for the local newspaper or community news sheet, talking on local radio, posting information outside the telecentre or some other central building, or giving presentations to particular groups, such as schools, chambers of commerce, or farmers’ associations. Another important mechanism for posting information is the telecentre’s own website, if it has one.

6.4 The Evaluation and Learning System for Acacia

These guidelines are designed to provide ideas and guidance on research and evaluation of community telecentres for research teams working within the Acacia Initiative and its partners. The guidelines are based on a review of good practice in the various aspects of research and evaluation design, collection of data, and the analysis of results. The guidelines have emphasized throughout that stakeholders should be included in the process, that their needs for information should be met, and that the studies should include the participation of local people and local institutions. Above all, the guidelines have emphasized the importance of using common research frames, models, methods, instruments, indicators, and analyses to not only strengthen individual studies but also provide a common ground for the Acacia Initiative to test broader hypotheses on the role and impact of community telecentres on information and communications and sustainable development in Africa.

The Acacia Initiative has always been envisaged as a learning venture, with special attention to measurement, evaluation, and feedback at all levels within the and across projects. Acacia can only achieve these goals with some common understanding and language about the social processes researchers are studying and the ways they study them. The process of feedback — called the Acacia Stakeholder Information System — needs itself to be managed and sustained. This is the role of the ELSA component of the Acacia Initiative.

A number of specific tasks in the ELSA portfolio have been proposed in these guidelines, including a facilitating role in the research design of telecentre evaluation studies and in the decisions to be made about data collection and analysis. A host institution is also needed to manage a common data repository (the Acacia research-data archives) and to host the electronic discussion group (that is, ATRN). The telecentres, themselves, may wish to have a network to share information and experience and to set up their own web pages.

Many of these roles are ongoing and are particularly important in anticipating follow-up studies and further data collection. The Acacia Initiative is working with a number of organizations in Africa and the world that are both stakeholders in the Acacia telecentre pilot projects and partners with IDRC in other projects. These partner organizations also look to ELSA to play a key role in furthering their understanding of the potential role of community telecentres in African development.

⁷ Mention of a proprietary name does not constitute endorsement of the product and is given only for information. Return

Document(s) 7 of 10