Document(s) 5 of 10

Indicators are at the heart of any evaluation, and it requires considerable effort to first identify them, then refine them, and, ultimately, agree on them. Indicators for assessing telecentres are a common thread linking the methodology of telecentre evaluations across various parts of Africa. Having common indicators for telecentre assessment is the key to comparative research in the Acacia Initiative.

Indicators are measuring devices. They define concepts, such as telecentre user or improved emergency response in terms of the measurements and data it is possible to collect and analyze. They define what data to collect and at what time intervals. For example, is telecentre user to be defined as “anyone who has used the telecentre once”? What about classification into users, regular users, and frequent users? How will these categories be defined? Is frequency of use the only relevant measure, or should the evaluators have some component that measures the length of the average visit and indicates the activity undertaken during each visit? For example, one study divided users into those who used the telecentre as a workplace, regular users, and occasional users. Can evaluators use definitions across different national programs, or does the concept need to be locally defined? There is no necessary right or wrong answer: the key is to select indicators that meet the objectives of the study and fulfill certain general criteria for indicators.

Although the development of indicators for telecentre evaluations is still at an early stage and the indicators are context specific, there is reason to hope for consensus on common core indicators that can be used to frame data collection for telecentre pilot projects in Africa and elsewhere. Key criteria in the Acacia Initiative are local relevance and reliability, together with robustness when used for comparison of one project or country with another. The indicators suggested in these guidelines are first-generation indicators for telecentres. Some were borrowed from other evaluation studies with similar goals and research hypotheses, and others were developed from Acacia research projects and applied in the baseline studies. After the telecentre studies have been implemented and their findings have been analyzed, the indicators will be refined and better grounded in a body of research results.

3.1 Developing indicators for telecentre projects

The evaluation team will establish indicators during the evaluation-framework planning process. To have good indicators, you need a clear vision of what you are trying to achieve and what you are trying to measure. Therefore, the first requirement for the systematic development of indicators is to have identified the results, objectives, outputs, and any key concepts associated with the project as part of the evaluation-system plan. The basic approach to creating indicators involves four steps:

Identify what is to be measured;
Develop trial measures;
Assess each trial indicator, using agreed-on criteria; and
Select the best indicators for a specific project (Table 4).

Developing indicators requires a good measure of common sense

Developing indicators involves several trade-offs. For example, it is not always better to have a lot of indicators; each indicator has a cost in terms of collecting data, as anyone who has designed (and eventually shortened) a questionnaire will attest, and the added value of each indicator will have to be assessed against the costs of obtaining the data. Some indicators may require data that cannot be reliably or consistently collected over time. Some data may require reinterviewing of the same respondent, but the sampling strategy fails to ensure that the same people are resurveyed.

It is important to recall that all indicators are based on assumptions about what is relevant, and indicators are therefore expressions of value to some extent. This is one reason why evaluators should discuss indicators with various key stakeholders before using the indicators, both to get the stakeholders’ views and perceptions and to ensure that the data will respond to their information needs. If a ministry needs to know how the area serviced by a telecentre changes over time or whether the telecentre is reducing youth unemployment, the evaluation needs to include some indicators of these changes to make it useful to this stakeholder.

Accounting for the competing needs of diverse stakeholders is also an important part of the trade-off process, and their involvement in this process will make it clearer to them.

What changes are foreseen?

In practice, it can be quite difficult to know exactly what type of change to anticipate and therefore to measure. But it is not worth glossing over ambiguities at the design stage, as they will only come back to haunt the evaluation study later on, which can incur costs in time, usefulness, and credibility. What change is anticipated or planned? A telecentre-impact evaluation may include changes in a state or condition (as in family income), an attitude (as in more interest in consumer goods), knowledge (as in learning a new language or new skills), and behaviour (as in using innovative farming methods). You should classify the changes according to whether they are absolute (something new), relative (changes in some already existing situation that increase, decrease, improve, or worsen it), or no change (maintenance of the status quo). You should also specify the indicators as clearly as possible to the relevant group (such as community members, farmers, women farmers, women farmers who are regular users of the telecentre).

Tables 4 and 5 are based on the recommendations of the Center for Development Information and Evaluation, an institute of the United States Agency for International Development (USAID). But these tables also reflect the experience and advice of many texts on how to develop indicators. Table 5 summarizes the criteria usually applied in assessing potential indicators. They are common-sense criteria and should be used flexibly. For example, direct measures are not always better than indirect measures, and quantitative measures are not always better than qualitative measures. The process of developing indicators is a combination of brainstorming, borrowing ideas from others, multistakeholder discussions, and being clear about definitions, criteria, goals, and priorities and very parsimonious about the number of indicators. In short, more work and hard decisions at this stage will later reap rewards in a more focused and cost-effective evaluation.

The proposed indicators fall under four main categories, each with several subcategories:

Telecentre performance indicators
• Basic telecentre parameters
Demand for services
Service performance
User behaviour and perceptions

• Sustainability indicators
  Financial sustainability
  Policy and regulatory environment
  Human-resource sustainability

Content indicators
Content demand
Information online
Sector-specific information

Impact indicators
Economic impacts
Social impacts
Impacts on organizations

Don’t try to do everything

The lists of indicators in these guidelines are checklists, intended only to start the ball rolling. They are not definitive, and they include indicators that are clearly alternatives. Aiming to include all the indicators on any list would almost certainly not be cost-effective. The first list (Table 6) is perhaps the only one where it is recommended that most, if not all, of the parameters be collected. These are also, as mentioned, first-generation lists, to be built on and improved with more research experience on telecentres. It is hoped they will help collaborating scientists identify a common set of indicators to make up the core of regional and international comparisons.

3.2 Telecentre performance indicators³

Telecentre performance has two important yardsticks:

The telecentre’s own goals and performance targets, as set out in its business plan, mission statement, or program-proposal documents; and
The satisfaction of the needs of its users.

The value of having common core indicators is that the researcher can objectively compare one telecentre’s performance (or one national program) with another in a cross-sectional analysis and reliably measure changes in the performance of individual telecentres or national programs over time. As noted earlier, the proposed telecentre performance indicators are grouped into basic telecentre parameters, demand for services, service performance, and user behaviour and perceptions.

3.2.1 Basic telecentre parameters

A number of qualitative and quantitative parameters together describe a telecentre: its location; origin, ownership, and management; facilities and equipment; services; and staff (see Table 6). Also important are its funding sources and means of generating revenue (discussed in section 3.3.1). The parameters proposed here constitute the recommended core description needed to compare various telecentres and establish a baseline to measure future changes. Additional information should be collected for particular telecentres and contexts. Evaluators should collect data for some indicators several times to measure changes in the telecentre. For instance, the start-up phase (usually the first year) is generally very different from subsequent years of operation. Telecentre services tend to expand, especially in business-support activities, and public funding and grants usually decline or end after the initial 1- to 3-year start-up phase.

These parameters can also provide information on what are often assumed to be success factors for telecentres, such as degree of community involvement in establishing and running a telecentre. Community involvement is measured in terms of the community’s role in the origin of the telecentre, its ownership and management, the community’s satisfaction with its performance (see Table 10), and basic community characteristics (see Table 6). In contrast, telecentres relying on international-donor initiatives or public programs, rather than the community or local entrepreneurship, are less likely to be financially sustainable after the initial funding runs out (ITU 1998).

Experience with telecentres in developed countries has underscored the importance of having well-trained and well-motivated staff to provide technical expertise, friendly support, and entrepreneurship. You can measure the contribution of human resources to successful telecentre operation, using both indicators of user satisfaction and objective measures such as hours of operation, location of the facility (including the nature of the host institution), and who employs and pays the staff. Experience with other technologies introduced into rural Africa has shown that these kind of variables also affect how the community accesses and uses a facility. The bottom line is that basic data should be collected on the physical facilities, quality of human resources, and the ways they combine to provide services for users.

3.2.2 Demand for services

Measures of demand for telecentre services (Table 7) should be included in a baseline community survey and, preferably, in any feasibility study undertaken for the telecentre project. The indicators will also measure changes in demand for services as the telecentre becomes established and better known.

3.2.3 Service performance

Telecentre performance depends on a combination of equipment and human performance, given the reasonable quality and reliability of the power source, telecommunication infrastructure, and financial structure. Many of the indicators shown in Table 8 combine these components in terms of the user’s experience of the service. The telecentre manager will need to work back from these indicators to identify the source of any problem in service delivery. Sometimes the cause will be equipment malfunction or breaks in power supply or telecommunication connectivity. Sometimes a major downtime will result from theft or vandalism, which leads back to the issues of security and risk management in protecting equipment, software, and people. Security is a major issue for telecentres, where expensive equipment is involved; theft has already been reported in the Acacia pilot projects (Khumalo 1998).

Service performance is also related to how the telecentre is managed and staffed. Telecentre staff play a key role in providing user-friendly technical support; ensuring good administration, risk management, and security of equipment; and educating people to adopt appropriate user behaviour (no food or drink near computers, no personal disks or software that may contain viruses, no fighting or running, etc.).

Service performance is also related to telecentre design and the type and quality of equipment and maintenance contracts purchased to meet the anticipated demand for services. One basic parameter critical to equipment performance is the total population (or number of users) a telecentre is supposed to serve and the amount of time any piece of equipment is in use. Section 3.4 deals with aspects of performance related to the availability of relevant information and applications.

To measure many performance indicators, researchers require longitudinal data. Information collected on a daily basis, recorded by telecentre staff using a standardized daily log, is the most direct and accurate. The telecentre staff should, at a minimum, keep a daily “trouble log,” in which they record problems with the equipment for later analysis and diagnosis. Evaluators can complement and cross-check these data by asking users how frequently they experience success or failure with the equipment. A sign-in procedure (either on screen or with paper and pencil) for users can provide these data, or the evaluators can undertake regular user surveys. Experience in developed countries strongly suggests that the evaluators should prepare summaries of these daily logs on a regular weekly or monthly basis and regularly review them with all staff to obtain feedback and elicit suggestions.

In addition to the general performance indicators shown in Table 8, other specific indicators are useful in assessing a telecentre’s performance in relation to Internet use and access. Table 9 is drawn from the recommendations of the US National Research Council (NRC) on developing indicators of Internet use in Africa (NRC 1998). Some of the data needed for the indicators may be available from a phone company, whereas others may have to be collected at the telecentre.

3.2.4 User behaviour and perceptions

The measures of user behaviour and perceptions cannot pretend to give a full picture of the role of the telecentre in the community; nevertheless, they are central to any evaluation of telecentres, and the user surveys you need for these measures are likely to be cost-effective. As well, the evaluators can conduct these surveys at the telecentre, where the respondents are likely to have an interest in the telecentre and be reasonably knowledgeable about it. Furthermore, such surveys can provide longitudinal data if each user receives an identity code and answers a short survey on each visit. The evaluators may find it is worth having a panel of users to form a longitudinal sample, in addition to conducting random sampling or establishing user login procedures. A longitudinal sample will provide a measure of change over time in a number of variables, such as telecentre-service use, frequency and length of visits, payments per visit, and changes in satisfaction and perceived benefits.

From telecentre users can be obtained two broad types of indicators:

Reports on behaviour (what services were used on each visit, for what purposes, etc.); and
Subjective measures of telecentre services and their benefits (Table 10).

Because these indicators should be susceptible to analysis by individual and group characteristics (age, gender, educational level, occupation, membership in organizations, etc.), evaluators should collect basic data on the respondents when they first agree to take part in the survey and should compare data from user surveys with those the telecentre staff collect on performance. User-survey data will provide both cross-checks and information from various stakeholder perspectives. Collecting data from users while they are at the telecentre increases the accuracy of the survey and takes advantage of the fact that the users share the evaluators’ goal in gathering information to improve the service.

3.3 Indicators of sustainability

The sustainability of telecentres is of considerable interest to everyone involved in the centres, especially if the telecentres receive assistance during the start-up phase in the form of reduced tariffs, special investments in equipment and infrastructure, or international-donor assistance. The big question is whether the telecentre will be financially feasible once the special grants end, the equipment needs to be replaced or upgraded, or the telecentre has to compete on a commercial basis (or at least run on less public funding). A number of telecentres have been established in Africa without the minimum level of income and infrastructure to ensure their financial success, and there is concern that these telecentres will fail, like other development projects before them, and bring the experience of failure to the communities as well.

The question of sustainability also arises for private-sector telecentres, although they usually have a smaller gap between start-up and operating costs and revenues, as they tend to take a more incremental approach (starting as a phone kiosk) and build up as demand and revenues allow.

The Universal Service Agency of South Africa (USASA) is the statutory body responsible for promoting universal access to telecommunications in South Africa. USASA asks prospective franchisees two questions about sustainability:

What ideas do you have for making a telecentre feasible and sustainable so that it can generate its own funds, maintain contracts, or receive external funding?
How will you ensure that the telecentre remains socially sustainable so that it continues to be relevant and used by the community?

These questions, together with the wider issue of a supportive policy and regulatory environment for the telecentres, define the key indicators of sustainability.

3.3.1 Financial sustainability

The basic indicator of financial sustainability is the situation where the revenues of a telecentre (including grants, in-kind support, and cash earnings) are greater than the expenditures and this happy situation is projected to continue for at least 3 years. However, this accountant’s view of financial sustainability is complicated by the fact that the community telecentres are a mix of public good and private service, and this mix is generally reflected in the funding structure. Most telecentres receive some public funding, at least during the start-up phase, although many commercial phone–fax shops and cybercafés in Africa operate without direct public support, particularly in densely populated urban and peri-urban areas.

The situation is further complicated by the special tariffs, grants, and regulatory arrangements made to support telecentres in their initial phases, especially in areas where they are unlikely to survive on their own earnings. The financial sustainability of a telecentre therefore depends on a number of intersecting factors related to the telecentre’s own budget, its local catchment area of users, and the wider policy and technology environments. However, in practice, telecentres may be seen as financially viable if they cover their operating costs.

The telecentre will have significant start-up costs in building or converting a suitable structure to house the equipment, in putting in suitable power and telecommunication supplies, and in equipping the telecentre. For a number of the pilot telecentres supported by the Acacia Initiative and its partners, public funding or international-donor support fully or partially meets these initial costs. In some projects, moreover, the government is waiving the normal criteria for installing telecommunication links (in terms of population, gross national product [GNP], or economic activity). Although these subsidies get the telecentres started, they do not ensure their financial sustainability in the longer term. Several of these pilot projects may, in fact, have a hard time achieving sustainability once the subsidies and grants end. In general, the budget for the start-up phase will be different from the budgets for later operational years, and, in some cases, the initial budget is not a good predictor of longer term sustainability.

Table 11 outlines the principle items in a telecentre budget that need to be taken into account in assessing a telecentre’s financial risk. Experience from telecentres in various parts of the world shows that the most commonly underestimated costs are those related to staff training, security (both physical and data security), and the updating and maintenance of equipment, especially computers. Uncertainties are built into the expenditures, especially communication costs. The telecommunication supplier establishes these costs, and the costs can change suddenly with a change in government policy. A preliminary evaluation of USASA’s experience in South Africa (Khumalo 1998) showed that one of the major problems was in the prices charged for telecentre services. USASA does not fix tariffs or prices, and the study concluded that the pricing structure varied between telecentres: prices could be well below or well above the actual costs. This clearly jeopardizes financial sustainability, directly and in terms of future user satisfaction.

On the revenue side, important considerations are how long the initial grants or public subsidies will continue and whether other sources of revenue will replace this often major initial funding. You can see in Table 11 a distinction drawn between revenues earned from the core business of the telecentre (which is not fixed but generally relates to connectivity services and the provision of computers and software) and those earned from ancillary activities. Often, the core business is unsustainable by itself — over time, successful telecentres increase the number and volume of their ancillary activities. Activities such as educational and business services depend more on there being qualified staff in the telecentre to complete a task than on simply giving the users access to the equipment to enable them to complete it themselves. Some telecentres in Europe provide a range of business-centre services for small and medium-sized enterprises and local organizations. These services include inputting and analyzing data, secretarial services, desk-top publishing, budget preparation, and reception. Most of these European telecentres expect that this part of their business will increase as a percentage of their revenue (ITU 1998).

The telecentre budget and business plan form the basis of only one approach to measuring financial sustainability. A number of other economic models and indicators of demand have been developed to predict the commercial feasibility of telecommunication services, based on the socioeconomic characteristics of rural and urban areas in developing countries. These indicators are also useful in evaluating telecentres. Some standard indicators of capacity to pay for telecommunications are GNP per capita, population density, penetration of electrical power in rural areas, and penetration of telephone service.

Rural areas of developing countries are generally thought to be able to pay 1–1.5% of their gross community income on telecommunication services (Kayani and Dymond 1997). In the poorest of these areas, this figure may be as high as 3% because of the lack of alternative communication services for people unwilling to make a long journey. ITU uses a figure of 5% of household income to estimate capacity to pay for telecommunication services (Ermberg 1998). In reality, these figures are rarely realized, for two main reasons: they assume that a telecentre is accessible to everyone when they need it, meaning that it is well located, is open, and is functional; and they take no account of collect calls or of incoming calls at the telecentre.

Studies in Kenya, Malawi, and Zimbabwe found 60% of the outgoing calls at rural pay phones are collect calls (DANIDA 1991). This takes no account of incoming calls, which are a common use for pay phones in rural and small-town Africa; indeed, in Mozambique, a country with a high out-migrant population, queues of people line up outside the pay phones on the weekends, waiting to receive their incoming calls. For a telecentre, this would mean providing a service and incurring fixed costs without receiving any revenue. It has been proposed that some account be taken of this phenomenon when assigning telecommunication charges to rural telecentres, as the telecentre is actually cogenerating the revenue paid out by the caller (usually in the urban areas) (Kayani and Dymond 1997). In South Africa, some telecentres charge a fee for receiving an incoming call.

The World Bank model for testing the feasibility of providing rural telecommunication service estimates average rural incomes with a formula that includes per capita gross domestic product (GDP), country purchasing-power-parity income-distribution figures, and rural population as a percentage of total population. The result (average rural income) is compared with the estimated capital cost of providing the service per line (based on population density and geographic factors) and the estimated annual revenue per line needed to cover capital costs and make a profit. The model then calculates the number of inhabitants required to support a single telephone line. As demand increases, the area of commercial feasibility also expands. This means that the ratio of marginal and unprofitable customers decreases, and a government regulator can use the model to calculate the tax and fiscal incentives needed to enable the telecommunication provider to serve marginal customers (Kayani and Dymond 1997).

Indicators based primarily on per capita income may be too conservative. A study in Botswana (CANAC Telecom 1990) estimated the demand, penetration, and revenue for an average village (1800 population) to determine the feasibility of installing private and public phones. It concluded that the average rural revenue would be US $1200 per line, which was below the level required for profitability. A loss of 3% on the annual revenue of the public telecommunication corporation was predicted, but 5 years later the demand from rural communities was more than twice that projected and village pay phones were earning up to 50% more than projected (US $2700 per line). These indicator errors were due to several factors, including the difficulty predicting demand for a service not yet available and the even higher costs that pay-phone users would have to pay for the alternative, which usually involved long journeys and uncertain results. Clearly, people place a higher value on their time than one might assume.

3.3.2 Policy and regulatory environment

A supportive policy and regulatory environment can make or break the financial and social feasibility of a telecentre program. The main indicators of a supportive policy environment are the following:

A commitment to providing telecommunication service to all parts of the country, including marginal rural areas;
Fiscal and regulatory measures to enable telecentres to become commercially sustainable;
Encouragement for the development of an ISP market; and
A nondiscriminatory policy on access to Internet services.

Even though new technologies — satellite technologies and wireless phones, for example — reduce the costs of providing telecommunications, servicing the rural areas still generally costs more and produces less revenue, as costs are a function of subscriber density. A key issue for the economic feasibility of telecentres in rural areas is whether the government is so committed to ensuring universal access that it is willing to provide the necessary support to telecommunications providers. Governments can provide the providers with incentives to service unprofitable areas. In a monopoly situation, governments can, for example, establish cross-subsidies between rural and urban areas. As competition increases among providers, subsidies may be targeted to provide service to unprofitable areas. In other situations, such as in Canada, the government may impose route averaging to subsidize the effective costs of calls in unprofitable areas (Hudson 1998). Another approach, followed in Peru, is to establish a rural telecommunication development fund, to which all providers contribute a portion of their revenues. A fund like this can also attract private-sector investment and loans. Most Latin American providers are required to extend services to rural communities above a certain size as part of their licence requirements (Kayani and Dymond 1997). Similar requirements to serve less profitable rural areas are being imposed in new licence agreements in South Africa and Uganda.

Most African countries do not yet have an overall policy framework for communications, but a number of them, including Ghana, Senegal, Mozambique, South Africa, and Uganda, recognize this need and are beginning to liberalize policies. Government policy clearly impacts on access to the Internet. Some countries are still restrictive in this regard, believing that their citizens will be unduly influenced by information coming largely from the industrialized world, especially the United States. More generally, governments can be more supportive of telecommunications in general and telecentres in particular by providing various fiscal incentives and subsidies, especially to offset the cost of telephone lines and the price of local and international calls. Other measures relate to the taxes and import duties on telecommunication equipment, computers, and software (NRC 1998).

Table 12 shows possible indicators of a supportive policy and regulatory environment. Not all of these indicators are readily available in some countries, and others will have to be defined according to the local situation. In some cases, only national-level data are available, rather than statistics broken down by rural and urban areas.

3.3.3 Human-resource sustainability

The question of human-resource sustainability should be of concern at both the level of the individual telecentre and that of the regional or national telecentre program. Shortages of adequately trained staff and losses of trained staff and technical experts to other employers, usually in the private sector, plague public-sector telecentre and telecommunication programs. Key indicators are salaries and benefits compared with those offered by competing employers, staff turnover rates, and investments in training. Another indicator is ratio of local qualified technical staff to imported technical expertise.

Investment in training is a measure of human-resource sustainability. Some fairly easy indicators to obtain are the number of technical ICT training courses provided in local (national) institutions over time and the number of graduates or diploma students they are producing. These numbers provide some measure of the pool of qualified personnel and how it is changing over time. If the pool of qualified personnel is at or below the level needed to support the telecentre program and telecommunication and Internet services, problems can be anticipated in human-resource sustainability, as demand for such technical personnel is likely to rise rapidly and outstrip the human resources currently available or in training. Demand and supply of specialized labour are notoriously difficult to manage, especially in the high-technology sector. However, in Senegal, it is reported, many unemployed recent university graduates are attracted to the telecommunications industry and could provide technical support to community telecentres.

In South Africa, the USASA established a 5-week training program for telecentre operators, which is certified by the Wits University Faculty of Management. It covers telecentre management (planning, finances, personnel, infrastructure); basic maintenance for phone, fax, photocopier, and computer system; how to train others; entrepreneurship and fund-raising; the role of the telecentre in supporting small businesses, schools, and the community; the use of computers for communications; and applications such as word processing, spreadsheets, databases, e-mail, and web browsing.

An evaluation conducted 1 year after the program began found that telecentre operators did not clearly understand their responsibilities and obligations as franchisees and generally did not know how to manage their businesses. In particular, most of the telecentres managed their financial records inadequately, and USASA had no system in place to monitor financial performance (Khumalo 1998). It is not known how many of these operators had completed the Wits University course. But to avoid these outcomes, the Acacia telecentre project being carried out in Senegal by Environnement, développement, actions du tiers-monde (ENDA-TM, Environment, Development, Action in the Third World) emphasizes the importance of adequate training for telecentre operators. The training program includes not only applications like word processing, spreadsheets, presentations, and the creation of e-mail accounts and web pages, but also skills such as management, project evaluation, data collection and analysis, and conflict resolution.⁴

Some measure of the investment in human-resource training as a proportion of overall telecentre-program costs is another indicator of human-resource sustainability, although appropriate benchmarks need to be established. A study of World Bank information-technology projects, for example, found an average of 24% of total project cost was invested in training and technical support (Hanna and Boyson 1993). In the United States, the information-technology industry spends 50–68% on training, even though the basic knowledge of incoming personnel is quite high (Norrish 1998). The investment in training for telecentre programs in Africa almost certainly should be no less than that in developed countries, but it almost certainly is. It is therefore important to examine human resources, particularly the training aspects, at both the level of the individual telecentre and the level of the national agency.

3.4 Applications and information content

Information, in the abstract, means little to the engineer, the agriculturalist, the farmer, the craftsman, or the doctor … . Considering information for information’s sake is a dead end.

— Menou (1993)

Social theories of information emphasize the importance of timing, the credibility of the source, and the relevance of the information to the receiver at the time it is received. The information available on the Internet has underscored the importance of another factor: the ability to sift through the information to find what is useful in a reasonable amount of time — in other words, the cost-effectiveness of the information search process. In the NRC model, information content is the key variable linking Internet supply and its supporting environment to impacts on organizations, markets, and sectors (NRC 1998). Telecentres have tended to focus on serving either the needs of local enterprises (and being businesses, themselves) or the community’s needs, such as education, health, and culture. Thus, privately funded and owned telecentres are more likely to concentrate on business applications, whereas government- or NGO-supported telecentres are more likely to concentrate on public issues, such as community development and education. In practice, many telecentres serve both types of user, and MCTs are explicitly designed to have a dual business–public orientation.

3.4.1 Content-demand indicators

The evaluation team will need to pay attention to the information needs of local users. Some measures of the demand for information, collected from key organizations and community leaders and the baseline community survey, will identify at least the perceived priorities for information at the outset of the telecentre project. Later, both community-wide and user surveys, asking the same questions, will provide direct measures of changing needs. It is important to pay particular attention to the applications and content that are valued by key institutions and services already in the community, such as medical facilities, schools, colleges, and government offices. The baseline survey of the main economic activities in the area will also provide data on these applications, such as price data for locally produced goods, even if the community members are not yet aware of the vast array of information resources available on the Internet or specialized networks.

Table 13 reports the results of an informal survey (Whyte 1998) of user groups in communities destined to host pilot-project telecentres in Mozambique, Senegal, South Africa, and Uganda. The survey was conducted to identify community needs in information content and communications.

Table 13. Information and communication needs expressed by communities in Mozambique, Senegal, South Africa, and Uganda.

Government Government regulations, legislation, procedures, “how to do it guides” Up-to-date information on taxes, incentives, subsidies, quotas, tax changes General public information on government Access to one-stop government electronic service

Agriculture Up-to-date information on markets, prices Data on pests, infestations, animal diseases, how to control them Improved (appropriate) technology for traditional crop cultivation, animal husbandry “How-to” information on new, more profitable, agricultural initiatives (e.g., mushroom growing, rabbit rearing, egg production for urban markets) Better information on improved animal breeds, veterinary information generally Telephone access to vets and artificial insemination services Communications to organize load sharing for truck transportation Listings of where seeds of specific qualities, quantities are available Listings of available spare parts for agricultural equipment Postharvest technology (cold storage, etc.)

Small business Information on prices, demand, competition in various markets Computerized small-business accounting systems (bookkeeping, profit, loss information) Inventories, stock management Best practices, business management, start-up Information on credit, small loans, revolving funds (how, where to apply) Opportunities for export, import procedures Electronic commerce

Health and environment AIDS, HIV information Information on family planning Health education, child care Information on water, sanitation, including water-related diseases Appropriate technology for latrines, waste management (including night soil) Energy technology, including biogas, solar driers Medicinal plants, traditional medicine, biodiversity Nutrition, recipes, new ways of cooking Telephone access to doctors, midwives, medical services Weather forecasts, crop infestations

Formal organizations (hospitals, schools, local government, NGOs, CBOs) Creating, maintaining computerized databases (patients’ records, student enrollment) Reporting to headquarters (notifiable diseases, crime incidents, monitoring, routine requests for supplies, etc.) Local communications network (ambulance dispatching, linking schools, NGOs, etc.) How to organize communities, establish new organizations, develop group dynamics Emergency-response communications Access to drug registries, medical expert systems Access to general reference libraries, online information

Education Distance learning (especially for teachers, students, unemployed youth) Adult literacy Skills upgrading, certification Learning new, income-generating crafts (especially for girls) General self-learning, self-improvement Group education sessions, using audiovisual equipment

Empowerment democracy “Only people armed with information have the power to do things” “We’ll get more improvements if we have the communications to ask for them” Access to newspapers, magazines (What is going on in the capital city, the world?) “Find out what our government is doing” “Teach young people about local cultures and traditions, instill pride in society”

Family, personal, informal sector Communications with absent family members, overseas migrants Communications with family members caring for children Money transfers for family, business Information on jobs Employment applications

Source: Whyte (1998). Note: CBO, community-based organization; NGO, nongovernmental organization.

Understanding the local needs for applications is thus the first step. But responding to them is a far harder and longer process and more difficult to measure. The telecentre-program managers and the individual telecentre operators are on the front line in facilitating people’ access to information that is really useful to them. This is a measure of their own ability to act essentially as community development officers, or animateurs. The degree of emphasis on this aspect of the role of telecentre operators appears to differ among the Acacia pilot projects. For example, in the ENDA-led Acacia project in Senegal, it is a key aspect of the operator’s role, and one could say that the telecentres are both content driven and firmly linked to participatory community development. The NGOs leading each telecentre have specific and distinct entry points into community development. These entry points range from education through to small-business development, traditional medicine, water and sanitation, women’s development, microcredit, youth, and promotion of local culture. The lead NGO brings together all the community organizations to support the community telecentre (see the ENDA website at www.enda.sn).

The following are some indicators of how a telecentre will perform in providing high-quality information (that is, locally useful and valued information):

The emphasis placed by the telecentre operator or management on applications;
Their level of knowledge of how to access that information or to link to specialized-application networks like HealthNet;
Their ability to create and disseminate local information through local web pages or through their participation in an applications network; and
The applications software and reference materials, such as CD-ROMs, they have collected in response to local needs.

3.4.2 Information online

Not only do individuals in developed countries have more access to information but there is more information relevant to their needs online or at the end of an automated phone system. The rapid increase in the use of the telephone and the Internet has led commercial companies, governments, and traditional information providers, such as libraries, to hurry to put their information online. The commercial use of the Internet, including banking and investing online, has exploded. Communities in developing countries express a similar demand for locally relevant information online, whether it is daily market prices, the posting of changes in government regulations, agency-staff direct phone numbers or e-mail addresses, or research databases for Africa. According to the Tradepoint Senegal project, however, a major bottleneck is getting government departments to go online with the information necessary to make electronic trading in Senegal work.⁵ The farmers and the entrepreneurs in the local chambers of commerce are so enthusiastic and ready to work online that national governments cannot respond fast enough to their demand for telecommunication services.

Indicators of the local relevance of information will need to include not only the demand but also the rate of change in supply. Supply can be tested directly by checking government, commercial, academic, and NGO websites and telephone-assistance numbers to see what information is available to users. However, the cost of calling government departments, if the right person cannot be found or the information cannot be transmitted effectively, can make the telephone ineffective as an information search tool. Similarly, a telecentre is no great help to anyone if the information sought is unavailable either on the Internet or through specialized networks (which can link Internet, e-mail, and even fax and reach almost anywhere). The evaluation team (and the telecentre management) can experiment by attempting to access a sample of key national and local institutions by telephone, fax, and Internet; repeat the experiment to test how the situation changes over time; and then compare the results of these experiments with reports from telecentre users.

3.4.3 Sectoral and local electronic networks

Another measure of the changing scene in Africa is the growth of regional and national electronic networks. These have not only provided access to the Internet by using links like Fidonet-based mail connections, radio connections, and e-mail sent to fax numbers but also created important cross-sectoral networks. For example, in Ethiopia, the Pan African Development Information System (PADIS) network, an initiative of the Economic Commission for Africa (ECA), comprises more than 1000 sites, including research and academic institutions (14%); NGOs (33%); governments (4%); businesses (6%); and individuals (17%). The remaining 26% are international sites (1995 data). The PADIS network produces several benefits for participants in Ethiopia: communication services at much cheaper rates than fax, telex, or telephone; an increase in collaboration in research and research meetings; and a less hierarchical pattern of interaction among researchers (Adam 1996).

Another useful network for telecentres serving NGOs to link into is NGOnet, which was started by the Environmental Liaison Centre International (ELCI) in Nairobi. NGOnet is a coordination centre and clearinghouse for ECA-supported African environmental NGOs. To provide these NGOs with cheap access to e-mail, it set up four centres with high-speed modems, and these centres can also provide the NGOs with a local line to connect to the Internet and local support, including training. The hosts are ELCI in Nairobi, MANGO (Micro Access for Non-governmental Organizations) in Harare, ENDA in Dakar, and ENDA–Arabe in Tunis. NGOnet uses Fidonet, a low-cost, grass-roots electronic communications network that has been operating successfully since the 1980s.

Other electronic networks of importance to developing countries are Schoolnet, SatelLife, and HealthNet. SatelLife uses inexpensive store-and-forward systems to provide information on public health, medicine, and the environment. Originally, it linked medical centres in Africa with medical libraries and research centres in North America and Europe. It operates HealthNet, which is an information service connecting health-care workers around the world and offering e-mail and conferencing, as well as health-related journals and publications online. With its satellite system, HealthNet can reach any remote area if the user has a computer, a terminal node controller, and a satellite radio.

In Uganda, a sustainable and self-funded e-mail service has developed from MULKA, a locally inspired national electronic network started by Makerere University, Kampala, Uganda, and an NGO (Musisi 1996). The service is linked to a regional project, East and Southern African Network (ESANET), which is supported by IDRC and links universities in the region. ESANET seeks to identify cost-effective data-communication modalities for the research community and to enhance the capacity of governments to collect and analyze data for public-policy decision-making.

These and other networks play important roles in making relevant information and low-cost communications available in Africa. They are also examples of the kinds of locally relevant information networks that telecentre operators need to know about. How familiar a telecentre management group or operator is with these and similar networks is yet another indicator of the quality and relevance of the information provided by a telecentre. The evaluation team can also ask users, particularly organizations, whether they are aware of such networks, whether they use them and how frequently, and whether they find their communication and information services more relevant and less time-consuming to access than searching the web.

3.5 Impact indicators

Finding measures of impacts on individuals, organizations, and the community is a key objective of most evaluation studies. Such measures relate to very important research questions for many local, national, and international stakeholders. Is the telecentre a positive force for community development? Does it benefit some people more than others? Does it act as a catalyst for other positive initiatives and innovations at the local level? Does it help people to help themselves? These questions convey the assumptions and vision of the promoters and funders of telecentre programs. Other questions are narrower and more practical. Is a telecentre is more beneficial to some economic sectors than to others? For every user who comes to the telecentre, how many others are indirect beneficiaries? Are there drawbacks to the telecentres, and who suffers as a result? What features of the telecentre are responsible for the greatest number of benefits and their most equitable distribution? How can these features be strengthened and replicated?

Who caused what?

Causality is one of the big conundrums in measuring impacts. Did the telecentre contribute to the rise in local economic productivity or the increased participation of women in local organizations? Or were these economic and social changes already occurring and did they themselves act as catalysts for locating the telecentre in that community? Clearly, economic potential, local leadership, and community initiative are factors in deciding where to locate telecentres, even for those telecentre programs in which the public-good rationale is strongest.

The best one can usually do, given that these are generally small-scale surveys and not large data sets suitable for endless statistical manipulation, is to obtain good baseline data; measure succeeding changes carefully; demonstrate a strong association between the telecentre and the economic or social change found; and apply the argument from “reasonableness” in judging the likely direction of causality. Collecting supporting data from several telecentre sites and from control communities without telecentres can considerably strengthen the case for a certain direction of causality. It is in this exercise that regional comparisons and consistency in measuring indicators will produce the greatest pay-off.

The evaluation planning process, particularly the multistakeholder discussions, will have provided the evaluation team with a large number of potential questions about impacts, and the evaluators will need to structure and prioritize these questions. One dimension to consider will be whether the impacts are expected to be immediate and short term, intermediate, or long term and how to translate these periods into a time frame for data collection. Many impacts may not appear for several years and cannot be directly measured within the time frame of the evaluation study. Unless the evaluation team makes a return visit several years later, these long-term impacts will be the most difficult to quantify.

To obtain indicators of impacts, the evaluators will have to collect data on characteristics of individual and household respondents relevant to the impact and research questions. As pointed out in section 2, it is better to collect disaggregated data on individual characteristics: disaggregated data can always be aggregated in the analysis, but aggregate data cannot be disaggregated. However, there is a trade-off between the expense and difficulty of collecting very disaggregated data and the level of aggregation to be used for the analysis. Classic examples are age and income. Although individual ages in years and incomes in dollars will produce interval data that can be manipulated, such data are difficult to collect from respondents, and the researchers usually ask for this information in terms of three to five levels (under 20 years, 20–40 years, etc.).

Except for the organizational impacts and changes to the community at large, the basic data-collection units for measuring impacts will be individuals and households. Particular attention must be paid to the choice of sampling frames to ensure that they capture adequate samples of the most relevant groups and can differentiate between them. Breakdowns such as gender, age, ethnicity, occupation, and language should be included, and perhaps also the less obvious variables, such as employment history, family-migration status, and participation in political parties or other measures of local activism. The characteristics of individuals, households, and communities suggested for the analysis of social and economic impacts are shown in Table 14.

Table 14. Characteristics of individuals, households, and communities for data analysis.
Level	Selected characteristics
Individual	Age, gender, marital status, children Income level Level of education, functional literacy Languages spoken Occupation, employment status Membership in community groups Category of telecentre user
Household	Location, quality of residence Numbers in household (adults, children, gender, relationships) Ratio of employed to unemployed adults Employment status, occupation of head of household Age, gender, marital status of head of household Income level Economic activity by sector Whether a household includes one or more users of a telecentre
Community	Population size, age, gender, ethnic distribution Settlement type, geography, environmental setting Environmental quality, major environmental problems Area of settlement, surrounding lands, land ownership Languages, culture, religion, ethnicity Income distribution, savings, credit Main economic activities (sectors), products Commercial activity, businesses, trade patterns Main institutions, organizations Physical infrastructure, services Distance to other services (medical, government, communications, libraries, education, markets, etc.) Schools, other educational facilities School enrollment, drop-out rates, completion rates Adult literacy rate Population growth rate, life expectancy Mortality, morbidity rates Disease prevalence Water, sanitation services Health-care programs, facilities, vaccination rates

The local context will generally define impacts on individuals, households, and communities in detail. The evaluation team can use the core set of potential impacts recommended in these guidelines as a basis for making comparisons if it measures them across projects and countries; however, the team will have to further refine the indicators in this core group to make them match local conditions and data availability.

Although in the next sections the impact indicators are broadly categorized as economic, social, or organizational, these are not watertight compartments.

3.5.1 Economic impacts

A number of statistical studies that have used longitudinal studies from many countries, including the United States, to correlate investment in telecommunications with per capita GDP or GNP have generally found causal relationships in both directions. An important characteristic of telecommunications is the tendency for each user’s potential benefit to rise with the total number of users — the opposite of the “tragedy of the commons.” Another is that, although both parties to a telephone call benefit from the communication, only one pays directly for the call. The general thrust of the studies done on telecommunications in developing countries, where the relative gains in cost-effective communications are initially high, is that the projects produce not only public-good benefits but also major benefits in efficiency and productivity. These benefits include better price information; reduction of travel costs, inventory, and downtime, when equipment is broken or needs maintenance; timely delivery of products to market; and energy savings (Hudson 1998).

How does one translate these findings to the community level, if one depends largely on primary data collected by the evaluation team? Household-level data on income, savings, etc., are sometimes difficult to obtain and to cross-check. Strategies to deal with this problem include the following:

Using key informants motivated to help the evaluation team;
Using indirect measures of wealth accumulation (such as owning consumer goods, like a radio, television, refrigerator, or bicycle), which can also sometimes be observed directly (for example, construction of a new house);
Asking questions related to spending patterns, rather than to savings; and
Using available statistics where possible (for example, local market prices).

A valuable complement to a household survey is to ask a panel of representative households to record a household budget of income and expenditures, using a specially designed booklet. Households in this panel will likely become significant participants in the data-collection process and should receive special treatment according to an agreed-on protocol. This can be as simple as providing them with an analysis of their expenditure patterns (always a revealing process) or helping them identify patterns in market prices and business cycles. Clearly, the best data on impacts will come from longitudinal studies, which can measure changes from a pretelecentre baseline situation. Therefore, many of the indicators in Table 15 assume that measurements of change will be taken at various times.

Table 15. Indicators of economic impacts.
Change	Potential indicators
Income, prices	Change in household income Percentage household income of migrant-worker remittances Average daily wage for unskilled labour, agricultural worker Prices obtained for sector-specific products Ratio of cash to subsidence crop production Value of exports (agricultural, nonagricultural) within, outside country Availability of credit Changes in household budgets
Work related	Percentage (especially youth) employed and earning wages in community Percentage of successful job searches using telecentre Percentage of households engaged in enterprises Percentage of households adopting improved technology, new products Increase in hours of service through reduced downtime, travel time (e.g., shops, mechanics, pharmacy, clinic, ambulance) Increased number of different markets for buying, selling Changes in occupational patterns
Wealth, property accumulation	Growth in number, size of community businesses Percentage of households owning specified consumer goods Percentage of households owning a vehicle Growth in individual, business telephone subscriptions Percentage of households with new construction, major improvements Percentage of households with electricity
Information search	Time to obtain information, communications Monetary cost to obtain information, communications Percentage of successful trips, attempts to obtain information, communications Time to place, receive orders for spare parts, supplies

3.5.2 Social impacts

Social indicators emphasize another measurement issue, in addition to the issues involved in measuring economic impacts: the need to define social indicators somewhere between those measuring “bottom-line” impacts, such as mortality rates (which, if the telecentre has any contribution to make, will be impossible to separate out from all the other contributing causes) and those measuring direct impacts on behaviour, such as the number of telephone calls made per household per month. Although direct impacts can probably be measured more easily, such data may not have great import at the level of social impact.

The indicators suggested in Table 16 were drawn from many possible choices and were selected for probable data availability and relevance. They were judged relevant on the basis of the expectations of change expressed by groups in the communities visited in 1998, when the communities were looking forward to the benefits of telecentres; and on the basis of the major research questions for telecentre pilot projects (see section 2.1).

Table 16. Indicators of social impacts.
Change	Potential indicators
Social structure, status	Number of households Percentage of households with migrant workers outside community Occupation of heads of household Percentage of professional workers residing in the community Turnover of professionals (teachers, nurses, etc.) in the community Ratio of employed to unemployed adults, youth
Health	Percentage of households with improved water supply Percentage of households with improved sanitation Child mortality rate Main childhood diseases Major causes of morbidity, mortality Percentage of children regularly visiting a health clinic Percentage of households with a member treated via telemedicine
Education	Adult literacy rate Highest educational level attained by head of household Children’s enrollment in school Youth, adult enrollment in training, skills upgrading Participation in distance-learning courses Competence in English, French, Portuguese as second language Competence in skills related to telecentre use (word processing, spreadsheets, simulation games)
Community action	Number of community organizations Active membership of community organizations Community-action projects Community newsletter, website, radio station Response times for emergency services Flyers, announcements
Behaviour	Use of telecentre (purpose, frequency, success rate) Use of alternatives to telecentre Pattern of work, recreational activities Patterns of travel to other communities, towns, capital Domestic violence, violence toward women Use of specialized professional services (veterinary, counselling, tax advice) Purchases based on information from Internet, e-mail Regular readers of newspapers, news sources online Changes in time budgets
Knowledge, values, attitudes	Self-assessed “local pride” Awareness of events in the country, the world Attitudes toward traditional culture, modernization Locus of control (I–E scale)a Reliance on telecentre services Value placed on telecentre as community facility
aI–E scale, internal-external-control scale, commonly used in psychological measures.

Some of these indicators are directly measurable; others are subjective indicators of attitudes and values. Subjective indicators are best measured indirectly on simple Likert scales, which the evaluators should pretest in focus groups or a small sample before undertaking the main surveys (see section 5). Again, not all these social-impact indicators will be useful in any one evaluation, and all will need to be defined more precisely in the local context. The evaluators should ask a panel of households or individuals to keep time-budget diaries. These will provide a wealth of information on changing social behaviour and patterns and can be rewarding for those who agree to keep them. Schoolchildren may be encouraged to keep time diaries as a school project. Others willing participants may be the members or leaders of local organizations who, with appropriate explanation, will understand the purpose and value of collecting these data.

3.5.3 Organizational impacts

Although many of the indicators already proposed for individuals and households can be used to measure changes in organizations, the importance of organizations to the development and life of the community and to the analysis of telecentres suggests that they should be accorded special attention in the evaluation. Therefore, a list of indicators of impacts on organizations is provided in Table 17. Formal organizations that are important to communities include institutions such as schools, chambers of commerce, and health clinics; businesses with at least one hired employee; and community organizations, NGOs, and committees with some formal structure and mandate. These all have some defined division of responsibilities, more or less explicit goals and objectives, and some hierarchy, usually.

Table 17. Indicators of impacts on organizations.
Change	Potential indicators
Operations	Use of databases, spreadsheets for financial, other administrative tasks Quality, timeliness of formal reporting Response time to fulfill requests, emergency response Use of registries, online expertise to carry out functions Use of off-site computer capacity to do work Use of reference libraries, downloaded software to improve performance
Networks	Networking within larger associations of member organizations Sharing information with other similar organizations Number of electronic networks of which the organization is a member Time, number of interactive discussion groups
Organization	Number, percentage of staff using telecentre, Internet Number, percentage of staff trained to use ICTs Implementation of the organization’s own information strategy Growth in activities, membership Ability to attract good leadership
Budget	Cost savings for information, communication functions Staff time savings for information, communication functions Investment in purchasing, leasing ICT equipment Change in revenue, expenditures
Perceived benefits, costs	Change in performance indicators Improved organizational structure, membership, leadership Dependence on telecentre to perform tasks Better networking Reported success stories Difficulties in keeping trained staff Financial costs
Outreach	Own website Number of requests, hits on website Production of electronic, print newsletters, bulletins Number of subscribers to newsletters, bulletins Percentage of outreach made available through fax, Internet, e-mail
Note: ICTs, information and communication technologies.

In practical terms, data on organizations can be obtained from formal records and people who can speak for the organizations. Usually, a telecentre will have a school, NGO, or library as a host organization, and most will have community committees associated with them.

Information and communication are critical to the success of any formal organization. Consequently, savings in time and money, together with better performance and reliability, are key issues for the evaluation. Like the indicators for individuals and households, those for organizations can also be direct measures of telecentre use and impacts and of how spokespersons perceive their costs and benefits. The impacts will relate to the efficiency of the organization, the outcomes it achieves, its decision-making processes and the decisions it makes, and the effectiveness of its networking and information sources in reaching its goals. As noted in section 3.4.3 (“Sectoral and local electronic networks”), formal organizations at the community level are already involved in electronic networking.

³ Performance indicators often refers more generally to how well a project is achieving its various objectives: this allows comparison of project performance with the targets and goals set out for it. For example, project performance indicators could be established to measure the extent to which a telecentre is helping to diversify markets for local entrepreneurs or providing particular information to women on child nutrition. These indicators are treated under “Impact indicators” (section 3.5) and “Content-demand indicators” (section 3.4.1), respectively. Telecentre performance indicators here refers more narrowly to the provision of services within the telecentre and the performance of the telecentre staff and equipment in providing those services. These have been treated separately, as they are a specific area of concern to telecentre programs in Africa, where telecommunication and other infrastructure are less available and reliable. Return

⁴ ENDA-TM is an NGO based in Dakar, Senegal. It is undertaking an Acacia-funded project to develop community-based and community-run telecentres in Senegal (www.enda.sn). Return

⁵ Tradepoint Senegal is an Acacia-funded project to make e-commerce available to small-scale entrepreneurs through decentralized access to ICTs in Senegal. Return

Document(s) 5 of 10