Document(s) 10 of 10

This section briefly introduces some of the methods and techniques that are most useful for studying and evaluating community telecentres. Its purpose is to help the reader select the most appropriate ones for a specific telecentre research project or evaluation study. As these guidelines can provide only a brief discussion of each technique, the emphasis is on their strengths, limitations, and suitability for various purposes. The reader may consult the bibliography for more detailed practical guidance on how to develop the instruments and apply the methodology.

Avoid the single-method solution

As a general rule, there is strength is having more than one method in any study. Some methods are clearly better suited to certain kinds of data and social situation, and reliance on a single method (usually a questionnaire survey) inevitably reduces the richness of the data and the possibility of cross-checking information. Data on how people use equipment is better observed than obtained later from an interview; why people wanted to use the equipment and how they felt after the experience can only be found out by asking them.

Match the method to the available human resources

Other important considerations include the time and resources available to the research project and particularly the availability of trained researchers and field workers. Some of the methods described here should be implemented by researchers with specific training in using them, at least in design, coding, and analysis. This includes the use of projective techniques and attitude scales. Other methods are more robust in their application, such as observation, performance reports, and self-assessments. Group techniques require a facilitator trained and skilled in leading group discussions. Questionnaire surveys are more difficult to design well than most people believe, but field assistants with limited training can effectively carry out a well-designed survey instrument in the field, and a well-designed survey instrument is the key to good data collection.

Match the method to the type of data needed by the stakeholders

Another consideration is the type of data needed by the various stakeholders. At the local level, highly statistical information is probably less useful than the more in-depth, qualitative kind that enables education and learning. However, potential investors in telecentres and international donors may require data with provincial or national validity, so a good sampling design is crucial. They may require financial data with statistical significance. The mix of stakeholders and their information needs will influence the research design, sampling strategy, and mix of methods.

5.1 Performance reports

One of the most cost-effective and valuable sources of information for both marketing the telecentre and evaluating its performance will be records or daily logs, so, from its opening, the telecentre should institute a regular monitoring system as part of the operator’s duties. Activity records can be filled in by telecentre staff or by the user, or they can be fully automated on the telecentre’s equipment. In all cases, it is important to maintain regular weekly or monthly tabulations, summary reports, and reviews. These will give the telecentre operator and management ongoing feedback on performance, show them where the problems lie, and suggest possible improvements. Without a regular review process, the accumulated data serve little purpose and soon become too daunting an accumulation to process.

Telecentre equipment can generate records that are useful in evaluating performance and financial sustainability. Jensen (personal communication, 1998⁶) proposed breaking down services into three types (Table 18). He suggested that, where the equipment is available, an automatic “till” should be the core of the telecentre record-keeping system, with each type of service prerecorded for a key, which when pressed, prints on the bill the cost of each service provided to the customer or records every transaction in the system for daily, weekly, or monthly reports for the telecentre operator. Such an automated system could also readily record and display varying rates for peak and off-peak hours and discounts for special groups or individual customers.

Table 18. Types of service records to be monitored in telecentres.
Record type	Applied to
Hourly rentals (peak and off-peak tariffs; special rates for certain groups)	Meeting space Television Audiovisual equipment, overhead projector, cameras, etc.
Unit sales	Fax pages Photocopy pages Postal services Stationery and other goods sold
Service minutes (peak and off-peak rates; individual and group rates)	Telephone calls Internet access Video-conferencing PAT for help in using equipment Business services (typing, web searches, spreadsheets, etc.) Provision of government information Training Book loans
Source: Mike Jensen, Consultant, South Africa, personal communication, 1998. Note: PAT, personal assistance time.

Telephones systems can also provide records of the time and duration of outgoing local, long-distance, and international calls, and the telecentre can charge for these calls at various rates, depending on the time of day and day of the week. In many telecentres, incoming calls are an important service not recorded by the telephone company, and the telecentre operator may have to record and charge for these calls. In some countries, itemized phone bills are available, which makes the record-keeping easier. In general, regular monitoring and log-keeping should demand as little of the time of the operator and staff as possible.

Computer systems can record online and offline usage and the number of pages printed and e-mails sent and received for all regular users who have an account and for occasional users with guest accounts issued by the administrator. Researchers can analyze these data for various pieces of equipment and users, including use of caches and bookmarks.

Administrators can also ask users to fill in a report on each visit. An automatic login screen would ask them to login using their telecentre user identifier or password, rather than their name. The logout screen could be designed to ask them for additional information, such as why they use the service and how satisfied they are with it. The alternative would be to ask users to fill in a sheet or book to report on the services they used (and even the staff help they received) when they pay.

Ideally, the system would automatically cover every user and every visit, providing the basic information on services used and income received that the telecentre needs to run as a business, as well as collecting information on the user, such as gender and address. One major concern here is with the telecentre users’ right to privacy, and each research team and the management of the telecentre would have to discuss this issue in its local and national contexts. However, log books are already commonly used in some African telecentres, and customers are accustomed to filling these in on each visit.

More service-oriented and evaluative information can be (sparingly) added to such a standardized recording system, or, more likely, can be obtained through voluntary user surveys conducted at the telecentre on particular days or on a sample of all users.

5.2 Questionnaires

Many books and guides provide advice on developing questionnaires for various purposes in various contexts. Some are listed in the bibliography. This section focuses on the design choices that should guide the selection of types of questionnaire, format for interviews, and types of questions. (Section 5.3 will discuss more advanced techniques that can be used for questionnaires to obtain information on how people feel about situations, rather than what they know about them.)

5.2.1 Choosing a questionnaire format

Questionnaires are either self-administered by the respondents or given by an interviewer. For those in which the respondents fill in the answers themselves, the layout and instructions must be clear, so that errors are minimized. For those filled in by the interviewers, the instructions may include probes (or supplementary questions to be asked) and codes (for initial analysis of the responses) (Table 19). All questionnaires must be pretested to ensure that they are clear, that the question order seems logical to the respondent, and that the questions and wording are presented without bias, offence, or ambiguity. Pretesting can also familiarize the interviewers with the questionnaire.

Table 19

Questionnaires are usually classified as structured, semistructured, or unstructured. Structured questionnaires usually have questions with an anticipated range of responses, so the answers can be coded or scaled beforehand. Researchers use structured questionnaires when they are reasonably confident that they know the range of answers and can therefore “close” the questions to limit them to certain responses. Unstructured interviews have mainly open questions, without any limitation on how the respondent should answer, and these interviews usually have a schedule of questions. But the interviewer may vary the order of topics to follow the lead given by the respondent, to make the interview more like a natural conversation.

The most unstructured interviews are sometimes called key-actor or key-informant interviews. Semistructured interviews are a combination of the two: they combine the advantages of each type as appropriate to the various topics covered in the interview. Thus, structured questionnaires have a specified order for the questions, and the majority of its questions are closed or precoded. With typically unstructured questionnaires, on the other hand, the questions are open, the responses are recorded verbatim, and the order of the questions varies. Both structured and open questions can refer to the present, the future, or the past, and in this way the questionnaire can retrospectively probe for past behaviour and events and prospectively probe for future intentions.

Questionnaires can be made more structured after sufficient pretesting has indicated the range of responses that can be expected from 90% or so of the sample. The advantages of structured questionnaires are that they can usually be administered more quickly and are less subject to interviewer bias and coder error. Structured questionnaires are used to treat large samples and large amounts of data, as they are usually the most cost-effective. They are used to gather purely factual information, rather than information on how people feel about sensitive issues.

Unstructured questionnaires and open questions are used if the answers are not known or categorized beforehand. The interviewer writes down the response verbatim. Later, the researchers will have to list these verbatim responses (or a reasonable sample of them) and construct coding categories that are relevant to the study and fit the majority of cases. To ensure consistency and reduce coder bias, at least two independent “judges” should be involved in developing the coding categories and in coding the responses under these categories.

The researchers should clearly use the unstructured approach if they do not know the range of responses before the survey and they expect to obtain much “richer” data from the variety of answers they record. The downside is that the interview and coding phases take longer and need to be administered by more experienced and better trained people. In practice, the research leader often takes responsible for the initial development of coding categories, so that he or she can get a feel for the data and how well the survey interview is working.

In questionnaire surveys, the researchers treat each interview as a unit of analysis and give equal weight to each one (unless they expect to do some statistical weighting later with subsamples). In key-actor interviews (sometimes called elite interviews) or unstandardized interviews, the researchers may decide not to treat the respondents’ answers equally. Some respondents will be better informed or more influential, and their responses will have to carry more weight in the analysis. Thus, whereas evaluators would handle a unique or different response statistically in a questionnaire survey, they may give importance to an unusual response in an “elite” interview over and beyond its statistical frequency. Also, in elite interviews the interviewer tries to let the respondent lead, even to the point of allowing the respondent to define the situation in his or her own terms. The aim is to have something that sounds like a discussion but is, in fact, a quasi- monologue by the respondent (Dexter 1970).

5.2.2 Selecting between alternative question formats

Question formats parallel those of questionnaires: questions can be open or closed, with precoded open questions falling somewhere in between (see Table 19). Using precoded open questions, interviewers gather a verbatim response and do not constrain the respondents. Moreover, pretesting these questions enables interviewers to code most responses directly, thus simplifying data processing. As a general rule, open questions on any topic are asked before closed questions so that the coded categories do not influence the open-question responses.

An example of a forced-choice question is given in Table 19. Such questions require the respondent to select from two or more alternatives the one that comes closest to their own situation or opinion. The alternatives must be simple and roughly opposite on some relevant dimension. These questions are sometimes difficult to administer because respondents feel that none of the alternatives offered fits their situation, and they need encouragement to select the one closest to it. Thus, their response is literally “forced.” These questions are mostly used if the researchers want the respondent to consider the alternatives and “select sides.” They have been used in personality measures and can be seen as a two-point-scale question.

Scaled questions require respondents to indicate their degree of agreement or disagreement with a given option by assigning it a value on a scale (Table 20). The scales (sometimes called Likert scales) vary from three points to a continuous line (usually 100 mm long) on which people mark their position for or against a particular item. Researchers can then simply measure this mark and convert it to a percentage. Scaled questions have an important advantage for analysis: they provide ordinal, rather than nominal, data. When properly constructed, the scales (such as attitude scales) can provide interval data and therefore more interesting possibilities for analysis.

Table 20

Also, most scales with verbal labels have three to seven points. More complex scales make the rating task too difficult for the respondent and do not provide ratings that are any more accurate — probably the opposite. The advantage of labeled scales is that the labels can be read out by an interviewer, and this format is therefore suitable for interviewer-administered questionnaires. The wording of the labels is known to influence the respondent’s ratings, so these questions require considerable pretesting (Whyte 1977). This is why linear scales with labels only at each end of the scale are popular. However, these scales require the respondent to self-administer the questionnaire and be able to visually measure proportions along a line. Such questionnaires may also appear to be more accurate than they are, as people probably don’t make such fine distinctions for most topics. Nevertheless, this method can be successfully used in group sessions; with group participation, the facilitator can mark the line, leaving more time for discussion.

5.2.3 Putting them together

The best questionnaire has varied question formats and maintains an interesting flow of topics. It should have a logical sequence from the perspective of the respondent, and each question should be clear and understandable. Sample groups used to pretest questionnaires must have characteristics similar to those of the respondent group. Without pretesting, it is simply impossible to anticipate all the ambiguities, conflicts, and difficulties that the wording, presentation, and order of questions will present in the field. Questionnaires have demonstrable order effects. To cancel out the order effect over the entire sample survey, researchers should develop two or more versions of a questionnaire, with the same questions occurring in different order. Similarly, scale questions have known order effects. For example, the right-hand label of the scale or the last-mentioned alternative is the most likely to be selected. The order of labels on scales and “positive” versus “negative” statements should therefore be randomized or at least varied.

Generally, questions on personal information, such as age, education, and income, appear near the end of the questionnaire, when the interviewer has a well-established rapport with the respondent. If information might be sensitive, questions can be devised to elicit this information in terms of ranges, rather than specific numbers, such as an income range (4 001–5 000 Ugandan shillings) or an age range (30–45 years). Ideally, the questionnaire should include some internal and external cross-checks to evaluate the validity of the data. An internal cross-check might be provided by asking two questions, spaced well apart, essentially requiring the same or consistent information but in different ways. An external cross-check might be provided by asking a question requiring data obtainable elsewhere, such as from a census, another survey, or telecommunication records.

5.3 Projective techniques

Projective techniques can help to obtain freer, less self-conscious responses, based more on feelings than on knowledge. They have been developed in research psychology and clinical psychiatry, and some can be adapted and simplified for use in surveys or group situations. Essentially, the techniques enable the respondent to “project” their own thoughts and feelings onto another person or organization, identified verbally or pictorially in the question. In contrast to the projective techniques used in the laboratory or the proverbial clinical “couch,” those in the field must be more superficial and simple enough for interviewers with a little training to administer. These techniques can give insight into people’s perceptions, attitudes, values, and personalities, which can in turn reveal patterns of behaviour and community dynamics. Such techniques may be less familiar but are worth considering as one ingredient in the methodological toolbox for studying telecentres.

Projective techniques range from simple word association to a request to create a story out of a given lead idea or picture or to play a role in a gaming situation. Some are biased toward literate respondents, and others require respondents to work with paper and pencil, and they are therefore more suited to group situations and key-actor interviews, rather than questionnaire surveys. Others, such as the semantic-differential test, take too long to administer in most field situations (the semantic-differential test will not be described here, although it provides interesting information on what word labels actually mean for people). The projective techniques introduced below are simple and have been used successfully in developing countries and field situations, but it should be noted that they all require pretesting in the field and in the language of the interview.

5.3.1 Adjective checklists

Adjective checklists are simple lists of adjectives presented to the respondent to describe any situation or topic, including the telecentre, the frequent users of the telecentre, or some other physical or social aspect of the community. The adjectives should be developed from pretests in which respondents provide open responses describing the attributes of the telecentre and community and express a range of feelings and characteristics of interest to the project. Don’t make the list too long or it becomes a wearisome task for the respondent, and don’t use the technique more than once or twice in the same questionnaire.

5.3.2 Sentence-completion tests

In the sentence-completion test, the respondent is asked to complete a sentence. For example, the sentence stem might be “When I think of family members who are working abroad, I … ” or “When I want to get news from town, I … .”

The stems are read out by the interviewer, and he or she records verbatim and later codes the respondent’s sentence completions (see example in Table 21).

Table 21

But constructing scales is rigorous and time consuming.

Only an experienced researcher can establish the initial categories for coding. Sentence-completion tests are one of a series of projective tests, ranging from word association to paragraph and story completion, and are suitable for questionnaire surveys in different parts of the world. They enable the respondent to answer freely, once the subject has been set by the sentence stem, and can reveal significant differences between individuals and between social groups. In a survey situation, one uses about six sentence stems, which means that the design of the survey can include a cross-check for internal validity. All projective tests, including sentence-completion tests, must be field tested to ensure that they provide effective measurements and that respondents understand and accept them. Sentence-completion tests do not work in some cultural situations and in some languages.

5.3.3 Scenarios

Since Kahn and Wiener (1967) popularized the use of scenarios in their book The Year 2000: A Framework for Speculation on the Next Thirty-three Years, scenarios have been best known for their use in foresight exercises. A scenario is defined as “a hypothetical sequence of events that focuses attention on causal processes and specific decision points.” Scenarios are not forecasts but plausible stories describing the future. Used in foresight exercises, they are structured to address

• Current issues, trends, and events of interest to research or policy;

• Determinable and somewhat predictable elements in the environment; and

• More uncertain elements (trend breakers, turning points, or weak signals of change).

A well-constructed scenario presents an internally consistent story about the path from the present to the future. A scenario is relevant to the issue of interest and the group, recognizable from the current perspective, and challenging: it contains some elements of surprise or novelty requiring the group to stretch its vision. Ideally, a scenario finds the balance between “thinking the unthinkable” and being grounded in the reality of today, so that it is believable (that is, either possible or probable). Scenario-building is a useful group exercise to help people imagine alternative futures for their organization or community. Doing the exercise is often as valuable as the results.

Surveys or focus groups can also include simpler and usually more structured scenarios. One advantage of scenarios is that people generally enjoy them and appear to readily “project” their own attitudes and feelings into the story. The challenge with scenarios is in the coding, and one way to ease this problem is to structure the scenarios so that they have three or four alternative outcomes and the interviewers ask the respondents to select the one they prefer or think the most likely to occur. This constrains the respondents’ freedom of expression but presents them with ideas they may not have thought of before, so it is a valuable technique, not only for use in group situations, but also for educating people.

5.4 Attitude scales

Attitudes are one of the most well-known and empirically investigated psychological concepts. They are defined as mental and neural states of readiness that are organized through experience and influence our responses to objects and situations (Allport 1935). Attitudes have affective (feeling and emotional), cognitive (thinking, mentally organizing), and behavioral components. In terms of stability, they are assumed to come somewhere between values (long term) and opinions (transitory). They are measured in terms of their direction (positive or negative attitude toward the object), strength, and consistency. Consistency as a measure is rooted in the theory of attitude formation and change, in which people are thought to seek balance or congruity in their attitudes.

Part of the reason why the concept of attitudes has become so well known is that researchers have given so much attention to measuring them, with the result that hundreds of attitude scales are available “off the shelf.” These are valuable because the correct construction of an attitude scale requires considerable time and care, and researchers need to validate it on various populations to test whether it consistently measures significant differences in attitude. The advantage of using an attitude scale is that, if properly constructed, it provides interval data, rather than nominal or ordinal data. In questionnaire surveys, they are more commonly used in shortened form, and the researchers analyze the results as ordinal data. These data are easy to analyze because the questionnaires are already constructed for rapid scoring. However, attitude scales are highly situation and culture specific; researchers need to design or test them, or both, in the local cultural context. They are useful for measuring differences in attitudes toward a telecentre between various groups, for example, or any change in the community. Used in baseline and follow-up surveys, they can also measure how these attitudes change over time and how people’s views of the new technology change with experience.

An attitude scale is a collection of statements, and the respondent indicates to what extent he or she agrees or disagrees with each statement. The attitude scale can have any one of the formats for scaled questions shown in Table 20. The statements should be short, contain only one idea, and use unambiguous language appropriate to the target population. Researchers should select the final statements used in the scale from a much larger number on the basis of pretesting to calculate the strength of each statement for or against the object. The final scale comprises those statements that appear to cover the range of attitudes found in the study group, and ideally they are “spaced” equally apart in attitudinal distance and consistent in discriminating between people. When they are read out by the interviewer, rather than being read by the respondent in a paper-and-pencil test, they need to comprise simple, easily remembered statements. Further reading on constructing and using attitude scales is provided in the bibliography.

5.5 Observation techniques

Asking questions is probably the main social-research technique, but a great deal can be learned from simply observing people’s behaviour during interviews and using the interviewer’s observations as data. Survey questionnaires can include questions for the interviewer, such as on house location, housing quality, or social interaction within a family or group. One way to reduce observer bias in interviewing is to use more than one observer and compare their findings. Some of the same issues in sampling and surveys arise with respect to observation techniques. The researcher needs to decide on the unit of analysis and the sampling location and time.

Generally, one selects the locations for observation points purposively, rather than randomly, because they depend on the purpose of the study, such as the evaluation of a telecentre, commercial phone shop, bus station, or market. One usually chooses the sampling time to ensure that various seasons, days, and times of day are included in the observation schedule. The duration of sampling will typically vary from a few hours to a whole day. Participant observation (see section 5.7) can have a more or less continuous observation period. Individuals and groups can also be selected as observation units, although here the issues of privacy become more important, and interviewers should obtain some form of informed consent. In a telecentre, the unit of observation can be a particular piece of equipment, a staff person, a group of schoolchildren visiting the telecentre, or the telecentre itself.

What kind of information could be better obtained through observation than through interviews? Principally, it is information that the respondent might not be aware of, particularly recall, or wish to divulge. This may include information about his or her interaction with the equipment in the telecentre or with staff or other users, such as how long it took them to complete their task, how often they had to try to make something work, or how often they had to have help. It may include the levels of noise and distraction at the telecentre, how crowded it was, how long people had to wait, what they did while waiting, how well maintained the telecentre was, and whether the environmental quality was acceptable.

Observation techniques can be structured or unstructured. After pretesting the questions, the researchers can develop structured observation schedules and train observers to use them to make simultaneous observations at various observation points. A formal approach to structuring observation is “behavioral mapping,” which identifies all the behaviours in a specified area or building. Repeated observations can show how people are using a new service centre, such as a telecentre, and how improvements to its physical lay-out and its hours of service can improve its function and efficiency and the ways people use it. Techniques for behavioral mapping require the researchers to undertake pretesting, select observation sampling points, establish coding categories, ensure cross-checks for validity and consistency, and conduct the field work.

Observation techniques usually demand that the observer does not influence the subject’s behaviour any more than is absolutely necessary. Observers should be part of the background. The longer they are in the background, usually the less they will affect people’s behaviour.

Sometimes direct observation is neither possible nor desirable, so researchers have to use indirect methods. Indirect observation usually measures behaviour through its impact on the environment, for example, changes in the width and wear of various tracks, as determinants of the paths people most commonly use. Indirect observation usually measures one or more of the following:

• Erosion measures (for example, wear on office equipment and furniture);

• Accretion measures (for example, dust on unused equipment, books); and

• Archival records (for example, sales records, paper used, phone logs).

5.6 Group techniques

Most of the Acacia evaluation studies of community telecentres plan to include some group processes to enable organizations, interest groups (such as farmers, youth, and women) and small groups of users and nonusers, or just community members, to discuss aspects of telecentre operations and impacts important to them. Group processes are valuable sources of information for the researcher and provide learning processes for the participants, as they are exposed to a wider spectrum of ideas and views. A number of techniques are available for researchers to use specifically with groups. Three of them are discussed here: focus groups, nominal groups, and Delphi techniques (see section 5.6.3). In addition, the researchers can use the projective techniques in section 5.3 successfully in group settings to elicit people’s feelings and attitudes toward telecentres, communication and information, and community processes, generally.

5.6.1 Focus-group techniques

Researchers can use focus groups to gather qualitative data to compare with survey data, but they should not compare them statistically with survey results. Qualitative data are particularly useful for exploring particular issues in greater depth, including people’s feelings and beliefs; identifying differences between groups within the community; and developing follow-up messages and education. A focus-group discussion or interview is particularly good for generating ideas and providing feedback.

A focus group usually comprises 6–15 participants and focuses on a few key topics or questions. In the discussion, the participants talk among themselves, and the facilitator intervenes as little as possible. In a group interview, the facilitator poses the questions and may have a more prominent role, but the process is very similar. The key is to have good facilitation and record keeping; thus it requires two people to run the sessions, even if the discussion is tape recorded. The facilitator or moderator should have been trained and be able to keep the discussion lively, meaningful, and on topic, as well as ensuring that everyone has a chance to speak. A facilitator will need to encourage people who are reluctant to voice their opinions or who feel marginal to the group, and the facilitator may at times need to hold the more loquacious ones in check. The richness of the data comes from the debate, and here the rapporteur must take excellent but structured notes on the range of opinions and the strength of views, as well as who seems to be leading and following. Experience has shown that it sometimes takes one expressed opinion to elicit another, contrary opinion, and thus a wider range of views may be expressed in a group situation than in individual interviews.

In some focus-group discussions, where some follow-up action is expected, the facilitator needs to help the participants find some middle ground or consensus, whereas in other focus-group discussions the participants can be left to simply provide a range of opinions. The CIET evaluation of telecentres in South Africa plans about 500 focus-group discussions and has developed a technique to make the process both quick and effective. Focus-group participants are selected purposively, as meeting specific criteria and as being stakeholders or members of a target group and thus being expected to have an interested in the topic and to have something to say about it. Researchers should hold focus-group discussions with all key-stakeholder groups, all key institutions and organizations in the community, and informal groups of users, nonusers, women, youth, farmers, and other economic groups like small-business people.

5.6.2 Nominal-group techniques

You use nominal-group techniques in face-to-face meetings where you want people to think about a question by themselves and then work on the responses as a group. This is both a rich and efficient way to generate ideas and obtain group input into the evaluation process or ranking of ideas. The group should be small (5–10 participants), and a single session usually takes 1–2 hours.

The participants sit around a table with a leader or facilitator, who opens the meeting by reading a question aloud to the participants. Each participant has a worksheet with the same question written at the top, and they take 5–10 minutes to write down their ideas, without discussing them among themselves. In the next stage, the facilitator goes around the table and asks each participant to contribute one of her or his ideas. These are written down and numbered so that everyone can see them. The process continues around the table until all ideas are “on the table.” The group then discusses each idea in turn, so that everyone understands it and they all share their views on it, but the group makes no attempt at this stage to resolve differences of opinion.

The next stage is to rank the ideas. When participants provide more than 10, usually the group selects the top 10 and then each participant ranks them, usually on cards. The facilitator reads out the cards, without identifying the participant, and then records a tally of all the votes. The group then discusses the ranking and may seek further clarification of some ideas or may even collapse two ideas into one. At the end of the process, the participants repeat their individual ranking of ideas on cards, and the facilitator tallies the scores to obtain the final group ranking.

5.6.3 Delphi techniques

Delphi techniques derive from three major findings on group processes. One is that assessments made by a group of people are more likely to be accurate than those made by the same individuals working alone. Second, a few individuals tend to dominate face-to-face meetings, and information is processed less efficiently. Third, people who receive information about the range of individual responses (including their own) to a particular question use that information to improve their own response. In this way, the quality of the assessment improves in each successive round after the participants have seen the results of the earlier round and can recast their own responses. Delphi techniques are thus designed to avoid the distortions of interpersonal processes within face-to-face encounters.

With Delphi techniques, researcher use regular mail or e-mail to send a list of questions or “items” to members of the group for them rank or scale, and they anonymously fill in their answers or ranking and return the list. The composite list of responses from all the members of the group is then circulated back to participants, and they rerank or reanswer in the light of the distribution of responses in the first round. They may also provide some explanation of their rankings or responses. In some Delphi processes, the participants rank their own expertise on the subject, and the views of those with greater expertise carry greater weight in the group rankings.

Through a series of rounds (usually two, because of the time and expense), the participants can reach consensus without seeing one another or knowing whose response is whose. Some Delphi processes aim to reach consensus, whereas others aim to generate as diverse a range of opinions as possible. In a “decision Delphi,” as the name implies, the aim is to reach decisions among stakeholders with different interests in a solution, when the issue is a divisive and contested one.

These techniques are more elaborate and time-consuming than nominal-group techniques, but the group does not have to meet face to face. In the case of the evaluation of telecentres, they may be more useful in eliciting the views of national and international stakeholders than those of community members; for community members, researchers can use simpler techniques.

5.7 Participatory and self-assessment approaches

In the methods discussed so far, the roles of the investigator and the respondents are clearly defined. Participatory research methods, in which the researcher is both investigator and participant, are increasingly used in social research, especially in community-development studies. As an approach, participatory methods focus more on the richness and validity of the initial data set than on the manipulation or analysis of data. Participatory research assumes that the researcher, who is also a participant in the action, will have access to more data and will be able to interpret them more meaningfully, even from the point of view of a respondent. The social relationships between the researchers and the respondents become more important and introduce more bias into the data than during an interview, as these relationships will have lasted longer and be more important to both parties. As a methodology, these methods make it difficult to cross-check the data, with the result that confidence in the reliability of the research data depends on the experience and skill of the participant observer.

The anthropologist Oscar Lewis related an anecdote about the individual differences in participant observers’ interpretations, when contrasting his study and that of Robert Redfield on the same community, Tepoztlan, Mexico:

The impression given by Redfield’s study of Tepoztlan is that of a relatively homogenous, isolated, smoothly functioning and well-integrated society made up of a contented and well-adjusted people. His picture of the village has a Rousseauan quality which glosses lightly over evidence of violence, disruption, cruelty, disease, suffering and maladjustment. We are told little of poverty, economic problems, or political schisms. Throughout his study we find an emphasis upon the cooperative and unifying factors in Tepoztecan society. Our findings, on the other hand, would emphasize the underlying individualism of Tepoztecan institutions and character, the lack of cooperation, the tensions between villages within the municipio, the schisms within the village and the pervading quality of fear, envy and distrust in interpersonal relations.

— Lewis (1951)

In fact, both observers — both experienced anthropologists — were right. Their own perceptions led them to emphasize various aspects of the community and seek out informants who also emphasize those aspects. Participant observers have to work hard keep their own preferences and allegiances in check and their observations neutral. It sometimes becomes impossible for the researcher to remain aloof, and many participant observers experience swings of emotion and identification with one group or individual over another as they observe the events of each day. It is important to also record the feelings of the researcher, as they are a necessary adjunct in interpreting the data. Field notes should be recorded soon after the event or at least daily, even if they are reinterpreted in light of later events.

Participant observers usually make extensive use of key informants, who can provide more insight into the research situation. Informants are often self-selected, in that they volunteer to help the researcher and should be chosen with care. Informants can have an axe to grind and have been known to be marginal to the group at the outset. To reduce the bias generated by key informants, it is useful to have informants from different groups in the community and provide them with some understanding of the research objective and the notion of objectivity.

Self-assessments range from those fully controlled and implemented by the organization or group involved to ones in which the organization commissions an external evaluator to undertake the work, with the organization or group as full participants. Several useful guides to doing self-assessments are available to enable organizations without previous experience or skills to undertake them. Generally, the starting point for doing a self-assessment is either a full-scale review of the organization or one with a focus on a specific problem. Typical reasons for doing a self-assessment relate to strategic decisions, such as those regarding organizational strengths and weaknesses, possibilities for growth or change in the organization’s mission and objectives, staffing, and finance (typically the need to raise new funds).

Before the organization or group starts the self-assessment, it is important for the executive or members to engage in a participatory process to agree on the following:

• Its purpose;

• Its scope;

• The data to collect;

• The key issues; and

• The cost and who will pay it.

This enables the organization or group to measure its readiness for doing a self-assessment and using the information it generates. The organization or group will require several types of readiness at the outset (Lusthaus et al. 1999):

• Cultural readiness — The culture of the organization or group should be such that it is open to suggestions for change or improvement;

• Resource readiness — The organization or group should have the resources (people, time, technology, money) to do the self-assessment and be prepared to commit these resources to this task;

• People readiness — Staff should be prepared to work together on the project;

• Leadership readiness — The leadership should be prepared to champion the process and provide it with the necessary support;

• Vision and strategy readiness — The group should have discussed their vision and strategy beforehand; and

• Systemic readiness — The organization or group should have the systems in place to provide the information needed for the assessment.

5.8 Household budgets and diaries

It may be valuable to have some more detailed information on certain aspects of behaviour or decision-making than can be reliably obtained through questionnaire surveys, observation, or group techniques. Another approach is to select a small subsample of people or households and ask them to keep diaries or daily logs of activities or expenditures. Clearly, someone in the household needs to be able to write and fill in the information or someone in the research team must make daily visits to maintain the record. The task should be clear, simple, and quick, or people will forget to fill in the information or make errors. For example, if a record of household expenditures requires respondents to decide under which category to place a particular item, they are less likely to record it and more likely to make errors in categorization when they do record it. The simpler the task is for the respondent, the better are the results. Thus, asking people to simply make a record of their phone calls and the letters they mail and how much they cost is better than giving them a complex chart to complete.

Economists are interested in financial accounting, but equally interesting for the social researcher is how people spend their time. Time budgets are a useful adjunct to household surveys to provide more information on how much time people devote to work, social interaction, home activities, etc. In the case of telecentre studies, it is more interesting to see whether the telecentres save people significant amounts of time. Anecdotally, it is understood that many people, such as head teachers, business owners, and organizational leaders spend considerable time traveling to town to order supplies, deal with government departments, seek information, etc. Asking some of these people to keep time diaries for selected periods before and after the telecentre opens would help to quantify its impacts on the amounts of time people devote to various tasks.

When designing expenditure logs, it is important to keep it simple for the respondent and to leave the coding for the researcher to do afterward. If time diaries are used to record all activities, the diaries are usually kept for 24-hour periods dispersed over various days of the week and times of year to capture any periodicity or seasonality in behaviour patterns. If the objective is simply to record all behaviour related to patterns in information searches and communications, respondents will have less to record and they can maintain the diary for a longer period. In all cases, it is important that the respondents understand why they are doing this and why the information they may feel is so mundane is actually valuable to the research or evaluation.

One place where daily logs of activity should be kept is the telecentre, and the operator should see record-keeping as a way to improve service and performance and as integral to the job.

⁶ Mike Jensen, Consultant, South Africa, personal communication, 1998. Return

Document(s) 10 of 10