I first met Gelia Castillo when she was a member of the Board of Governors of the International Development Research Centre (IDRC), where I worked for several years as head of the economics program. One of the themes that ran through her commentaries was the need to ensure that the research we supported was used in decision making. On IDRC's Board, and on many others, she has continually reminded us that development research should be in the service of the poor and should result, not just in books and articles, but in social change.

This paper attempts to respond to some of the questions Gelia has confronted us with: How can we improve communications between researchers and policymakers? What can be done to increase the utilization of research results? And how can research meet short term needs without compromising its role in the development of new concepts and ideas?

Outline

Summary
Introduction
"Demand-side" Problems of Utilization: The Policy Making Process
Differences in the Logic of Economists and Policy Makers
"Supply-side" Problems of Academic Research
Research Utilization in Developing Countries
Guidelines for the Enhancement of Research Utilization
Client-oriented Research and Research Brokers
Criticisms of Client-oriented Research
Alternative Uses of Social Science
Implications for Research Funding Agencies
Reference
Notes

Summary

This paper examines the relevance for developing countries of the largely US-based literature on the utilization of social science research in public policy making. Many characteristics of bureaucratic decision making impede a science-based approach and call for a style of policy research that takes greater account of political and administrative feasibility. At the same time, the importance of 'basic' social science research in defining problems and developing analytical concepts is affirmed. The paper concludes with recommendations for agencies financing research in developing countries.

INTRODUCTION

The 1980s saw a heavy emphasis on economic policy making. As macroeconomic crises shifted attention from sectoral to national and international issues, the quality of policy making was given increased importance as a factor in promoting stabilization and growth. A frequent observation is that policy making could be substantially improved if it were based on better information and relied more on the principles of analysis and evaluation developed by economists and other social scientists.

Current discussions of this subject in developing countries might benefit from an examination of the literature on research and policy making during an earlier period in the United States. Under Presidents Kennedy and Johnson, the federal government launched ambitious social programs and at the same time attempted to increase the 'science base' of policy making. It attempted the latter by transferring a system of Planning, Programming and Budgetting from the Defense Department to other departments and by funding advisory and evaluative research related to those social programs. Subsequently, a wave of studies (most notably Weiss, 1977 and Lynn, 1977) assessed the impact of that research. The findings were disappointing - social science appeared to have had little direct impact, measured by the direct adoption of specific recommendations. However, these studies did identify basic differences in the ways in which policy makers and academics analyze problems and make decisions. They also identified the more fundamental contributions of social science as 'research for knowledge' rather than 'research for action'.

This paper surveys that literature and highlights its importance for developing countries. It examines the differences between policy makers and researchers, and recommendations for overcoming those differences. It then critiques the recommendations and discusses alternative concepts of social science's impact. It concludes by examining the implications for agencies financing research in developing countries.

'DEMAND-SIDE' PROBLEMS OF UTILIZATION: THE POLICY MAKING PROCESS

Four aspects of the policy making process are frequently incompatible with the utilization of social science research: policy objectives, the timing of decisions, who takes decisions, and the decision-making process.

Policy objectives. Rigorous analysis requires a clear definition of a problem and the variables to be measured. Government policies and programs are not often amenable to such analysis because they tend to have loosely defined and multiple, even contradictory, objectives. Stated and real objectives may differ. Furthermore, the relationship between means and ends is not simple. In policy making, as elsewhere in life, ends are not always chosen first. "Ends are chosen that are appropriate to available or nearly available means"; they are not fixed, but explored, reconsidered and modified (Hirschman and Lindblom, 1962). Finally, the pervasive role of government in society and the increased politicization of ethical issues (e.g. population policy, human rights, genetic engineering) have brought highly value-laden issues into the political arena; these are not easily amenable to research or evaluation (Rose, 1977).

Timing. For a variety of reasons, the need for research often becomes apparent too late. Because of inertia and more urgent priorities, governments are usually not receptive to suggestions for improvements unless there is a serious and self-evident problem. They tend to think about changing policies only when time and funding have run out; at that point, it is too late to carry out research (Wilson, 1978).

Timing usually pre-empts evaluative research. New policies and programs are usually launched in a research vacuum, partly because ex ante appraisal techniques tend to be less reliable than those for ex post evaluation. The latter requires an existing program to study (Sundquist, 1978). Furthermore, it is only after a program has been established and a clientele created that an effective demand exists for research (Lynn, 1978). For these reasons, policy implementation tends to precede rather than follow research.

Who makes decisions. The model of a researcher advising a decision maker applies only weakly. In most cases, decisions are arrived at through multilateral bargaining. Even in cases where a client agency requests advice, there is no guarantee that it will be the appropriate audience for the results (e.g. a study done for the Ministry of Education which finds that the principal bottleneck to better student performance is poor nutrition: Weiss, 1978). This problem has led some observers to recommend that policy analysis focus on a policy area rather than on specific agencies (Lamb, 1987), although it is not clear how this would be operationalized. Furthermore, policies are 'made', in varying degrees by many actors, including senior civil servants, technicians, advisers and technical assistants (sometimes foreign); it is incorrect to think that only cabinet ministers are 'policy makers'. Finally, (1977) many 'policies' are not the result of conscious decisions at all; they are simply the sum of previous ad hoc actions and inaction (Weiss, 1977). As a result of these conditions, it is often extremely difficult to identify a client for research.

The decision making process. The loosely defined and often inconsistent objectives of many government policies result from the process by which they were formulated. In multilateral bargaining, it is often impossible to obtain consensus on anything more than broad statements of principle; accuracy and specificity must be sacrificed (Rose, 1977). Furthermore, these bargains would not hold up if the costs and tradeoffs involved were made explicit (Verdier, 1984) 
see note 1. Research that examines the ex ante feasibility or ex post achievement of objectives is thus extremely difficult, and analyses that highlight costs and tradeoffs are threatening. Research always has the potential to upset delicate agreements; to take debates out of political back rooms where they can be controlled by the actors involved (Verdier, 1984); and to generally reduce the freedom of policy makers who request studies and then find themselves under pressure to follow unwelcome recommendations (Davis and Salasin, 1978).

Finally, there are practical problems that prevent decision makers from making better use of research. Often, governments are afflicted with too much information; attempts to absorb the data already available can delay and complicate decision making (Sharpe, 1977). Furthermore, most senior policy makers have very little time to read: the average US congressman works an eleven hour day, of which eleven minutes are spent reading (Verdier, 1984).

DIFFERENCES IN THE LOGIC OF ECONOMISTS AND POLICY MAKERS

Some of the objectives and values of economists are particularly divergent from those of policymakers. To the extent that their objectives are identifiable, policy makers tend to emphasize distributional concerns (i.e. winners and losers); economists emphasize efficiency. Policy makers tend to define goals in (sometimes arbitrary) quantitative terms rather than in financial terms, as economists do (e.g. reducing pollution by 25% rather than investing in pollution control up to the point that marginal returns equal marginal costs: Leman and Nelson, 1981).

In measuring the achievement of objectives, economists and policy markers also differ. Policy makers tend to assess costs and benefits in terms of the number of people affected, rather than financial costs and benefits (Verdier, 1984). Partly because of the vagueness of many program goals, they assess performance in terms of inputs rather than outputs (e.g. number of new hospital beds rather than improvements in health: Behn, 1981). They also weigh losses more heavily than gains, since the credit accruing to the originator of the policy is asymmetrical. As Verdier (1984, p. 432) says, "a policy that hurts five people and helps five, produces five enemies and five ingrates".

There are also differences in the decision making criteria employed by policy makers and those recommended by economists. While economists emphasize the future costs of a potential project, policy makers place much weight on sunk costs to justify further investment, since these reflect the amount of credibility the policy maker has invested, the size of the project's constituency and its expectations (Behn, 1981). Opportunity cost usually does not figure heavily in policy makers' calculations (Leman and Nelson, 1981); projects and programs are assessed in their own terms, without close reference to alternative uses of funds (particularly alternatives in areas outside the decision maker's control). Finally, the issue of compensation is critical to policy makers; for economists it is usually an afterthought. Economists tend to find a solution satisfactory if, in theory, the losers could be compensated. To push a policy innovation through, policy makers must usually ensure that they will be compensated, and have mechanisms to do so.

Many sectoral ministries are also dominated by sectoral specialists (i.e. engineers controlling infrastructure policy; doctors controlling health policy). Economic analysis is often absent or done as an afterthought when financing is sought for the investment programs.

The last area of divergence is in the means favoured to influence the behaviour of economic agents. In part because many policy makers have a background in law, they often favour legal and regulatory instruments (Rhoads, 1978). That is, they try to affect behaviour through legal prohibitions and by redefining rights and duties. Economists, by contrast, emphasize economic incentives, manipulating these so that the desired behaviour comes to be in the agent's self-interest.

'SUPPLY SIDE' PROBLEMS OF ACADEMIC RESEARCH

University research is often unsuitable for use by policy makers. It often takes much longer to produce results than a policy maker with a short deadline can tolerate. It is frequently highly critical, without positive suggestions for action, in keeping with the self-image of many academics as gadflies. It often avoids simple recommendations that can be acted upon and instead analyzes the advantages of various alternatives. There is also a tendency for some researchers to learn tools and techniques and then search for problems to apply them to. Streeten (1988, p.640) calls this "the law of the hammer according to which a boy, given a hammer, finds everything worth pounding, not only nails but also Ming vases".

The state of social science research is such that consensus is rare. The incentives in academia are to question and overthrow existing theories and replace them with new ones. A state of conflicting views and information is therefore normal (Aaron, 1978) see note 2. This undermines the confidence of potential clients when they realize that for every study they examine, another can be found that provides opposite conclusions. Policy makers rightly judge that such research is more likely to complicate a debate than to resolve it (Weiss, 1977), and may even delay badly needed action in the face of conflicting advice (Aaron, 1978).

Academics also tend to search for general laws and patterns of behaviour - these reveal phenomena of greater theoretical and long run importance than highly specific observations. Funding agencies favour this approach because it provides a greater return to the research dollar. Consultants also look for general lessons, since these can be applied to new assignments at little marginal cost (Szanton, 1981). Policy makers, however, are not interested in generalizations - they want answers to the specific problems they face, even though such 'small' problems may not attract the interest of researchers.

Finally, the easiest kind of policy-oriented research is program evaluation. Policy makers are generally more interested in forward-looking research, however. Evaluations may actually be counterproductive by provoking a defensive reaction from the object of study (Szanton, 1981).

RESEARCH UTILIZATION IN DEVELOPING COUNTRIES

The wealth of material on research utilization in the US is in contrast to its paucity in developing countries. This paper attempts to glean from the former literature generalizations applicable to at least some developing country situations. However, a number of provisos should be made. First and most obviously, the degree of openness in the political system in most LDC's (both openness to influence by the electorate and researchers, and openness to investigation) is more limited than in the US. However, in those cases where researchers are permitted to express their views on policy issues, the suggestions made subsequently about increasing the likelihood of utilization are applicable. In some countries, the probability may increase from 40 to 80%, in others from 4 to 8%, but the principle is the same.

Second, data are scarcer and less reliable in LDCs. This will make 'quick and dirty' policy analysis more difficult and may lead researchers either into basic data collection or into data-free theory and modelling.

Third, in many LDCs, interest groups are less articulated than they are in industrialized countries and the demand for policy analysis from non-government clients is likely to be weaker.

Fourth, while the overt role of domestic interest groups may be weak or repressed, the influence of external agencies like the IMF over policy making is far greater than in any developed country. However, this 'policy dialogue' may actually serve to increase government demand for research, as ammunition needed in negotiation.

Fifth, in those societies where politics is highly ideological, researchers and research institutions tend to be similarly divided, often with explicit partisan affiliations. The process by which research influences policy resembles a lottery: a researcher can hit the jackpot if his or her party achieves power, but may then be quite marginalized during succeeding regimes. This phenomenon is common in Latin America.

Sixth, cabinet ministers in LDC's often tend to be more technically competent for their portfolios than their counterparts in North America or Britain. In Canada, for example, ministers often hold very different portfolios during their careers and rely on their staff for technical expertise. In Latin America, it is not unusual for a sectoral minister to hold a PhD in the relevant discipline and thus be in a better position to make independent assessments of information see note 3.

Similarly, casual observation suggests that cabinet ministers in some developing countries (again, Latin America provides the greatest number of observations) come from a wider variety of professional backgrounds than those in North America, where lawyers and businessmen predominate. One might expect the lesser weight of lawyers might make developing countries less prove to regulatory solutions, though the results do not seem to bear this out.

Seventh, decision-making tends to be highly centralized, placing heavy burdens on a few key individuals.

Eighth, administrative capacity for implementation is weaker, so the practical implications of any recommendation are critical.

Finally, it has been observed that the distinction between the formulation and implementation of a policy may be quite difficult in developing countries (Fine, 1990). Particularly in negotiations with external agencies, an apparent agreement on a policy may be nothing more than the avoidance of overt disagreement. Only after funding is in place do the hard decisions start to be taken. As the effects of the policy change become visible, resistance mounts, involving conflicts among a larger number of actors at different levels and in different agencies (Thomas and Grindle, 1990). This suggests that recommendations aimed simply at defining a desirable policy are unlikely to be effective; only if they are pursued throughout the implementation process will they stand a chance of success.

GUIDELINES FOR THE ENHANCEMENT OF RESEARCH UTILIZATION

Many authors have made suggestions about the design and dissemination of social science research with the intention of increasing the likelihood of its utilization. Some authors have discussed the conditions under which utilization is most likely to occur; others have gone further in providing quite specific recommendations. Many of these flow logically from the diagnoses presented in previous sections of this paper.

Weiss and Bucavalas (1977) found, somewhat counter-intuitively, that the quality of the research, including the reliability of its methodology, did have an important bearing on its credibility and impact. They also found that research that challenged existing assumptions and ways of doing things was not necessarily rejected and was often highly valued.

Faulhaber and Baumol (1988) looked specifically at the conditions under which economic research is likely to be utilized. The adoption of economic methods or recommendations was mostly likely in the following circumstances:

1. when they pertained to critical future decisions (e.g. forecasting techniques useful for investment in the stock market).

2. in situations where competitive pressures are strong, and there are pressures as well as incentives to innovate.

3. when a technique provides an accurate signalling function (e.g. a forecast that is not necessarily based on a correct diagnosis of underlying causes, but which itself influences expectations and behaviour).

4. when an agency or firm is highly accountable and must justify the decisions it takes.

5. when recommendations take into account their effects on income distribution.

1. Avoid explicit evaluations. Clients are more likely to respond to positive suggestions for change than to criticism of past performance.

2. Give the client credit for successful innovations; he will certainly have to take the blame for failures.

3. Don't try to develop complex methods on the job; stick to simple, tried and true ones. These are less risky and more comprehensible to the client.

4. Try an experimental or pilot project to test a recommendation before proceeding to full-blown implementation.

5. Aim for situation-specific solutions, not generalizable laws. Good solutions will eventually catch on.

1. Learn about the history of the issue. By researching previous arguments, the analyst can identify key interest groups, areas of disagreement and data gaps, as well as changes in context that may influence future bargaining.

2. Find out who will be making the decision. Target the recommendations to those groups and present them in a form appropriate to the audience.

3. Timing is critical. Recommendations should be presented when they are most likely to receive attention. Generally, it is best to get into the debate early before positions harden.

4. Learn everyone's interests and arguments.

5. It's OK to think like an economist but don't write like one. Emphasize the decision at hand, the underlying problem, and options to solve it. Minimize methodology, jargon and equations.

6. Keep it simple. Where it is essential to explain complex features of an issue, illustrate them simply, using examples where possible.

7. Policy makers care more about distribution than efficiency. Explain what groups will be affected by the proposed measures, avoiding general references to 'welfare losses for the economy'.

8. Take implementation and administration into account. Don't propose measures that are technically optimal but too complex or costly for an agency to administer.

9. Emphasize a few crucial and striking numbers. Use statistics that emphasize the number of people affected, rather than aggregate dollar figures.

10. Read the newspapers. More generally, try to gain access to the same sources of general information as the policy maker, since these sources influence their perceptions.

1. Be economical about the use of economics. Apply economic analysis only to problems where it is relevant. Emphasize basic economic principles.

2. Discount for political demand. If the first-best solution is infeasible push for the second-best and make it as good as possible.

3. Dare to be quick and dirty. Partial analysis is better than none.

Some authors who stress policy impact have gone farther than this and propose the development of a distinct type of inquiry: policy analysis. Behn (1981, p. 200) defined this activity as follows: 

the examination of a particular policy problem in an effort to determine what the government should do; usually but not always, it is prepared for a particular policy maker who wants to make, has to make, or is able to make a specific decision (or take a specific action) about the policy problem.

In Behn's definition, the essential difference between a researcher and a policy analyst is that the latter has a particular user in mind for the research product and previous contact with the user. ("If you don't have a client, you're not doing policy analysis": Behn, 1985, p. 428.) This view is controversial. If the implication is that the client must be situated within government, it is highly restrictive and eliminates much focused, applied research that is appropriately critical of government. At the least, one could argue that research carried out for non-government clients should be considered policy analysis. At most, one could argue that economists traditionally view society as a whole as their client when pointing out various inefficiencies, and that some see their role as defending the interests of unorganized but disadvantaged elements of the population. Research carried out in these circumstances, if it provides specific policy recommendations, might also be considered policy analysis. A better term for the approach advocated by Behn might be 'client-oriented research'.

An alternative or supplement to the specialized policy analyst is the research broker who, instead of providing policy advice himself, acts as an intermediary between policy makers and the research community. The broker responds to a client's needs by seeking out needed information (or a researcher who could provide it); synthesizing and condensing information; and providing technical assistance to help the client interpret the data (Davis and Salasin, 1978).

This role would be a difficult one to say the least. Brokers might be liable to the 'shoot the messenger' syndrome and could be used as convenient scapegoats for policy failures. They might also be pressured to suppress embarrassing reports and to tell clients what they want to hear (Sundquist, 1978).

The precarious nature of the broker's existence has led some observers to doubt the feasibility of such an approach. The broker is an idea "off-touted and rarely instituted" (Weiss, 1978, p. 70); there are few cases to empirically evaluate. Some experimentation with this promising but risky idea would probably be useful. One could reduce the vulnerability of the position by having the broker funded by and perhaps reporting to an external funding agency; the advantage of the broker's greater autonomy might well outweigh the loss of commitment by policy makers to use a service they are not paying for.

In practice, the main limiting factor may be the availability of suitable people to play the broker's role. The combination of technical skills, diplomacy, entrepreneurship and relative risk-indifference calls for an exceptional individual.

CRITICISMS OF CLIENT-ORIENTED RESEARCH

The client-oriented approach to utilization has a number of deficiencies. At the empirical level, it simply has not worked well. The great wave of studies surveyed in this paper were done largely to find out why such an enormous investment in policy-oriented research had been so rarely utilized. It is clear from Section 2 and 3 that the ways in which policy makers and researchers analyze information and make decisions are fundamentally different in many respects. It is not particularly useful to deny this problem and exhort researchers to 'try harder'.

Some measures can certainly be taken to reduce the gap. The various guidelines cited earlier are not unhelpful. They are limited, however, by the need to apply them in varying circumstances. Most of those listed earlier could just as easily been phrased as their opposites. For example:

1. Make use of pilot projects before moving to full scale.

vs..

'Seize the moment'. Go straight to implementation while the opportunity exists, including a monitoring and evaluation component. You may not get another chance.

2. Be prepared to sell your proposal, from early discussion through to implementation. (Except in circumstances where your credibility will be greater if you are seen as detached.)

3. Find out who's making a decision and target your results. (Except when it is a non-decision, resulting from unconscious, uncoordinated actions.)

At a more philosophical level, there are other problems with client-oriented research. Selecting the right client is obviously a critical decision, but how do you do it when you don't know in advance what results you will come up with? (see Section 2). If the client is an individual, how do you know that he or she will be in the same post when the research is completed?

More fundamentally, how legitimate are the interests of any single client? Any given client is likely to have a partial view of its own needs, let alone those of society, and a weak understanding of the broader repercussions of satisfying those demands. Farmers may want more subsidies, cheaper credit or higher prices, but they are unlikely to calculate the effects on the fiscal deficit, farm employment, or inflation. Client-oriented research does not create an awareness of those conflicts, or an understanding of the broader social system and the legitimacy and interdependence of various interests within it.

The question of time lags is probably more serious than the literature implies. The problems are not only the mechanical ones of coordination, but relate to the setting of research priorities. Except for extremely narrow topics, the research process is lengthy, while the demands of clients are immediate. Furthermore, the crises clients face are frequently the result of previous errors or of trends sent in motion some time ago, but whose effects are felt only now see note 4. A suitable slogan to illustrate the danger of this approach might be "Client-oriented research: tomorrow's solutions for yesterday's problems".

ALTERNATIVE USES OF SOCIAL SCIENCE RESEARCH

So far this paper has been relatively pessimistic about the utilization of social science research. The remainder of the paper is more optimistic. It contends that many of the negative assessments of research impact are based on a mis-specification of 'impact': using a broader definition, the utilization of social science research has in fact been considerable.

The most frequent and most important way in which social science research actually affects policy seems to be through its effect, often slow and cumulative, on widely-used concepts and methods. This was the principal finding of a massive 1975 study of research utilization in the United States (reported in Weiss, 1978) and the observation seems to be more broadly applicable. The contribution of social science research is not so much in proposing specific solutions to already well-defined problems, but rather in defining the problems and providing an array of methods with which to analyse them. These can be extremely important contributions. "Determining what issues are discussed in the policy making process may be the single most powerful political act" (Seekins and Fawcett, 1986).

Several terms have coined to describe this more diffuse model of research utilization, in contrast to the highly focused, client-oriented approach described earlier. Pelz (1978) refers to conceptual vs. instrumental research; Weiss (1977) to enlightenment vs. social engineering; and Rich (1977) to knowledge for understanding vs. knowledge for action (summarized in Snell, 1983).

Problem definition can take many forms. It can consist of detecting or imposing a pattern on data, for example, a trend toward worsening income distribution (Rein and White, 1977). As Weiss (1978, p. 31) points out, it can focus attention and "help to turn what were non-problems or private problems into policy issues (such as child abuse), help to convert existing policy issues into non-problems (e.g. marijuana use), (or) drastically revise the way that a society thinks about issues (e.g. acceptable rates of unemployment)". In developing countries, changing approaches to research on the informal sector have had a major influence on how that phenomenon has been viewed over the last twenty years. Once seen as an embarrassing symptom of backwardness to be eradicated, research led to a greater acceptance of the informal sector as legitimate, and more recently through de Soto's (1987) work, as a positive force for development.

In fact, it could be argued that the most significant contribution of social science research is at the most general level, in the generation of ideas and ideologies. History shows that ideas can be very powerful. The writings of Raul Prebisch had a tremendous influence on Latin American policy makers and led directly to the wave of import substitution that transformed the continent's economic structure in the fifties and sixties. The subsequent implementation of conservative policies had equally far-reaching effects and was also strongly influenced by the intellectual currents of the day. In both cases, ideas took root in an environment and a time when policy makers were receptive to them.

All of these approaches to problem definition contribute to what Verdier (1984) calls 'structuring the terms of the debate'. This can include setting the agenda (for example, predicting long term trends that will eventually require the attention of policy makers or putting forward specific problems for discussion) and subsequently, injecting into that debate certain concepts and methods used in social science. Concepts like 'marginalization' made their way into policy discussion from social science literature; so did analytical methods for appraisal and evaluation.

IMPLICATIONS FOR RESEARCH FUNDING AGENCIES

What does all this imply for agencies financing research in developing countries?

First, one should recognize that the likelihood of utilization of any kind of research (social or technical, instrumental or conceptual) is quite small. Furthermore, while the impact of instrumental research is limited it is highly identifiable; the impact of new concepts and problem identification is far-reaching but difficult to attribute. Furthermore, there are rarely total victories or losses in any policy arena and policies are frequently reversed or eroded with changes in personnel or circumstance. In some areas, such as tax reform, continuous revision rather than once-and-for-all change seems to be the rule (e.g. Perry and Cardenas, 1986). This also complicates the identification of research impact: what appears to be a strong impact in the short run may be eroded in the long run, while basic research which illuminates certain constant relationships may be drawn on years later to support or justify a policy change.

An approach which creates the conditions for both kinds of impact is desirable. This could involve a portfolio approach, financing a variety of projects, each intended to produce a different type of impact. Alternatively, it is possible to finance long term research programs from which both conceptual and instrumental impacts can be derived.

The content of such research should be such that it creates an understanding of basic behavioural relationships and a thorough knowledge of existing data and data sources. Research agenda that lead to such knowledge and that deal with long-term issues that have short-term implications can be tapped to provide short-term policy advice. An example is a multi-year research program financed by IDRC in Latin America since 1983, examining savings and investment behaviour and the functioning of financial markets see note 5. Knowledge gained through this research has frequently been applied in policy recommendations related to management of inflation, capital flight, and wage and price policies.

Similarly, a multi-phase network on debt bargaining see note 6 has yielded both 'instrumental' impact (progress in the adoption of recommendations about provisioning requirements for commercial banks) and 'conceptual' impact (by reinforcing general principles about cross-conditionality, no net transfers by least developed countries and so on). In the former case, it is relatively easy to trace a policy change back to a specific recommendation and to claim credit for it. In the latter, the researchers contribute to an ongoing debate; their contribution is partial and less identifiable. In the long run, however, the acceptance of broad principles may have greater effect on the debt problem. Both types of impact are valuable, so we need an approach that does not rule out one or the other a priori. While short term impacts can be derived from long term research programs, conceptual innovations rarely result from highly specific, client-oriented projects. It is this asymmetry which makes the case for program support a powerful one.

The literature on policy analysis also gives some suggestions about research approaches that are most likely to influence policy. For example, traditional economics is probably not adequate. Something like the style of policy analysis advocated by Richard Behn is probably more suitable, without the single-client orientation.

Many of the limitations of economics could be mitigated by paying attention to factors which affect the feasibility of implementation. One of the traditional role of economics in identifying specific inefficiencies, their costs, and who pays them, should receive more emphasis. This need not imply that quantitative economic criteria are overriding, but that the costs of tradeoffs, where estimable, are made explicit. For example, countries may deliberately choose to forego the putative efficiency benefits of trade liberalization in return for greater cultural or political autonomy. It helps in making such a decision, however, to know if the price of such autonomy is 2% of GDP or 20%.

In addition, economists should extend their analysis into the implementation phase. In making policy recommendations, they should not stop at recommending the first-best technical solution, but rather present a variety of ranked options, indicating the efficiency and distributional consequences of each. Who are the winners and losers in each scenario? It may be possible to design instruments to compensate the losers (rather than simply saying that efficiency gains will be sufficient that they could, in theory, be compensated). It may also be possible to map out alternative sequences of policy implementation, so that the introduction of measures in sequence progressively neutralizes opponents and strengthens supporters. Improvements in modelling and computer techniques are making this increasingly feasible.

This style of research is not common, and there may be a need for specific graduate training to meet this need. Since researchers comfortable with interdisciplinary methods are scarce, there may at times be a need for multidisciplinary teams from economics, political science and/or public administration. This approach also requires knowledge of the history and evolution of policy issues and familiarity with the institutional context and decision-making process of government. These cannot be acquired through short-term projects done by individuals; the earlier recommendation of greater reliance on program grants is thus reinforced.

Another process which needs some rethinking is the setting of research agenda. There is currently a fixation on the part of policy makers and donors with problem solving. This is understandable, but we often forget that there are other ways of dealing with problems, principally by avoiding them in the first place. Too often 'problem-oriented research' means trying to put Humpty Dumpty back together. One of the most important roles of research is to alert policy makers and others to incipient trends, so that they can take appropriate action before it is too late. Those trends can contain opportunities as well as dangers. It can be argued that one of the reasons the Asian NIC's have done well, at a time when others have done poorly, is that they have anticipated problems and taken advantage of opportunities, rather than simply responding to crises.

Donors have an important role to play in supporting theoretical research, though they are often reluctant to do so. They should recognize that the distinction between "theoretical" and "empirical" is in no sense equivalent to "useless" and "useful". Theoretical research can be extremely useful. A plausible, verifiable theory about how peasants respond to increases in crop prices, or savings to changes in interest rates is of obvious relevance to poverty and can be extremely useful. On the other hand, the collection of masses of data on an irrelevant topic benefits no one except computer manufacturers. The aim should be to support practical, applicable research, be it theoretical or empirical, rather than applied research per se.

It is sometimes argued that donors should prefer to support government agencies rather than universities or private centres, in order to increase the likelihood of impact. The reverse also could be argued: that, non-government research centres may have bettter trained people with more time for research, are less likely to have their findings 'smothered' and less prone to shifting their stance with the government of the day.

Neither generalization is likely to be robust. Policy impact can best be achieved by adapting the process of consultation to local circumstances. Often this consists of the formation of teams of researchers from government and academia; formation of inter-institutional steering committees (to provide direction for university research); frequent consultations and seminars with policy makers; training of government officials by academics; and so on. The flexible application of mechanisms such as these is likely a more effective means of increasing the probability of utilization than a priori decisions about institutional affiliation.

Whatever kinds of institutions and mechanisms are supported, greater attention should go to dissemination. Donors should be prepared to pay the costs of those 'frills' which enhance the quality and utilization of research: training, networking, replication of studies, and dissemination of results through conferences, books, working papers, abstracts and the like. The familiar 'project cycle' syndrome must also be broken, whereby researchers have an interest in finishing a project quickly in order to get on to the next income-earning activity, while donors want to finish it in order to close the books and begin the job of spending next year's budget. Follow-up activities which refine, repackage and disseminate results to different audiences should be seen as legitimate and important, often more so than new data collection exercises. The various 'commandments' in Section 6 may be useful to researchers making their first forays into policy advice, and the mechanism of a 'research broker' is worth experimenting with.

In general, the literature should caution donors against excessive risk-aversion or emphasis on purely instrumental research. This is at least as true for developing countries as for developed ones. Particularly important is the role of research that detects and analyzes trends (e.g. the implications of new materials and technologies for primary commodity exporters; developments in global financial markets; issues likely to arise in global negotiation over climate change).

Also important is the contribution of research to problem definition. Over the last decade, there has been a wave of interest in increasing the role of the market. This has complicated the role of policy analysis, since it is increasingly difficult to distinguish ends from means. Many changes that seem most appropriately to be viewed as means (e.g. privatization) have come to be seen as ends in themselves. In such conditions, the fundamental role of research in defining problems and setting the terms of the debate becomes even more crucial.

REFERENCES

Aaron, Henry J., Politics and Professors: The Great Society in Perspective. (Washington, D.C.: Brookings Institute, 1978).

Alkin, Marvin C., A Guide for Evaluation Decision Makers. (Sage, U.S.A., 1985) 

Behn, Robert D., "Policy Analysis, Clients and Social Scientists" in Journal of Policy Analysis and Management, Vol. 4, No. 3, pp. 428-32 (Spring 1985).

Behn, Robert D., "Policy Analysis and Policy Politics" in Policy Analysis, Vol. 7, No. 2, pp. 199-226 (Spring 1981).

Blinder, Alan S., Hard Head, Soft Hearts. (Addison-Wesley, Reading, Mass., 1987).

Cairncross, Alex, "Economics in Theory and Practice" in American Economic Review, Vol. 75, No. 2 (May 1985).

Davis, Howard R., and Susan E. Salasin, "Strengthening the Contribution of Social Research and Development to Policy Making" in Lawrence E. Lynn Jr., Ed. Knowledge and Policy: The Uncertain Connection (National Academy of Sciences, Washington, D.C., 1978).

de Pablo, Juan Carlos, "How to End Up an Utter Failure as Minister of the Economy". ICEG Occasional Paper 12, San Francisco (1988).

de Soto, Hernando, El Otro Sendero (Editorial Sudamericana, Buenos Aires, 1987).

Faulhaber, Gerard R. and William J. Baumol, "Economists as Innovators: Practical Products of Theoretical Research" in Journal of Economic Literature, v. XXVI (June 1988).

Fine, Jeffrey, "Graduate Education in Economics for Africans" (African Economic Research Consortium, Nairobi, August 1990).

Glad, Betty, "Multidisciplinary Studies and the Relationship of Scientific Research to Public Policy Making" in Political Psychology, Vol. 9, No. 3 (September 1988).

Grumm, John G., "The Analysis of Policy Impact" in Fred Greenstein and Nelson Polsby, Ed. Policies and Policy Making (Addison-Wesley, Reading, Mass., 1975).

Gulhati, Ravi, "Who Makes Economic Policy in Africa and How" in World Development, Vol. 18, No. 8 (August 1990).

Harberger, Arnold C., "The Economist and the Real World". ICEG Occasional Paper , San Francisco (1989).

Hartle, Douglas, Public Policy, Decision Making and Regulation, (IRPP, Butterworth & Co, Toronto 1979).

Haverman, Robert, "Policy Analysis and Evaluation After 20 Years" in Policy Studies Journal, Vol. 16, No. 2 (Winter 1987).

Hirschman, Albert O. and Charles E. Lindblom, "Economic Development, Research and Development, Policy Making: Some Converging Views". Behavioural Science , Vol. 7, No. 2 (April 1962).

Jenkins-Smith, Hank C., "Professional Roles for Policy Analysts: A Critical Assessment" in Journal of Policy Analysis and Management, Vol. 2, No. 1, pp. 88-100 (1982).

Lamb, Geoff, "Managing Economic Policy Change: Institutional Dimensions", World Bank Discussion Paper No. 14 (1987).

Leman, Christopher, and Robert H. Nelson, "Ten Commandments for Policy Economists" in Journal of Policy Analysis and Management , Vol. 1, No. 1, pp. 97-117 (1981).

Leonard, David, "What is Rational When Rationality Isn't?" in Rural Africana, No. 19-20 (Spring-Fall 1984).

Lindblom, Charles and D.K. Cohen, Usable Knowledge. (New Haven: Yale University Press, 1979).

Lipton, Michael with Richard Longhurst, New Seeds and Poor People. (Baltimore: John Hopkins Press, 1989).

Lynn, Lawrence E. Jr., "The Question of Relevance", in Lynn, Ed. op cit.

Meltsner, Arnold J., Policy Analysts in the Bureaucracy University of California Press, Berkley, 1976).

Nelson, Robert H., "The Economics Profession and the Making of Public Policy" in Journal of Economic Literature, Vol. XXV (March 1987).

Patton, Michael Q., "The Evaluator's Responsibility for Utilization" in Marvin C. Alkin, Ed. Debates on Evaluation (Sage, Newbury Park, California, 1990).

Pechman, Joseph A., "Making Economic Policy: The Role of the Economist" in Greenstein and Polsby, Ed. op cit.

Pelz, Donald C., "Some Expanded Perspectives on the Use of Social Science in Public Policy" in J.M. Yinger and S.J. Cutler, Ed. Major Social Issues: A Multidisciplinary Perspective (New York: The Free Press, 1978).

Perry, Guillermo and Mauricio Cárdenas, Diez Años de Reformas Tributarias en Colombia (FEDESARROLLO/CID, Bogotá, 1986).

Rein, Martin and Sheldon H. White, "Policy Research: Belief and Doubt" in Policy Analysis, Vol. 3, No. 2, pp. 239-71 (Spring 1977).

Rhoads, Steven E., "Economists and Policy Analysts" in Public Administration Review, Vol. 38, No. 2 (March/April 1978).

Rich, Robert F., "Uses of Social Science Information by Federal Bureaucrats: Knowledge for Action Versus Knowledge for Understanding" in Weiss, Ed. op cit.

Rist, Ray, Ed. Program Evaluation and the Management of Government (Transaction, New Jersey, 1990)

Roe, Emery M., "The Expatriate Advisor as Senior Policy Analyst" in Policy Studies Review, Vol. 7, No. 3, pp. 519-36 (Spring 1988).

Rose, Richard, "Disciplined Research and Undisciplined Problems" in Carol Weiss, Ed. Using Social Science Research in Public Policy Making (Lexington Books, Lexington, Mass., 1977).

Saasa, Oliver, "Public Policy Making in Developing Countries: The Utility of Contemporary Decision-Making Models" in Public Administration and Development, Vol. 5, No. 4, pp. 309-321 (1985).

Seekings, Tom and Stephen B. Fawcett, "Public Policymaking and Research Information". The Behaviour Analyst, Vol. 9, No. 1, pp. 35-45 (Spring 1986).

Sharpe, L.J., "The Social Scientist and Policy Making: Some Cautionary Thoughts and Transatlantic Reflections" in Weiss, Ed. op cit.

Snell, Warren, "The Utilization of Social Science Research in Public Policy Making". Australian Quarterly, Vol. 55, No.4.

Streeten, Paul, "Reflections on the Role of the University and the Developing Countries" in World Development, Vol. 16, No. 5,pp. 639-640 (1988).

Sundquist, James L., "Research Brokerage; The Weak Link", in Lynn, Ed. op cit.

Szanton, Peter, Not Well Advised (Russel Sage Foundation/Ford Foundation, N.Y. 1981).

Taylor, L., Structuralist Macroeconomics: Applicable Models for the Third World. (New York: Basic Books, 1983)

Thomas, John W. and Merilee Grindle, "After the Decision: Implementing Policy Reforms in Developing Countries" in World Development, Vol. 18, No. 8, pp. 1163-1181 (August 1990).

Thurow, Lester, Dangerous Currents: The State of Economics (Random House, N.Y., 1983).

Tisdell, Clem, "Sustainable Development: Differing Perspectives of Ecologists and Economists, and Relevance to LDC's" in World Development, Vol. 16, No. 3, pp. 373-384 (1988).

Verdier, James, "Advising Congressional Decision Makers: Guidelines for Economists" in Journal of Policy Analysis and Management, Vol. 3, No. 3, pp. 421-438 (1984).

Weiss, Carol, Ed. Using Social Research in Public Making (Lexington, Mass.: Lexington Books, 1977).

Weiss, Carol and Michael J. Bucavalas, "The Challenge of Social Research to Decision Making" in Weiss, Ed. op cit.

Weiss, Carol, "Improving the Linkage Between Social Science Research and Policy" in Lynn, 
Ed. op cit.

Weiss, Carol, "Evaluation for Decisions. Is Anybody There? Does Anybody Care?" in Alkin, 
Ed. op cit.

Whitehead, L., "Political Explanations of Macroeconomic Management: A Survey" in World Development, Vol. 18, No. 8, pp. 1133-1146 (August 1990).

Wilson, James, "Social Sciences and Public Policy" in Lynn, Ed. op cit.

Wittrock, Bjorn, "Social Knowledge, Public Policy and Social Betterment: A Review of Current Research on Knowledge Utilization in Policy Making". European Journal of Political Research, Vol. 10, pp. 83-89 (March 1982).

NOTES

1. As Hartle (1979) puts it, "[Cabinet] ministers are like undertakers: they are paid to disguise what everyone knows to be painfully true".

2. It is thus a mis-specification to refer to academic research as "supply-driven"; it simply responds to a different set of demands and incentives than client-oriented research.

3. This underscores the importance of de Pablo's (1988) advice to would-be Ministers of the Economy: "Never assume that the failures of your predecessors were the result of incompetence". 

4. This problem is even more severe for technology-oriented research than policy research, since lead times in the former tend to be even longer (Lipton, 1989).

5. The participating research centres in this program are FEDESARROLLO (Colombia), PUC (Rio de Janeiro), CEDES (Argentina), CIEPLAN (Chile), Universidad Catolica (Bolivia). See Ahorro y Inversion en Latinoamerica, IDRC: MR207s, Ottawa, 1988.

6. S. Griffith-Jones, ed. Managing World Debt. (Wheatsheaf, 1988).

7. E. Rodriguez and S. Griffith-Jones, ed. Tangled Webs: Cross Conditionality, Banking Regulation and Third World Debt. (Macmillan, forthcoming).