In recent consultancy work, I have been advising Food and Agriculture Organisation (FAO ) and the World Bank (WB) on policy implementation in developing countries. Readers may be interested in the following observations on the different approaches used in these organisations.
In an FAO paper, it has been suggested that analysts use a comparative institutional approach to the delivery of agricultural policy (Sandiford and Rossmiller (SR) 1996). Case studies and commentaries have appeared in various FAO publications and seminars, and a symposium was presented at the 1998 Conference of the Agricultural Economics Society at Reading University (Williams 1997, Haebig et al 1998). The emphasis in these papers is on the implementation of policy as enacted in the legislature, or agreed with international donors, and asks whether the intention of the legislators/sponsors is being met in the delivery process and whether such processes meet wider criteria such as the meeting of set targets, avoiding unwanted effects (equity and income distribution), and delivering at least cost. The structure/conduct/performance framework (SCP) is employed to bring out the contrast between original intentions and actual implementation in the SR paper. Structure is used in this sense to describe intentions or aims laid down by the policy makers. Conduct is used to identify how a policy/loan programme is interpreted and managed in practice. Performance criteria are used to assess how well implementation meets the original objectives of the programme in terms of delivering the scheduled benefits to the targeted recipients (SR 1996, pp.7-12).
In most government systems, such review of policy programmes is an ongoing process. Reviews may not be made public but are likely to include many of the features of the SR proposals. SR state that they are trying to develop a framework for structured thinking about policy implementation that will assist them to do their advisory work better (SR 1996, p.1).
The Rural Development Division (RDV) of the World Bank (1996,1997) has recently instituted "portfolio" reviews of individual countries to ascertain the additional steps needed to improve the existing portfolio of loans and services in countries and regions as well as new entries. The portfolio reviews evaluate whether suitable policies are being introduced and whether projects are addressing the key development objectives effectively, including environmental and social objectives. The outcome of these portfolio reviews has been the decision to improve sector analysis and increase and strengthen monitoring and evaluation of projects, including the use of improved performance indicators.
In the policy advice area, the Bank has decided to put more emphasis on correcting inappropriate policies, weak institutions, and physical constraints to better policies (WB 1997, pp.139-140). In addition, the Bank has developed updated implementation criteria through its Operations Evaluation Department (WB 1997, p.S4), Evaluation of existing projects could be improved by more careful defining of project quality (relevance, efficacy, efficiency, contribution to institutional development and sustainability), identifying the features of a project that determine project quality (rigorous analysis, stakeholder participation, flexibility, and effective monitoring and evaluation), and introducing measures to improve implementation (establish 'ownership' of projects, maintain controls over lending, introduce better specification of partner obligations, and encourage use of pilot projects more frequently).
These developments therefore suggest that better policy could result from greater study of policy programmes in an ex post framework, The question of ex ante analytical constructs is still a requirement for policy advisors within departments, but if lessons are to be learnt from previous experience, then an ex post framework is required that is reasonably universal and which can be adapted to the job in hand by international advisors and by in-house advisors in the various ministries around the world.
SR propose that the unit for studying agricultural policy implementation is the policy delivery system (PDS), which is defined as the total modality of implementing a given policy. The PDS includes the unique set of institutions, individuals, processes and rules that together deliver the benefits of policy to a target group and enable control to be exercised to ensure adherence to the rules of access. Rules of access refer to legislative-type provisions that define who the beneficiaries are meant to be.
In order to make improvements to existing policy settings, they say it is necessary to know what the policy system looks like and how it operates, and what criteria to employ in its evaluation. Without knowing what is happening in the system, it is not possible to identify what is going wrong and how it is going wrong, and therefore to suggest remedial action. It is not that the system is not working but rather it is not working for those for whom the stated policy objectives said it ought to be working, i.e. the bulk of farmers
, By looking at policy implementation as a delivery system, ex post analysis will cope with the complexity and sheer diversity of agricultural policy within and across countries (SR 1996, pp.S-7).
The authors state that they are looking toward a framework for structured thinking about PDSs that can assist in making policy advice more relevant, reliable and workable. They believe that the analysis must be context-specific and say the aim is not to develop a prescriptive model that would apply across national boundaries. Within these restrictions, they suggest that the SCP framework for analysing private markets could be applied to the task (Koch 1980). The SCP framework evaluates market performance in the form of output and prices in relation to firms' conduct (behaviour) and the market structure of the industry (which are themselves inter-related). These principles can be applied to the behaviour and performance of governments rather than fIrms with suitable modification of outcomes to take account of the wider application of public policy (SR 1996, p.7).
Structure is then defined as the institutions, instruments and processes that are set up to deliver the chosen policy objective. It should include functions that are apparently intended to be performed (by agents) at different stages of the delivery process so these can be checked out subsequently. An analysis should include a description of policy instruments, formal and informal rules to be observed, functional responsibilities of institutions, and the actors involved at different stages.
The conduct component of the analysis covers how the institutions and individuals in the system operate to ensure and monitor compliance and effect delivery of the benefits to the target recipients. This is an analysis of the way the system works in practice - the actual as opposed to intended functions performed, and where and how private or extraneous interests and activities affect the way the system works.
The performance component of the analysis is an assessment of how well the policy delivery system meets the original objective in terms of delivering the scheduled benefits. Performance should be examined in terms of effectiveness, efficiency, enforceability and equity. Effectiveness conveys the idea of delivering the intended benefits - are the beneficiaries actually benefiting by the policy being implemented?
Efficiency asks about the costs of the programme and whether other means of delivery were considered? Enforceability conveys the idea that is possible to make sure the (proposed) benefits reach the targeted beneficiaries
- i.e. that there are no impediments to the process of delivery, nor creaming off beforehand. Equity is defined as the assurance of equal opportunity to access to the benefits under the rules
.
The evaluation guidelines suggested by the WB fIt into one or more similar categories. Project quality includes relevance, efficacy, efficiency, institutional implications, and sustainability. Determinants of project quality include relevance, stakeholder participation, flexibility, logical frameworks, rigorous analysis, realistic risk assessment, sustained implementation, and effective monitoring and evaluation. MarUl{Jemtnt responses include adjustment of loan portfolios, working with clients, improved economic and risk analysis, limitations on new lending, pilot experimentation, strengthening monitoring and evaluation, and improved portfolio management. Improved analysis and assessment relate to policy evaluation while most of the remainder relate to policy delivery.
The SR proposal thus seeks to define an ex post review procedure that focuses on what the named policy was meant to achieve, how it was implemented, and how successful it was in meeting its original aims. It is analytical rather than operational, and it seeks to draw lessons about policy formation and implementation in many different environments that can be useful in the future. The WB focus is more operational and as a result places more emphasis on implementation problems particularly on monitoring and evaluation which is absent from the SR proposal. It seems clear that the problem of perversity in implementation, which both organisations identify, arises from a lack of supervision and trust. Improved procedures for monitoring and evaluation can help to counteract this. Ultimately, the RDV notes, better implementation follows from better specification of projects and greater beneficiary participation.
The SCP framework is borrowed from economic analysis of private markets (SR 1996, p.7). In the FAO proposal it is being applied to aid and assistance projects in developing countries. There is thus quite a leap between the original concept and the application of it. The meaning of the terms changes. In addition, the international aid organisations are partners with recipient countries in the implementation process and hence have further transaction costs to overcome before smooth implementation of their jointly sponsored programmes.
The framework does add to our understanding when it draws its traditional distinction between structure and conduct. The distinction asks the analyst to separate institutional problems from delivery problems in public policy. Identifying conduct problems could well lead to different solutions to existing problems than loosely castigating the whole policy package as a policy design problem. Asking the right questions in a policy evaluation does not necessarily require a full SCP framework, but such a framework or a similar one is still likely to be most useful to policy analysts inside and outside government
SR and RDV appear to be reaching out to look for a general paradigm that assists policy advice to developing countries from an international aid point of view. The approach clarifIes that national governments have a continuing in-house problem to maintain accountability and efficiency in their administrative systems with suitable policy review techniques including monitoring and evaluation. International aid donors have a vested interest in national policy reviews of programmes involving aid funds, as accountability and efficiency are not in the donor's control but firmly in the control of each national government. Special techniques of monitoring and evaluation are required to satisfy both parties in these circumstances.
Robin Johnson, Wellington, New Zealand
Haebig, M., Johnson, R., Rossmiller, E., Sandiford-Rossmiller, F., Urban, K., and Williams, R. (1998), The Changing Environment for Agricultural Policy Implementation, paper presented to Annual Conference of the Agricultural Economics Society, University of Reading.
Koch, J.V. (1980), Industrial Organisation and Prices, 2nd ed, Prentice-Hall, New York.
Sandiford, F., and Rossmiller, G. (1996), Many a Slip: Studying Policy Delivery Systems, Paper presented to the Annual Conference of the Agricultural Economics Society, University of Newcastle-upon-Tyne, 27-30 March 1996, also printed in Williams (1997).
Williams, R. (1997), The Political Economy of the Common Market in Milk and Dairy Products in the European Union, FAO Economic and Social Development Paper 142, FAO, Rome.
World Bank (1996), Refonning Agriculture: The World Bank Goes to Market, Report No 15883, The World Bank, Washington, D.C.
World Bank (1997), Rural Development: From Vision to Action, The World Bank, Washington, D.C.