The issue is this: 98% of the relevant information we need to write a good plan is unknown at the time we write it. We more or less know where we want to go, but the map we draw to get there is based on heroic assumptions about the terrain ahead well beyond our vision. We will certainly have gathered as much information as possible to help us create this map – government statistics about the project area, analyzed results from our own community surveys and focus group discussions, documentation from meetings with community leaders, and so forth. We may even have multiple scenarios mapped out to manage multi-dimensional risks that give our map a sense of flexibility to likely changes in conditions. But even with all this, a constant stream of unforeseen events, new information, and unpredicted behavior will quickly unravel our confidence in the assumptions on which our plan is based.

There is a saying, ‘reality always wins.’ Unexpectedly heavy rains will halve the planned number of community participants in workshops over the winter. A report showing the community’s school to be lagging will suddenly shift their priority from the health sector to education. The death of a key community leader will drain the momentum out of a whole line of activities that she had championed. Election violence will create security risks that will keep staff out of the community for three weeks. A string of unannounced donor visits will throw off an entire month of work, leaving staff playing catchup for the quarter. The head office will announce changes in the financial software and chart of accounts for the coming fiscal year that will require several members of the administration team to attend training workshops, and they will then fall behind on requisitions and payments, forcing delays in planned development activities. Two staff members will give birth and three will get sick in an oddly coordinated attack on productivity right in the middle of the annual planning season. The dollar will weaken unexpectedly and suck 5% of the budget out from under foot. At times it will feel like Murphy’s Law is in effect. Reality isn’t polite, it doesn’t wait for an invitation, it barges in. Reality always wins.

Double Jeopardy

We fall into the myth of the plan twice, once when we write the myth, and again when we try to manage to the myth. When we plan, we omit a reasonable cushion, making it unrealistic. When we manage, we apply an unreasonable rigidity, making it mythological. When things don’t go as planned – when reality doesn’t follow the myth – we make ever more heroic efforts to improve the planning process and increase the incentives to stick to the plan. That is, we make the plan even more detailed and rigid, falling further into land of make believe.

When Writing Plans. We write plans that fully utilize all available time and resources. This is foolish. Do we suppose that nothing will make unexpected demands on our time and resources or reduce our capacity? I did an experiment recently with a field team of development professionals. We tested their ability to plan their activities three months in advance – activities that were already based on the annual plan! They did alright the first month out, but by the time they got to the third month, over 30% of what they actually did no longer matched what they had planned to do. Yet most planning is done at least 12 months in advance, and project designs often have a 3-5 year time horizon.

Certainly we are not unaware that ahead in the darkness lurk beasts of the unexpected. Yet we are stoic in our planning, assigning all resources, finding a place for all capacity. Taking the offensive, we push hard to improve the planning process, attempting to capture the demons of uncertainty.  We adopt ever more detailed DME standards and templates that implicitly assume sci-fi levels of prescience. We start the planning process earlier so that our time horizon is even further out than before. Yes, we should think things through before we act, and yes, we do owe it to our donors to give a pretty good idea about how their money is to be spent. But most of the plans I’ve read over the years evidence not so much professionalism and responsibility as collective self-deception.

When Managing Plans. Here is where the more significant problem with the myth comes in. It’s one thing to make plans for a specific number of participants, and a specific number of purchased materials, and specific costs for things, and so forth – its another to prefer the myth of the plan over the reality of … reality. That is, when we apply rigidity to the plan and provide incentives to stick to it despite new information and changing conditions, we move into a realm of unicorns, fairy dust, and bad development practice.

The Straightjacket: bind the budget to the plan, then make it a performance issue to stay within the margins.

Here’s a typical practice: bind the budget to the plan through the logical framework, fence it in with margins on each activity line, and then make it an employee performance issue to stay within these margins. If you’re making widgets, this makes sense. Good project management is about the ability to follow a plan – accomplish scheduled activities on time and under budget … if you’re making widgets. If you’re doing community development, though, do you really want to create incentives to follow a plan that was written while most relevant information was still hidden? This may be good in some ways for the organization, but doesn’t’ necessarily make for good development practice. Why would we voluntarily put ourselves into such a straight-jacket? We need the ability to make mid-course corrections. Once the real terrain comes to light and we realize that our planned route that looked so straight and perfect on paper six months ago will actually take us across a mountain range of under-capacity, across a gorge of community disinterest, and through a labyrinth of political complexity, we will be sorry to have bound ourselves to the path.

Since I’ve been on the subject of ‘facipulation’ in my last two posts, it’s worth noting that binding development facilitators to a plan is a pretty good way to get them to start facipulating their discussions with the community. Ignoring new information from the community is a key facipulation skill. Finding ways to push off dealing with new events and conditions is too. Armed with the ability to say, ‘well, it’s not in the plan for this year, but we can talk about it in the next planning cycle’ is a great way to side-step reality when your organization’s plan is out of step with the community’s preferred future. Hear that? That’s the sound of your project stepping a rung or two down somebody’s participation ladder.

Good project management in this context requires a team, structure, and policy environment that encourages adjustments to new information and events. It requires creativity, agility, flexibility, and very good communication between staff, volunteers, community members, and donors. In mythological terms, we should prefer the creative agility of Hermes over Odysseus’ lashing of himself to the mast.

How Then Shall We Plan?

A good plan has three essential elements that define both how it is written and how it is managed.

1.  Shared Objectives. The community, the development organization, and donors should agree on the general objectives of working together. Not to be confused with the activities that will move them all toward these, the objectives describe the changes a ‘successful’ project would contribute to bringing about. This is the core of the plan – to work together to make this preferred future a reality.

2.  Agreed Methodology. The partners should also agree how they are going to move forward toward these objectives. This includes a starting set of activities, but it allows for some of these to be dropped and for others not imagined up front to be included as future conditions become known. It also includes, perhaps more importantly, a process for working together – how, who, and when will the partners negotiate the next steps together; how will they deal with new information and events; what will the process be for monitoring progress; and how will decisions be made about the scarce resources available to the partners as they move forward.

3.  Guiding Principles. Finally, a set of principles that will guide decisions as tradeoffs and hard decisions come down the road. What are the values and principles against which two forks in the road should be considered?

Rhetoric vs. Reality

When it comes to participation, the gap between rhetoric and reality yawns wide. How many of us have written project proposals peppered with phrases like ‘bottom up’, ‘self reliance’, ‘empowerment’, ‘citizen control’, and ‘local participation’ without fully grasping to what we were committing ourselves? While ‘participation’ has become fully and widely embedded in development speak, it has a long way to go in actual development work.

I suspect that many development practitioners assume they understand and ‘do’ participation because they get the meaning of the words and phrases associated with it and feel philosophically aligned with them. Most of us know that there are actual methodologies that have been developed and tested by others and written about in published papers and books. But, let’s be honest, how much of the development literature do practitioners actually read, and how much of what little we know do we actually put into practice?

The reality is, our actions in a community are constrained primarily by what our organizations consider their mission and core competencies, what our donors are willing to fund, and what we have time to do. Anything outside of these constraints just isn’t going to happen, so what good are truly participatory methodologies to us? How many development organizations are really ready to deal with local wishes once expressed, to deal with local knowledge once articulated, to use local resources once identified, to deal with local capability once mobilized, …in other words, to deal with local participation once invited?

Are we really interested in sharing power with (or releasing power in) the community? Are we operationally prepared to reverse the flows of information and accountability? Are our agendas and timelines, our people and processes, our standards and policies even capable of adjusting for the mess of self-mobilized, empowered, and initiative-taking stakeholders? I don’t think so. And that’s too bad. Talk is cheap. (Or in the case of writing project proposals, talk is lucrative.) Participation, on the other hand, is messy, slow, unpredictable, and rarely done. And yet, it is good practice. It is to our collective shame that we can’t more often match our rhetoric with the reality of our work.

Participation as a Spectrum

The gap is fairly easy to hide because participation isn’t a binary reality, but one with types and degrees. We just use the rhetoric associated with participation nearer to one end of the spectrum, while behaving according to participation nearer to the other end. Take for instance this simple four-element spectrum developed for a World Bank discussion paper†:

1. Information-sharing: people are informed in order to facilitate collective individual action.
2. Consultation: people are consulted and interact with an agency, which can then take account of their feedback.
3. Decision-making: people have a decision-making role, which may be theirs exclusively, or joint with others, on specific issues of policy or project.
4. Initiating action: people are proactive and able to take the initiative.

One can imagine how a project report might make it sound as if you were engaging people in the decision-making process as equal partners, that the community was helping to develop the plans, and participating in the activities, as well as the monitoring and evaluation of the project (level 3-4), when in reality the community development facilitators acted more in line with level 1-2, merely sharing information about the project in order to get ‘buy-in,’ exchanging food for work to ensure community involvement in the activities, and using focus group discussions to extract information from the community for the monitoring and evaluation work.

In his 2005 book “Ideas for Development”‡ Robert Chambers produces a more thorough treatment of participation as a spectrum that considers the objectives, the roles and relationships, the actions, and the ownership of the development process and the actors. I’ve reproduced this  table below.

I find this table helpful in that it gives us something to reflect on to see the gaps between our organizations’ rhetoric and reality. I suspect that most development practitioners have worked for organizations that aspire to be – and talk as if they were – operating at a degree of participation at the ‘partnership’ or ‘transformative’ level. But, I suspect that most of these organizations actually operate somewhere between the nominal and consultative levels.

What Facipulation Looks Like

Why do I feel so manipulated?

Facipulation is the act (intentional or not) of obscuring the gaps between the rhetoric and reality of participation. We facilitate meetings and processes in such a way as to get our preconceived ideas articulated, our preferred agenda adopted, our desired activities moved forward, …our needs met. And yes, this is at the expense of the communities’ ideas, preferred agendas, desired activities, and needs. Facipulation is not good practice.

I had planned to write here a series of cheeky “you know you’re a facipulator if…” statements, but the folks over at Stuff Expat Aid Workers Like have already done such a fine job of describing some typical facipulation techniques with ample cheek, I’m just going to refer you to them. SEAWL is written by two of the few development bloggers I follow; you can follow them on twitter @talesfromthehood and @shotgunshack. Check it out, have a good laugh, then do some navel gazing and have a good cry.

________________________

* As with all great terms, ‘facipulation’ has many inventors. I’ve been using the term for years falsely thinking I was the only one – I had even hoped with great naiveté that I would get credit for having coined this clever little mashup. But, I have discovered with some disappointment that its usage is already wide-spread. This fake dictionary entry is my own creation, but the word’s actual origin – that is who gets credit for its first use – is already lost in the soup of its popularity. The first usage I found was as a chapter title in a fairly obscure 1995 business book called Facilitating. It made its way into the urban dictionary in 2008. One of my favorite definitions was actually written by a self-described “ambiguity expert” (i.e. life coach) here. At any rate, so much for my presumed originality.

 † Bhatnager, B. and A.C. Williams (eds) (1992) Participatory Development and the World Bank: Potential Directions for Change, World Bank Discussion Papers 183, World Bank, Washington DC

‡ Chambers, Robert (2005) Ideas for Development, Earthscan, London, UK

[Evaluating with Purpose is a multi-post series that makes a case for development organizations to adopt better evaluation strategies and methods. Part 1: The Evaluation Charade unmaks the gap between what organizations think they are doing about evaluation and what most are really doing. Prospective posts are Part 2: what makes a good impact evaluation, Part 3: what makes a good evaluation strategy, and Part 4: transformative evaluation.]

_________________________

For someone whose job description is built around doing evaluation work, I can be remarkably hostile toward project evaluations. They tend to be expensive, distracting, and mostly useless. They can be surprisingly costly, often requiring projects to set aside 5-10% of the project funds that could otherwise have been budgeted to activities that further project objectives. They eat up a lot of staff time, not so much during the evaluation itself, but in the set up for it – designing indicators, measuring baselines, managing monitoring activities, etc. But most critically, very few ever accomplish the stated purpose of the evaluation – to provide rigorous evidence about whether the project was any good or not.

Evaluation Types

To be clear, there are different kinds of project evaluations and I’m not out to pillory them all. You can classify evaluation types by either purpose or method, but since form follows function, the basic typology is based on the evaluation’s purpose. Here are some of the main types.

• Needs Assessment: evaluations that look at a potential site or community to assess the needs that justify a project design.  These often utilize third party data and community interaction to define the situation and build a case for directing a specific intervention toward a target population.

• Design Analysis: evaluations that assess the logic of the project design, asking if the theory of change makes any sense. They assess if there is a complete, coherent, causal chain from the project inputs and activities to the project objectives as well as the validity of the assumptions made in the chain’s links.

• Process Evaluation: evaluations that determine if the project was implemented as planned. Did the money get spent as budgeted, did the services get delivered as designed, did the benefits reach the right people, was the project participatory and culturally sustainable, did anything unexpected happen?

• Impact Evaluation: evaluations that look at whether or not the project produced the desired effects. The project happened and lives were changed – impact evaluation asks if these changes can be causally attributed to the project.

• Cost-Benefit Analysis: efficiency evaluations that compare different interventions and determine what each costs to produce an equivalent impact. In other words, they identify the biggest bang for the buck.

However, 95% of the time when people say, “We are going to evaluate a project”, what they’re talking about is impact evaluation.

At least that’s what they think they’re talking about.

95% of the time, what they then go on to do is not actually an impact evaluation, and this is where my big beef about project evaluations being expensive, distracting, and mostly useless comes in.

Why Evaluate?

When someone decides to evaluate a project, what they generally have in mind is answering the question, “Was this a good project?” And by “good” they mean that it achieved the change objectives it was designed for. A literacy project was good if people learned to read; an economic development project was good if people’s wealth increased. Since only good projects are worth doing, repeating or scaling-up, organizations evaluate in order to learn what works and to demonstrate it to stakeholders. In sum, mot project evaluations are done for learning and accountability.* These are noble reasons.

Learning: Organizations may want to know if an innovative project design or new intervention is effective. Perhaps they want to know if something that had worked well in one context also works well given new conditions in a different context. It could even be that they want to know if a new less-costly way to implement a project is as effective as the old expensive way. You can only learn these things with confidence if you carefully evaluate project impact.

Accountability: Most organizations also evaluate their projects because they have to. Donors and Boards generally require at least some of their projects to be evaluated. They want to see evidence that previous projects were good before they’ll agree to fund or support more of them. Accountability ties consequences to performance; bad projects get killed, good projects repeated or expanded.

At least this is what most organizations think they get out of their project evaluations.

Evaluation and Self-Deception

The truth is that most project evaluations fail to provide either learning or accountability. Here’s three reasons why.

1:   They don’t inform decisions.

Evaluation is an empty exercise if it can’t trigger change. In theory, they are backward looking, but forward informing. Nevertheless, the same stakeholders who demand project evaluations often refuse to be guided by the results. What leaders believe about a project can easily override what an evaluation report says about it, and the evaluator’s recommendations must compete with organizational constraints,  personal loyalties, political considerations, and economic incentives that have at least as much sway in determining what happens next.

As a result many evaluation reports fail to influence decisions they were designed to inform. Evaluation reports get filed away quickly with little real reflection on their implications. There may be some perfunctory discussion and finger wagging around negative findings, but this is largely just a charade – everyone pretends the evaluation mattered a great deal when in fact it matter not at all.

I think the root problem is that few stakeholders really believe the evaluation reports they get. The evidence in most just isn’t that compelling and often the findings are little more than opinions, and the professional recommendations more like personal suggestions. If the project manager is well liked and trusted, the evaluator’s take simply counts for less. The reports may be used to bolster a position already taken, but they’ll rarely sway someone off their position.  They’re just too easy to disregard when they don’t say what you want them to say.  The truth is, few evaluation reports have evidence too compelling to dismiss. And if this is the case, then maybe its quite appropriate that few evaluations actually inform decisions. But now I’m getting ahead of myself.

2:   They aren’t assimilated.

Many development organizations simply don’t have the capacity to absorb the “lessons learned”, which ends up being a misnomer then anyway since they aren’t really learned. Assimilation becomes particularly problematic in organizations with policies that require all projects to be evaluated. For two reasons: First, compliance replaces learning as the organization’s de facto evaluation objective. Organizations end up focusing more on ensuring the standards were adhered to, the templates used, and the deadlines hit then on figuring out how to achieve better development results.

Second, the diminishing returns on evaluation information approach zero as the number of evaluated projects increases. Consider what a comprehensive evaluation policy implies for large organizations. My own estimations are that the largest of these can produce up to 1000 evaluation reports a year. How can any organization handle that quantity of information in a meaningful way? It’s like drinking water from a fire hose – most of it just sprays every which way but in. The marginal value of the nine-hundredth evaluation report pales to its cost. Of course, somewhere in all those reports are some really important evaluation findings that the organization should take to heart. But hearing those findings above the noise of all the others is nearly impossible, and so the organization fails to learn its own lessons “learned”.

3:   They don’t answer the critical question.

Recall, the critical question of an impact evaluation is whether or not the project was good (i.e. Did it achieve its outcome objectives?) We can assume that most development projects have a genuine intention to improve the lives of the beneficiaries; the problem is that there is a gap between the intention and the result. In Spanish they say “del dicho al hecho hay un trecho”. (Thanks to Dan Levy for that one.) In order to narrow this gap, you must first have a reliable measurement of the project’s impact. The problem is that most evaluations utilize methods that are too weak to produce a reliable measurement.

The most commonly used technique for estimating the impact of a project is to take measurements of the key indicators before the project was implemented and compare them to measurements of the same indicators afterwards. This pre and post technique may initially sound like a good idea, but it is profoundly flawed. In nearly every project you can imagine, there are myriad variables external to the project design that influence the results the project is trying to achieve. The effects of these other variables obscure the impact of the project such that you can mistakenly conclude that a good project is bad and a bad project good.

In order to isolate the impact of the project from the effects of these other variables, an evaluation must have a good estimate of the counterfactual – what would have happened to the beneficiaries had the project never happened. Obviously, the counterfactual can never be measured directly – that would require conducting an evaluation in two parallel universes. However, there are many evaluation techniques to estimate the counterfactual; some are more reliable than others, but all have one thing in common – a control group. A control group is a set of people that is similar to the group of project beneficiaries, but the control group doesn’t benefit from the project. I’ll go into detail about this in the next installment of this series “Evaluating with Purpose – Part 2: What Makes a Good Impact Evaluation.” But for now, the critical point is this: nearly all project “impact” evaluations fail to create a control group, which means they’re not really impact evaluations at all!

So what are they? And what question do they answer? Well, I don’t know what they are apart from a misnomer. But the question they answer is not what project implementers and stakeholders think they answer. They only answer the question of what happened, but they can’t tell you why. Now some organizations recognize this, but instead of changing their evaluation methods, they sophisticate their interpretation instead by drawing a distinction between “contribution” and “attribution.” But what does an organization learn by looking at the results and saying “We think we contributed to these”? Remember, a project can have a positive impact despite negative results and vice versa. And what kind of accountability is there in this equivocating stance? The organization can take credit for contributing to positive results while side-stepping responsibility where negative results occur.

The Great Pantomime

Development organizations spend millions of dollars every year – millions! – on evaluations that don’t answer the fundamental question of whether or not their projects are good. They utilize evaluation techniques that they know (or should know) can’t produce reliable measurements of impact. As a result, what they think they learn from project evaluations may be perniciously inaccurate. Good projects get killed, bad projects get scaled-up. As a result, what accountability they think they have is eroded to a charade – a vain gesture of professionalism and responsibility. If organizations were really concerned about learning and accountability, they wouldn’t allow such sloppy evaluation work to be done. They would demand a rigorous demonstration of impact. They would rethink the real cost of evaluation strategies that value quantity over quality. They would stop sophisticating and be more transparent about what they really get from their evaluations.

Stop the charade.

_________________________

* There is a third purpose – transformation – which I’ll save for a future blog post

Mask image by Simon Howden / FreeDigitalPhotos.net

Gauges images by Michal Marco / FreeDigitalPhotos.net