Even if the economic crisis hasn’t fostered as many changes in business practices as most of us desire, at least there is an increased interest in businesses and organisations to learn more about how resources are spent and what exactly the outcomes are. You might have never heard of impact evaluation before, but listen to Adam Alagiah, because you might have found a career path with a lot of future.
Impact evaluation is, or should be, at the core of any business or organisation seeking to create change. While there are many different methods and approaches to conducting impact evaluation, the basic principle is that of testing, rigorously, whether A causes B, and to what extent. The temptation to assume that an activity (A) causes a certain outcome (B) is strong, particularly if we have invested our time, energy or resources in that activity. Yet all too often this kind of assumption is simply not valid, and there is a danger that our time, energy or resources are being wasted, or that the same outcome could be achieved much more quickly/easily/cheaply in a different way.
This principle applies to a range of different sectors. A government might propose doubling nurses’ salaries, promising improved standards in hospitals. A supermarket chain might introduce a new loyalty card scheme to boost in-store sales. Or an NGO might give out free textbooks to improve literacy in a developing country. These are all actions which intuitively seem likely to cause the desired outcomes. Unfortunately, it turns out our intuition is often wrong. Particularly with complex outcomes such as hospital standards, sales figures or literacy levels, what we might expect to drive changes may turn out to do nothing of the sort. Impact evaluation is the science (or maybe the art) of testing exactly what impact a given activity has. When done well, it can also determine the best way to achieve a desired outcome.
Impact evaluation is, or should be, at the core of any business or organisation seeking to create change.
There are a variety of methods which can be employed to evaluate impact. The gold standard in impact evaluation is the randomised controlled trial (RCT). These are usually complex and expensive, but they do provide the most robust estimate of impact. At the other end of the scale are evaluations which simply measure outcomes before and after an intervention has been introduced. These can be cheaper and less sophisticated, but cannot be said to give a good indication of the impact of the intervention. The key to good impact evaluation is estimating the counterfactual – what would have happened if there had been no intervention. (The counterfactual must always be estimated – until we discover time-travel it is impossible to actually observe what would have happened!) By comparing the counterfactual estimate with observed outcomes, we can calculate the real impact our intervention actually had.
At present conducting rigorous impact evaluation is still new to most sectors and industries. RCTs have long been used in trials to evaluate the impact of new medicines – medical science is years ahead in this respect. The adoption of these rigorous techniques to evaluate social outcomes has largely been championed by development economists, keen to obtain an accurate measure of whether aid money was actually having an impact. Yet slowly this approach has spread into other areas, including government policy-making in Europe and elsewhere.
Given the variety of different methodologies, and the breadth of industries in which it can be used, there is no ‘typical’ job in impact evaluation. But there are certain skills that are likely to be useful whatever and wherever you might be trying to evaluate the impact of an intervention. Firstly, a solid understanding of the principle of impact evaluation, and the strengths and weaknesses of different methodologies. It does not always make sense to run a large-scale RCT; simple before-and-after measurements may in some cases be sufficient depending on your needs. Second, strong statistical and numerical skills are important in being able to analyse data, as is familiarity with statistical software such as STATA or SPSS.
There is no ‘typical’ job in impact evaluation. Strong statistical and numerical skills are important.
Finally, it is important to have a familiarity with the specific industry or intervention being evaluated. A well-designed evaluation can generate sound estimates of the impact of A on B, but understanding why this is the case takes a good understanding of the subject matter. It may be enough to know that giving a child medicine cures a disease. In many cases, however, understanding ‘the why’ can be just as important as measuring the impact. Giving out free textbooks may not improve literacy, but if this is only because pupils have no pencils to fill in exercises, then we don’t necessarily need to rule out free textbooks as a useless intervention. A good impact evaluation will test a number of different hypotheses, allowing us to understand which is the best way of achieving a desired outcome. Forming these hypotheses requires more than a superficial understanding of the subject matter.
Just as there is no typical impact evaluation job, so there is also no standard route into a career in impact evaluation. There are courses that cater specifically to those aiming for careers in evaluation – at present the majority are likely to be focused in the international development sector. But other degrees and courses would be perfect preparation too; those offering statistical or econometric elements would be ideal, particularly if they offer training in relevant software. Other industry or sector-specific courses would also be good preparation, and may include modules on evaluation in those contexts.
As demand for rigorous impact evaluation increases and spreads across sectors and industries, there will also be a concomitant rise in demand for individuals with the skills and sector-specific knowledge to carry out these evaluations. While there is no sure-fire way path to securing such a career, building these skills will help put you ahead of the pack.