Impact evaluation of business support programs: when to do it and how to measure impact

Key Takeaways

• Measuring a program's impact means comparing what happened to the companies it served with what would have happened to them without it. That scenario, the counterfactual, cannot be observed and has to be estimated with a comparison group (Gertler et al., Impact Evaluation in Practice, IDB and World Bank, 2017).
• Comparing before and after, or participants against those who chose not to take part, does not measure impact. The Inter-American Development Bank (IDB) and World Bank handbook calls these counterfeit counterfactuals.
• The evaluation method is decided by the rules used to select companies. If there are more applicants than places, a lottery; if there is a score with a cutoff and many companies near it, a regression discontinuity; if the order of entry is drawn at random, randomized phase-in.
• With 200 to 400 firms per group, none of the business training studies reviewed by McKenzie and Woodruff (2014) had the statistical power to detect a 25% increase in sales. Failing to find a significant effect is not the same as finding there was no effect.
• In Start-Up Chile, a regression discontinuity showed that the entrepreneurship school increased funding and employment, and found no effect of cash and workspace alone for firms near the cutoff (González-Uribe and Leatherbee, 2018).

"What impact did the program have?" is the question that comes at the end, when the funder asks for the report or when renewal is on the table. Most programs cannot answer it rigorously because nobody decided, before selecting the companies, what they would be compared against.

If you design or fund a program with companies in Latin America, what follows will help you settle three things before you open the call: whether an impact evaluation makes sense for your program, which method your selection rules allow, and what has to be in place for the evaluation to be possible.

What is an impact evaluation?

An impact evaluation establishes the extent to which a program, and only that program, caused a change in an outcome for the companies it served. To do that, it compares those companies with a comparison group that represents what would have happened to them without the program.

The IDB and World Bank handbook sets three conditions for that group to be valid: without the program, it has on average the same characteristics as the treated group; the program does not affect it; and it would react to the program the same way the treated group does. If the comparison group does not meet those conditions, the estimate is biased.

It is not the only kind of evaluation. The monitoring and evaluation guide of Colombia's DNP, the National Planning Department, distinguishes five types of evaluation. Three of them are the most used in business programs; the last column is our own judgment:

When is an impact evaluation worth doing?

An impact evaluation is worth doing when its result will change a big decision, such as scaling, replicating, or renewing a program with substantial resources, and when the program has enough companies and a selection rule that allows a comparison group. The IDB and World Bank handbook suggests three questions before deciding: what is at stake, whether evidence on this type of program already exists, and whether there are resources to do it well.

J-PAL, MIT's evaluation lab, adds when it is not a good idea: if the sample is very small, if the cost of data eats into the program budget, or if the program will end before results arrive, unless those results can inform future editions.

With 30 or 60 companies in a single edition, it is hard to estimate precisely how much the program caused in sales, and that does not leave the program without evidence. There are three paths. Measure outcomes that vary little across companies, such as adopting a practice or applying for a loan, which need smaller samples. Keep the same selection rule across several editions and evaluate the cohorts together. And in the meantime, run a good outcome evaluation with a baseline, follow-up measurement, and a contribution analysis based on the theory of change.

Why doesn't comparing before and after measure impact?

Comparing before and after does not measure impact because it assumes that, without the program, companies would have stayed the same, and over months or years that is almost never true. The economy moves, customers change, and companies that join a program are often already in a moment of change. The handbook calls it a counterfeit counterfactual.

Comparing participants with those who chose not to take part has another problem: selection bias. Those who sign up tend to be more motivated or better prepared, and those differences explain part of the result.

Data from the Global Accelerator Learning Initiative (GALI) show this clearly. In its report A Rocket or a Runway? (2021), covering 2,599 ventures that applied to 212 accelerators, participants brought in, over two years, nearly twice the resources of rejected applicants across sales, investment, debt, and donations. But at application they already had more funds and more years in operation. The authors warn that rejected applicants are not a control group and that the difference is only a rough indicator.

How to design an impact evaluation of a program step by step

1. Define the question and the outcome that matters

Write the question in one sentence: what decision will be made with the answer, and which company outcome answers it. The outcome comes from the theory of change and should be written as an indicator, with its technical sheet, as we explain in indicators for business support programs.

Prefer outcomes that vary little across companies relative to the effect you expect. McKenzie and Woodruff found that statistical power is much higher for outcomes like adopting a practice or applying for a loan than for sales or profits, which vary a lot from one company to another and are reported with error.

2. Decide the evaluation and who will do it before you open the call

The comparison group almost always comes from the way companies are selected, so the evaluation is decided at the same time as the call. The IDB and World Bank handbook puts it the other way around from common practice: the program's operating rules define the evaluation method. The guide on how to select companies in an open call explains how to leave the selection ready for evaluation.

Also decide who evaluates. The impact evaluation is best done by a team different from the one operating the program, so that whoever reports the result is not the one who produced it. The operator does have to put the conditions in place: the selection records, the baseline, and contact with comparison companies, and that should be written into its terms of reference.

3. Choose the impact evaluation method based on how companies get in

Three questions about operations define the method: whether demand exceeds the places available, whether selection uses a score with a cutoff, and whether the order of entry can be drawn at random.

When it is not acceptable to leave companies out, randomized phase-in gives all of them the same chance of getting in first. And when some selected companies do not take part, the places they free up are filled from a waiting list ordered by lottery, never at discretion, and each company is analyzed in the group it was assigned to.

4. Check whether the sample is large enough to detect the effect

With few companies, an evaluation may conclude that the program had no effect when it did. Before committing, estimate the minimum effect that could be detected with the available sample, and decide whether that effect is reasonable. The handbook notes that deciding which minimum effect would justify the investment is, above all, a policy question.

The evidence on business training shows the risk. Many experiments work with 200 to 400 firms per group, and none of those reviewed by McKenzie and Woodruff had the power to detect a 25% increase in sales. McKenzie's 2020 meta-analysis estimates average effects of about 10% on profits and 5% on sales, too small for most individual studies to detect.

Two things change the calculation a lot. If only part of the selected companies take part, the comparison measures the effect of offering the program, and the required sample grows with the square of that loss: if half take part, you need about four times as many companies. On the plus side, power improves without adding companies if the analysis controls for the baseline value of the same outcome, if there are several follow-up rounds instead of one, and if the lottery is run within groups of similar companies.

5. Collect the baseline in both groups

The baseline is measured before the program starts, with the same instrument, in both the treated and the comparison companies. Measuring only the treated companies leaves the evaluation with no starting point for the other group.

Keep in mind that the program can change the way companies report. McKenzie and Woodruff warn that training can change how sales and profits are reported even if they do not change in reality, and that non-response on income questions tends to be high. Whenever possible, combine self-reports with administrative records, with the company's authorization.

6. Measure when the effect can show up, and avoid losing companies

The six- and twelve-month measurements we propose in the main guide are enough to see the first changes and to support a contribution analysis. If you also want to attribute an effect on sales or employment, a longer horizon is needed: eight of the 13 studies McKenzie and Woodruff reviewed measured at one year or less, and in Start-Up Chile outcomes were measured up to almost five years later.

The other risk is attrition. Small firms in developing countries close at a rate of about 8% a year, according to data cited by McKenzie, and in GALI's data rejected applicants answered follow-up surveys at a 47% rate, compared with 77% for participants. A company that closes is not dropped from the sample: its closure is an outcome. What biases the estimate is one group responding less than the other, so report each group's response rate and budget follow-up for both.

7. Check the assumptions before reading the result

Every method rests on an assumption that has to be checked with data. In difference-in-differences, that the groups had similar trends before the program. In regression discontinuity, that companies do not bunch just above the cutoff and that their prior characteristics do not jump at it.

An IDB evaluation of local productive clusters in Brazil shows why (Figal Garone et al., 2012). In the full sample, program and comparison firms had different trends before the program, so that estimate was not reliable. After matching or reweighting firms so their prior trends looked alike, the employment effect was still positive and grew over time, from about 3 to 6 workers in the first year to about 15 to 17 in the sixth. When you report, also say what size of effect the sample could detect and what it could not.

Real case: Start-Up Chile and which part of the program worked

Start-Up Chile is an acceleration program sponsored by the Chilean government. It gave each venture seed capital of about 40,000 dollars without taking equity, plus workspace. About 20 of the 100 companies in each cohort also got an entrepreneurship school, assigned through a pitch competition with a score cutoff.

Juanita González-Uribe and Michael Leatherbee used the program's two cutoffs, the admission score and the pitch score, to run regression discontinuities (González-Uribe and Leatherbee, 2018), with data on 3,258 applicants. They compared those just above each cutoff with those just below, and measured outcomes between one and almost five years after entering the program.

The result separated the program's components. They found no evidence of an effect from cash and workspace alone on funding, scale, or survival, at least for firms near the cutoff. The entrepreneurship school did have an effect: it raised the probability of getting additional funding by 21%, increased capital raised from about 37,000 to about 112,000 dollars on average, and took employment from 0.9 to 1.8 people.

The authors note that the central difficulty in these evaluations is separating the effect of the program from the effect of having selected the best. Regression discontinuity solves that only near the cutoff, which is why its results speak to the companies at the edge of selection.

Frequently asked questions about impact evaluation of business programs

What is the difference between an outcome evaluation and an impact evaluation?

An outcome evaluation measures what changed in companies after the program; an impact evaluation measures how much of that change is due only to the program. The first does not need a comparison group and the second does. For many programs, a good outcome evaluation with a baseline and follow-up measurement is the realistic option.

Who should do the impact evaluation, the operator or a third party?

It is best done by a team different from the one operating the program, so that whoever reports the result is not the one who produced it. The operator should collect the baseline, keep the selection records, and facilitate contact with companies, and those obligations should be written into its terms of reference.

How many companies does an impact evaluation need?

It depends on the size of the effect you want to detect, how different companies are from each other, and how many of those selected actually take part. As a reference, with 200 to 400 firms per group, none of the business training studies reviewed by McKenzie and Woodruff had enough power to detect a 25% increase in sales. Outcomes that vary little, such as adopting a practice, need smaller samples.

Can rejected companies serve as a comparison group?

They work as a reference for a contribution analysis, but not to attribute impact, because selection makes them different from accepted companies. The comparison usually flatters the program, and works against it if the most disadvantaged companies were selected. They become a valid comparison group when selection uses a score with a cutoff and companies near the cutoff are compared, or when entry among eligible companies was decided by lottery.

What if my program cannot have an impact evaluation?

Run a well-designed outcome evaluation: a baseline, a measurement at closing, and follow-up at six and twelve months, with indicators at each link of the theory of change. That allows a contribution analysis and lets you argue, with evidence, that the program contributed, without claiming how much it caused. If the program repeats, keeping the same selection rule across editions opens the door to an evaluation later on.

What to have ready before selecting

For an impact evaluation to be possible, three things are needed before the program starts: a selection rule that defines the comparison group, a baseline in both groups, and a budget to follow them after closing. With those in place, the final report can answer the impact question with the rigor the program's size allows, and say honestly what could not be detected.

If you share the selection rules of your next open call with us, we will tell you what kind of evaluation they allow and what needs to be ready for an independent evaluator to measure.

Request a design session | How we work with institutions

Read also: How to design a business support program | Indicators for business support programs