What we talk about when we talk about impact

What does “impact” mean in the context of the Private sector engagement funds? Ben Wandivinit, Monitoring and Evaluation Officer at LuxDev and member of LuxAidBusiness4Impact team, shares some of his perspective to unpack this often misunderstood concept. Curious to know how we measure change that truly improves lives? Read on.

When you hear the word “impact,” what comes to mind? In my experience, the answer has shifted over time and continues to do so. It’s a term that means different things to different people, and that’s part of the challenge. For me, the word carries a sense of scientific rigour. I used to associate it mostly with (quasi-)experimental designs, control groups, and inferential statistics — all in service of determining how much one variable explains a change in another. It meant thinking carefully about acceptable confidence intervals, sampling, and effect sizes. In short, the more ‘academic’ lens on impact measurement.

But impact isn’t always used in such a technical sense. In development cooperation, the term often follows “outcomes” in a theory of change, more often than not without very clear, evidence-based causal links or ex-post evaluations to verify whether the impact actually occurred. It also can often get blurred with “results” or “outcomes.”

Many bilateral and multilateral agencies speak of the “impact” they achieve when they are, in fact, reporting on project results. Still, we cannot expect everyone in our sector to know the term in its academic sense. And even for those who do, there is always a gap between the ideal and the feasible. Proper impact evaluations are costly — often hundreds of thousands of euros (so, in the case of the Private sector engagement funds, they are sometimes even more expensive than the evaluated project’s co-financing budget itself)— and time-intensive, taking many months or even years. That is time and money we rarely have, meaning that ambitious goals must often be balanced with a heavy dose of pragmatism.

So how do we approach impact in the LuxAid4BusinessImpact project?

With 80+ (projected) co-financed projects to monitor — under the Business Partnership Facility, the LuxAid Challenge Fund, and the LuxAid Demonstration Fund — running full-scale impact evaluations for each one simply isn’t realistic. Instead, we focus on key indicators that go beyond simple counting of how many people bought a product or were trained. We look at how many new end-users of a co-financed product or service actually improved their livelihoods. Improvements take many forms — from savings and increased revenue to better nutrition outcomes. We combine company reporting on these indicators with end-user surveys to gain a nuanced understanding of the changes people experience as a result of accessing these services or products. These surveys not only provide a more holistic picture of impact but also serve as an indirect way to corroborate company-reported data.

This approach may not meet the standards of academic impact measurement — and that’s fine. What matters is that we keep striving for credible, meaningful evidence that helps us understand whether our support is making a real difference in people’s lives. That, after all, is the kind of impact worth talking about.

Ben Wandivinit