Step-by-step guide

For example, say you provided 6 months of free mental health support for a group of adults that started in December and ran until June. People could gain improved life satisfaction, and decreased anxiety, which you could measure through surveys throughout the intervention.  

However, you may find that a group not receiving the intervention also reports an improvement to life satisfaction and anxiety too.  This could be simply because the weather has improved. It would likely be a smaller improvement, and subtracting the two would then give you how much your intervention is responsible for specifically. This will reduce your value but you’ll be able to more robustly defend attributing any positive social impact to you. 

The perfect comparison group is known as the counterfactual; what would have happened to your participants if the intervention hadn’t happened. In some practice you may have heard this described as ‘deadweight’. It’s possible that some participants might experience a positive (or negative) outcome anyway. The counterfactual describes a situation that cannot be observed in reality, so in practice it must be measured using the best available evidence. A wide range of methods can be used to do this, ranging from the very robust but expensive to the easier but less reliable.     

You should find a comparison group that is the best possible comparison to your intervention group. Possible sources include: 

• Your participants before the intervention – this is the easiest comparison group to achieve, but it doesn’t account for the fact that your participants are influenced by other factors and might have experienced positive outcomes anyway. For example the time of year – people can feel happier in summer than in winter. 

• A similar group who will receive the intervention later – this is harder to achieve because it relies on a staggered roll out, but there’s less issue with bias from the external factors influencing the two groups. 

• Differentiating between levels of involvement – this is harder to achieve because it relies on there being different levels of involvement, and being able to survey people who aren’t involved, but there’s less issue with bias from the external factors influencing the two groups. 

• A similar group from national data – this is relatively easy to achieve because the national data is readily available for many interventions involving wellbeing. The closer you can match the national data to your sample the better, e.g. by certain demographics. This only works if there’s national data available that’s relevant to your intervention and group. 

For considering ‘what would have happened anyway’ with a counterfactual or control group it is important to assess ethical considerations in relation to your chosen approach, for example if individuals in the control group would be prevented from accessing necessary support to address their physical or mental health, or social needs, this would not be a viable option for your research. 

Assessing the counterfactual can also apply to interventions not involving people. It is explained how on each value page of Measure Up as this is different for each non-people based intervention. It often is about factoring in the negative aspects, for example the increase in driving inspired by providing a new electric car, or the emissions in the manufacture and delivery of that car. 

There are also other causality factors in relation to your impact to consider: 

 – attribution which considers how much is down to you, and how much is down to other interventions, sources, organisations, or circumstances. 

 – displacement which considers whether creating change in one person, or group of people’s lives, or a part of the environment, it has prevented a positive outcome somewhere else, or for someone else.   

 – duration considers how long the outcome lasts for which may be longer than the period of intervention.  An organisation will also need to consider how long they are accountable for the outcome. 

 – drop off looks at the extent the outcome changes over time, it can also look at how long the outcome is attributable to organisation. 

It’s likely your intervention won’t be responsible for the whole change so don’t be disheartened if the Platinum level value comes out lower than the Gold. You will get credit for achieving a much more realistic view of the actual impact of your intervention.