Six ways to attribute community outcomes, from randomised trials to process tracing

Attribution asks whether the community caused an observed change. Different methods answer that question under different conditions. Choose the design before seeing results and involve an evaluator where the financial or policy consequence is material.

The UK government's Magenta Book concerns public-sector evaluation. It is used here as an authoritative methods reference, not as a rule for commercial community teams.

What to take away

Randomised experiment

Eligible units are assigned to intervention and control groups by a valid random process, then comparable outcomes are measured. When well designed and implemented, this can support a strong causal estimate.

It may be impractical or unethical to withhold a needed service, and members can influence one another across groups. Sample size, non-compliance and attrition require specialist design. Randomising invitations is not enough if treatment and measurement are poorly defined.

Quasi-experimental comparison

Methods such as difference-in-differences use a comparison group without random assignment and rely on specific assumptions. This may suit a staged rollout or a service introduced in one part of an organisation.

Groups can differ in hidden ways. Product launches, seasonality and marketing may affect them unequally. Test the assumptions and report sensitivity rather than presenting the technique's name as proof.

Matched cohort

Researchers match participants and non-participants on observed characteristics, then compare outcomes. This can be practical with customer records, subject to purpose, governance and data quality.

Matching cannot remove differences that were not observed. People who voluntarily join a community may already be more engaged, so retention differences remain vulnerable to selection bias.

Interrupted time series

Repeated measures before and after launch can show whether the level or trend changed at the intervention point. This is stronger than comparing one month before with one month after.

It requires stable definitions and enough observations. Another change at the same time can provide an alternative explanation. Mark tracking breaks and policy changes.

Contribution analysis

The team states how activities are expected to produce outcomes, gathers evidence along the chain and examines other causes. It can support a reasoned contribution claim when a counterfactual is unavailable, but it does not automatically provide an effect size.

The Magenta Book's analytical methods annex describes contribution analysis and related theory-based approaches. The strength depends on a plausible theory, implementation evidence and serious examination of alternatives.

Process tracing

This method examines a case and tests whether evidence expected under a proposed causal mechanism is present, while seeking evidence for competing explanations. It can suit an important organisational outcome with rich records and no comparable group.

It requires disciplined hypotheses and skilled analysis. A sequence of events alone does not prove causation.

Select the design

Write the decision, outcome, unit, intervention, comparison, period, confounders, data and ethical constraints. Then choose the strongest feasible method. Combine quantitative estimates with process and qualitative evidence where this answers distinct questions.

Pre-register or time-stamp the main outcome and analysis where the stakes justify it. This reduces freedom to switch measures after seeing which result is favourable. Record deviations and their reason rather than pretending the original plan was followed.

Assess spillover. Members can share answers with colleagues who were assigned elsewhere, and operators may change their behaviour across the whole organisation. Contamination weakens a simple group comparison but may itself reveal part of the service effect.

Plan missing-data handling. People who leave the community or stop answering surveys may have worse experiences. Complete-case analysis can therefore overstate benefit. Report attrition and test plausible alternative assumptions with an evaluator.

Review fairness and burden. A stronger design does not justify unnecessary tracking or withholding critical help. Obtain data protection, ethics or legal input appropriate to the study.

If no design can isolate impact, state that the evidence shows association or plausible contribution. Honest limits support better decisions than an exact percentage built on voluntary membership and a weak comparison.

Before you act

Common questions

When is a randomised experiment suitable for attributing community outcomes?

A randomised experiment suits when eligible units can be assigned by a valid random process and comparable outcomes measured. It may be impractical or unethical to withhold a needed service, and members can influence one another across groups. Sample size, non-compliance and attrition require specialist design.

What are the main limitations of quasi-experimental and matched cohort methods?

Quasi-experimental methods rely on specific assumptions and groups can differ in hidden ways. Matched cohorts cannot remove unobserved differences, and people who voluntarily join a community may already be more engaged. Both are vulnerable to selection bias and confounding from product launches, seasonality or marketing.

How should a team proceed if no design can isolate impact?

Say the evidence shows association or a plausible contribution instead of giving an exact percentage. A precise figure built on voluntary membership and a weak comparison would be misleading.