Community measurement mistakes, from undefined activity to false attribution

Measurement fails when a convenient platform event is treated as the member outcome. The ten mistakes below were selected because each can change a management decision while remaining hidden in a polished report.

What to take away

Define active behaviour with its event, period and exclusions, and keep the underlying count beside every rate.
A percentage means nothing without its named population and size, because changing the denominator moves results.
Replies are not resolutions, so sample member journeys and keep acknowledgement separate from completion.
Platform analytics miss failed registrations and other channels, so combine records with research and show the boundary.
Association is not attribution, so plan a comparison or make a narrower contribution claim.

1. Leaving "active" undefined

Login, page view, post and event attendance describe different behaviour. Publish the event, period and exclusions. Preserve the underlying count with the rate.

2. Reporting a percentage without its denominator

A response rate can use all members, visitors, questions or eligible questions. Name the population and show its size. Changing the denominator can move a result without any change in service.

3. Counting replies as resolutions

A quick answer may be inaccurate or irrelevant. Sample the member journey and define what an appropriate resolution requires. Keep acknowledgement and completion separate.

4. Ignoring people who never entered

Platform analytics exclude failed registrations, inaccessible journeys and people who chose another channel. Combine records with research and make the boundary visible.

Government guidance on measuring service success applies to UK public services, not commercial communities. Its emphasis on the end-to-end journey and combining metrics with research is nevertheless a useful check.

5. Treating an average as the whole service

Mean response time can hide serious cases waiting much longer. Use distributions, priority groups and qualitative review. Do not invent a percentile target without operational evidence.

6. Comparing incompatible periods

A product launch, holiday, rule change or tracking update can alter member mix and volume. Annotate the series and avoid interpreting a simple before-and-after change as community impact.

7. Confusing association with attribution

Members who join may already be more loyal or skilled. Their later retention does not prove that participation caused it. Plan a suitable comparison or make a narrower contribution claim.

The Magenta Book explains evaluation approaches and the importance of selecting methods for the question. It concerns government evaluation and needs adaptation to the commercial decision.

8. Hiding work outside the platform

Expert answers, privacy requests, incident meetings and manual data repair may occur elsewhere. Reconcile queues and staff time. An efficient-looking dashboard can depend on unrecorded labour.

9. Rewarding volume

Targets for posts, comments or invitations can encourage noise and pressure. Pair activity with quality, safety and member effort. A quiet service may still resolve high-value needs.

10. Collecting every available field

Extra segmentation increases privacy risk and analytical flexibility without necessarily improving decisions. The Information Commissioner's Office's data minimisation guidance says personal data should be adequate, relevant and limited to what is necessary for the purpose.

Correct the reporting process

Maintain a metric dictionary, decision owner and change log. Before publication, ask an independent reviewer to reproduce calculations and identify plausible alternative explanations. Remove measures that do not support a choice.

Give the reviewer raw counts, definitions, source queries and a sample of complete journeys. A screenshot is not enough to reproduce a rate or discover whether deleted accounts and staff activity were handled consistently.

Record corrections openly. If a tracking fault changes earlier results, identify affected periods and decisions. Quietly replacing a chart prevents colleagues from understanding why a conclusion moved.

Set a challenge question for every headline: what else could produce this pattern? Then name the next evidence needed. A surge in answers could follow better member contribution, a staff campaign or repeated low-value replies.

Keep incentives away from proxies that teams can maximise without improving the member outcome. Pair targets with quality sampling and explicit safeguards. If the measure begins shaping harmful behaviour, change the management system rather than demanding a cosmetic dashboard adjustment.

A report is trustworthy when an adverse result remains visible, uncertainty is stated and the owner can explain what action the evidence permits. More charts cannot compensate for an undefined outcome.

Before you act

Publish the event, period and exclusions behind active.
Name the population and show its size.
Keep acknowledgement and completion separate.
Annotate the series before comparing periods.
Reconcile queues and staff time outside the platform.
Give the reviewer raw counts, definitions and source queries.

Common questions

Why can a percentage mislead without its denominator?

A response rate can use all members, visitors, questions or eligible questions. Naming the population and showing its size matters because changing the denominator can move a result without any change in service. The article says to name the population and show its size.

Why does member retention not prove that participation caused it?

Members who join may already be more loyal or skilled, so their later retention does not prove that participation caused it. The article recommends planning a suitable comparison or making a narrower contribution claim instead of assuming community impact.

What should a reviewer receive to reproduce calculations?

Give them the original numbers, how measures are defined, the queries behind them, and some full member journeys.