Reviews
Part of Reviewing community products and providers with an evidence hierarchy and repeatable tests
Community selection checklist: normalise quotes, then apply gates
Compare two community supplier quotes like for like: scope the service, normalise fees, add internal cost, set mandatory gates and record conditions.
Two supplier quotes are rarely comparable as they arrive. One covers a different member volume, and another leaves implementation out. A selection works only when both describe the same defined service, so the member outcome and the delivery boundary come first.
What to take away
- Price only suppliers that answer the defined service, not a broader marketing category.
- Normalise fees, taxes, period, volumes and included work before comparing totals.
- Add internal staff, implementation, migration, assurance and change to every quote.
- Treat any mandatory failure as a gate, not a low score.
- Keep conditional approvals in a register and pause signature while an essential condition stays open.
Price the same service
A quote is comparable only if it answers the same service. Write the member group, task, geography and service hours, and separate software, implementation, moderation, facilitation and expert answers. Reject proposals that answer a broader marketing category but not the defined service.
Scope and eligibility checklist
- Write member group, task, geography, hours
- Separate software, implementation, moderation, facilitation
- State mandatory access, safety, outcome requirements
- Identify buyer work and dependencies
- Set inclusion rules before searching
Typical candidate types include self-hosted forum software such as Discourse, SaaS community platforms and specialist moderation agencies. A product and an agency solve different layers, so quote and test them separately, then test the handoff where both are needed.
Set inclusion rules for the candidate list before searching, and state the mandatory access, safety and outcome requirements. Those requirements become the gates later, so they need to exist before any figure arrives.
Confirm what each quote promises
Treat a quote as a claim until the contract repeats it. Confirm the contracting entity and relevant office, date every feature, pricing structure, certification and partnership, and mark provider claims separately from your own observations. Ask for proposal and contract confirmation of material promises.
First-party evidence checklist
- Link supplier to official product page
- Confirm contracting entity and office
- Date features, pricing, certifications, partnerships
- Mark provider claims separately from observations
- Ask for proposal and contract confirmation
A promise to provide a document later is not completed due diligence. Record it as a condition with an owner and a deadline, so that it cannot be mistaken for a pass when the totals are compared.
Normalise the fees
Two quotes can be compared like for like only once they cover the same period, member volume and included work. Normalise fees, taxes, period, volumes and included work, then add internal staff, implementation, migration, assurance and change to both.
Illustrative arithmetic only. Supplier A quotes £12,000 a year for 500 members and £3,000 implementation. Supplier B quotes £18,000 a year for 1,000 members and no implementation fee. Pro-rate Supplier B to 500 members, which gives £9,000.
| Line | Supplier A | Supplier B |
|---|---|---|
| Quoted fee | £12,000 for 500 members | £18,000 for 1,000 members |
| Fee at 500 members | £12,000 | £9,000 |
| Implementation | £3,000 | None quoted |
| Internal moderation | £5,750 | £5,750 |
| Year-one total | £20,750 | £14,750 |
Internal moderation is 5 hours a week for 46 weeks at £25 an hour, which is £5,750 for both. The comparison uses the same member volume, period and internal cost, and it leaves tax out so that the volume effect stays visible.
Pro-rating is a simplification. Check how each supplier's pricing model behaves at your volume before relying on it. Also check whether each quote includes VAT: the standard VAT rate is 20%, so a quote that excludes it needs the tax added before the totals are compared.
Score with weights, then apply gates
Score each weighted criterion from 1 to 5, but treat any mandatory failure as a gate, not a low score. Record it as disqualifying or as a condition that blocks signature. A high average must not offset a failed access, safety or exit test.
| Gate | Passed when | A failure is recorded as |
|---|---|---|
| Access | Configured journeys meet WCAG 2.2 and work with relevant assistive technology | Disqualifying, or a condition that blocks signature |
| Safety | A reported harmful post is removed, the reason recorded and the reporter notified, in a test on the candidate's configured plan | Disqualifying, or a condition that blocks signature |
| Data | Controller and processor roles are mapped by processing activity under UK GDPR and the Data Protection Act 2018 | A condition with an owner and a deadline |
| Exit | A representative export opens independently, outside the platform | Disqualifying, or a condition that blocks signature |
The Information Commissioner's Office (ICO) regulates data protection, and the National Cyber Security Centre provides supply chain security guidance that can inform the data gate. Neither endorses a candidate.
Security certificates such as Cyber Essentials or ISO/IEC 27001 are evidence, not a pass: tie each to its scope and validity date. Assign qualified reviewers to the data and security evidence.
Run the gates on the same journeys
Evidence for a gate comes from a test, not a pitch. Give every candidate the same member and operator journeys, and record plan, configuration, people, date and result. Include failure, appeal, administrator succession and export, and preserve manual steps and unresolved issues.
Demonstration and test steps
- Give every candidate same member and operator journeys
- Record plan, configuration, people, date, result
- Include failure, appeal, succession, export
- Preserve manual steps and unresolved issues
- Disclose desk research versus testing
A supplier-led demonstration shows only what occurred under those conditions, so have intended operators repeat material tasks. For exit, confirm that relationships, files, permissions and moderation history all arrive intact. Test a serious report, a privacy request and an outage across each handoff. The evidence hierarchy used when reviewing products shows how much weight each kind of proof carries.
Record conditions before signature
Keep conditional approvals in a register with evidence, owner and deadline, and pause signature while an essential condition remains open. A register turns a promise made in a meeting into an obligation that someone can check.
| Condition | Evidence needed | Owner | Deadline |
|---|---|---|---|
| Export repeated on the configured plan | Export opened with files and moderation history | Buyer's data lead | Before signature |
| Contracting entity confirmed | Entity named in the contract | Buyer's procurement lead | Before signature |
| Document promised by the supplier | The document itself, dated | Named supplier contact | Before signature |
Price the transition, parallel running and member communication before approving exit terms, and recheck all volatile facts before signature. The preferred candidate should win because current evidence fits the stated need and risk, not because it collected the most features.
Before you sign
- Both quotes answer the same defined service.
- Fees, taxes, period, volumes and included work are normalised.
- Internal staff, implementation, migration, assurance and change are added.
- Every mandatory requirement has a gate and a test result.
- A representative export has been opened independently.
- Every open condition is in the register with an owner and a deadline.
Common questions
How do I compare quotes for different member volumes?
Normalise fees, taxes, period, volumes and included work, then add internal staff, implementation, migration, assurance and change. Pro-rate to the same member volume and keep the internal cost identical for both, so only the suppliers' own figures differ.
When does a failed test become a gate?
When it fails a mandatory access, safety or exit requirement. Record it as disqualifying or as a condition that blocks signature. A high weighted average does not offset it.
What belongs in the conditional approvals register?
Each condition, the evidence needed, its owner and its deadline. Do not treat a promise to provide a document later as completed due diligence, and pause signature while an essential condition remains open.
