Five community risks worth rehearsing, from a report surge to a compromised moderator

Community management risk scenarios are useful when they specify what changes, who decides and how recovery is proved. A generic register headed "reputation", "data" and "moderation" rarely tells the duty operator what to do at 9 pm. Rehearse the five cases below with the actual platform, people and escalation contacts.

What to take away

A report surge hides one severe case

A campaign or news event produces hundreds of reports. Most concern repeated arguments, but one contains a credible threat or disclosure. The early warning is not volume alone; it is an ageing high-priority queue or a report that lacks an assigned owner.

Contain by separating severity from arrival time, preserving the original material and assigning a decision lead. Give public responders a holding line that does not reveal private details. Recovery requires review of the missed signal, not a celebratory total of closed tickets.

Where a service falls within the Online Safety Act, Ofcom's illegal content duties guidance explains risk assessment, protections, records and review for in-scope providers. It does not establish the status of a particular account or outsourced team.

An AI summary changes the allegation

A handover tool assigns words to the wrong member or turns an unconfirmed allegation into a fact. Freeze automated summaries for that case class, restore the source record and identify every decision that relied on the faulty text. A person should correct downstream records and assess whether affected people need contact.

The ICO's AI transparency guidance covers purposes, retention and sharing where personal data is processed. It is under review, so check the current position and obtain advice rather than copying a historic notice.

A moderator account is compromised

Signals include an impossible login, changed recovery details, bulk deletions or unfamiliar exports. Remove the account's access, preserve logs, rotate exposed credentials and bring in security and data-protection owners. Do not let the same person investigate using the possibly compromised session.

The ICO personal data breach guide describes detection, investigation, internal reporting and record requirements, as well as circumstances requiring notification. The accountable specialist must assess the real incident, people affected and applicable timescales.

Incentivised praise appears without context

Members receive a benefit for posting but the relationship is not visible. Pause reuse of the material in marketing, retain the campaign instructions and identify where posts, ratings or excerpts appeared. Establish who commissioned, moderated and published them.

The CMA's fake reviews guidance addresses fake and concealed incentivised reviews and information derived from reviews. Qualified advice should determine the organisation's role and corrective action. Deleting inconvenient evidence without a record makes the later review harder.

A supplier change breaks moderation controls

An update alters permissions, export fields, automated labels or the route for member reports. Stop non-essential configuration changes, compare the current setup with the approved baseline and move critical work to the documented fallback. Contact the supplier with timestamps and reproducible examples.

The National Cyber Security Centre recommends monitoring AI supply chains and documenting relevant data, models and prompts. Apply the same practical discipline to any component that can affect member access or operator decisions.

Run the exercise to a decision

For each case, give the team incomplete information in stages. Record detection time, first safe action, owner, escalation, preserved evidence, member impact and restoration test. Include an out-of-hours handover and an unavailable supplier contact.

Close the rehearsal only after someone proves the recovery route on a representative case. Convert failures into a named action with a date. Repeat after a major platform, policy, supplier or audience change, because a response plan tied to the old system is only a document.

Before you act

Common questions

What early warning signs indicate a report surge is hiding a severe case?

An ageing high-priority queue or a report that lacks an assigned owner signals a severe case may be buried. Volume alone is not the indicator. Contain by separating severity from arrival time, preserving the original material and assigning a decision lead.

What should be done when an AI summary changes an allegation?

Freeze automated summaries for that case class, restore the source record and identify every decision that relied on the faulty text. A person should correct downstream records and assess whether affected people need contact. Check the ICO's AI transparency guidance for data protection requirements.

How should a compromised moderator account be handled?

Remove the account's access, preserve logs, rotate exposed credentials and bring in security and data-protection owners. Do not let the same person investigate using the possibly compromised session. The ICO personal data breach guide describes detection, investigation, reporting and record requirements.