Assess all manual and computational changes and roll back unreliable contributions (Control)

Evaluate and validate changes to the evidence base—especially when they are introduced by automation, bulk edits, or new contributors—before they become part of the dataset.

Why this matters

• Protect integrity of the evidence: Small metadata errors or inconsistent screening/extraction decisions can bias synthesis results.
• Detect unintended side effects: Automated transformations (dedupe, normalization, LLM support) can silently introduce systematic errors.
• Maintain accountability: Controlled evaluation makes it clear which changes were validated, accepted, or reverted—and why.

Illustration of change transparency and control

Practical implementation

• Use reviewable change units (small commits, tagged steps, pull requests) so changes can be validated efficiently.
• Validate with automated checks where possible (e.g., schema checks or validation rules) and manual checks for central fields.
• For algorithmic steps, benchmark against a human-coded reference sample (agreement/error analysis) and revert if reliability is insufficient.
• Record validation outcomes in the repository (commit messages, PR discussions, or a short changelog) to preserve decision rationale.