Building on the C5-DM framework for data management in literature reviews (Wagner et al. 2026), this vignette illustrates how data management principles can be implemented in a literature review. The framework foregrounds data conceptualization, collection, curation, control, and consumption as foundational activities that shape the transparency, reliability, and reuse of literature review outcomes. The vignette is organized into two complementary parts. The middle column presents a systematic literature review following established reporting conventions. The right column explains how the manuscript is internally grounded in explicit data management decisions aligned with the C5-DM framework, adding an interactive layer of annotations that makes these decisions visible. The vignette thus serves as a concrete illustration of good data management practice in literature reviews—one that readers can follow directly in their own work while also sharpening their understanding of what to look for when evaluating other software solutions and data management approaches.
Note5C-DM Framework
This column explains how the data management principles are implemented. ⬇️
Plan
This review focuses on worker selection decisions in online labor markets. In line with qualitative systematic reviews (Higgins and Green 2008; Smith et al. 2011), it aims at collecting evidence from prior empirical studies and aggregating it.
The review is conducted using a shared GitHub repository, which was synchronized locally by the team.
Search
We specified search strategies for the Crossref and DBLP application programming interfaces (APIs)1 using the core keyword microsourcing and a set of semantically related synonyms. We also reused samples from prior reviews (Wagner, Prester, and Paré 2021; Fiers 2023). The resulting query formulations were systematically tabulated to document the conceptual scope of the search and to enable consistent execution across data sources (see Table 1).
| Source | Search strategy | Search results |
|---|---|---|
| Prior review: Fiers (2023) | Fiers2023_search_history.json | Fiers2023.csv |
| Prior review: Wagner, Prester, and Paré (2021) | WagnerPresterPare2021_search_history.json | WagnerPresterPare2021.bib |
| Crossref (API search) | crossref_search_history.json | crossref.bib |
| DBLP (API search) | dblp_search_history.json | dblp.bib |
The search strategies are stored in JSON format together with the raw data files in the data/search directory, in line with the standard of Haddaway et al. (2022).
Dedupe
Metadata was prepared using CoLRev and extensions (Wagner and Prester 2025). Deduplication was done using BibDedupe (Wagner 2024).
NoteManually supervised task — Deduplicate
Trigger: Records in data/records.bib are in colrev_status = {md_prepared}
Responsible: Review coordinator
Command:
colrev dedupeOutput (in data/records.bib): Duplicates resolved and records set to md_processed (ready for prescreen)
Validation: Run colrev validate after committing
History filter: dedupe
Prescreen
For prescreening, we tested a new LLM-based prescreened. Comparison with prescreening decisions of GW showed low reliability with the llm-prescreener. Results were therefore reverted and a fully manual prescreen was implemented.
NoteManually supervised task — Prescreen records
Trigger: data/records.bib updated
Responsible: Two independent coders (GW and JP)
Protocol: screening procedures and criteria (see protocol/screening.md)
Output: records_screened.bib
History filter: prescreen
For full-text screening, PDF documents were retrieved and prepared.
NoteManually supervised task — Retrieve and prepare full texts
Trigger: Records in data/records.bib with colrev_status = {rev_prescreen_included}
Responsible: Review coordinator (pdf-get/pdf-prep) and individual reviewer (pdf-get-man/pdf-prep-man)
Commands:
# 1) pdf-get: automated retrieval and linking of PDFs
colrev pdf-get
# 2) pdf-get-man: manual retrieval for remaining missing PDFs
colrev pdf-get-man
# 3) pdf-prep: automated PDF preparation (OCR/text extraction/checks)
colrev pdf-prep
# 4) pdf-prep-man: manual fixes for remaining problematic PDFs
colrev pdf-prep-manOutput (in data/records.bib):
- After retrieval, records are updated with
file = {data/pdfs/<...>.pdf}and advanced tocolrev_status = {pdf_imported}. - After preparation, records are advanced to
colrev_status = {pdf_prepared}. - Failures are flagged as
pdf_needs_manual_preparationand must be resolved viacolrev pdf-prep-manbefore they are set topdf_prepared.
Notes: PDF retrieval depends on institutional access, licensing constraints, and rate limits.
History filters: pdf-get · pdf-get-man · pdf-prep · pdf-prep-man
Records were screened independently, as described in the following.
NoteManual task — Screen full texts
Trigger: Records in data/records.bib with colrev_status = {pdf_prepared}
Responsible: Two independent coders (GW and JP)
Protocol: full-text screening procedures and criteria (see protocol/fulltext_screening.md)
Process:
- Each coder screens the full text independently and records an inclusion/exclusion decision.
- Disagreements are resolved through discussion (or third-coder arbitration, if needed).
- Decisions and reasons are logged in the record fields (e.g.,
screening_criteria,screening_decision,screening_reason) according to the protocol.
Output (in data/records.bib):
- Included full texts:
colrev_status = {rev_screen_included} - Excluded full texts:
colrev_status = {rev_screen_excluded}(with documented exclusion reason)
History filter: screen
Data extraction
In line with the methodology of systematic reviews (Higgins and Green 2008; Smith et al. 2011), we selected structured data forms to extract evidence from the studies.
NoteManual task — Extract evidence
Trigger: data/records.bib updated
Responsible: Two independent coders (GW and JP)
Protocol: data extraction form and protocol (see protocol/data_extraction.md)
Output: extracted_evidence.yaml
History filter: data
Synthesis
The narrative synthesis is in the paper document in Markdown format, allowing for larger teams to work on the same document (similar to the covid19-review). The current status of the project is automatically updated with every change and reflected in the PRISMA flow diagram (Figure 1, in line with the recommendations of Page et al. (2021)).
Table 2 provides summary of extracted evidence.
Show the code
from pathlib import Path
import yaml
import pandas as pd
import matplotlib.pyplot as plt
yaml_path = Path("data/evidence_platform_work_biases.yml")
with yaml_path.open("r", encoding="utf-8") as f:
doc = yaml.safe_load(f)
df = pd.DataFrame(doc.get("papers", []))
cols = [
"study_id", "citation_key", "year", "platform", "platform_type",
"method", "data", "sample",
"bias_type", "bias_mechanism",
"outcome_affected", "evidence_level", "direction_of_bias",
"key_result", "notes",
]
df = df[[c for c in cols if c in df.columns]]
df["year"] = pd.to_numeric(df.get("year"), errors="coerce").astype("Int64")
df = df.sort_values(["bias_type", "evidence_level", "year"], na_position="last").reset_index(drop=True)
compact_cols = [
"citation_key", "platform_type", "bias_type",
"outcome_affected", "key_result"
]
df_compact = df[compact_cols].rename(columns={"citation_key": "study"})
df_compact| study | platform_type | bias_type | outcome_affected | key_result | |
|---|---|---|---|---|---|
| 0 | RosenblatEtAl2018 | ride_hailing | algorithmic_visibility_bias | task_access | Algorithmic control mechanisms differentially ... |
| 1 | KuehlEtAl2022 | online_freelance | experience_lock_in_bias | matching_probability | Early experience thresholds significantly redu... |
| 2 | WoodEtAl2019 | online_labor | geographic_bias | wages | Workers in low-income countries receive system... |
| 3 | HannakEtAl2017 | online_marketplaces | rating_bias | reputation_scores | Online reputation systems encode demographic b... |
| 4 | CookEtAl2018 | accommodation | rating_bias | booking_probability | Hosts with African-American–sounding names rec... |
Data availability
To make the review reusable, the data was published on GitHub under the CC BY 4.0 license2.
References
Fiers, Fien. 2023. “Inequality and Discrimination in the Online Labor Market: A Scoping Review.” New Media & Society 25 (12): 3714–34. https://doi.org/10.1177/14614448221128379.
Haddaway, Neal R., Melissa L. Rethlefsen, Melinda Davies, Julie Glanville, Bethany McGowan, Kate Nyhan, and Sarah Young. 2022. “A Suggested Data Structure for Transparent and Repeatable Reporting of Bibliographic Searching.” Campbell Systematic Reviews 18 (4): 1–12. https://doi.org/10.1002/CL2.1288.
Higgins, Julian PT, and Sally Green. 2008. Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series. John Wiley & Sons, Ltd., Chichester, UK.
Page, Matthew J., Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, et al. 2021. “The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews.” Systematic Reviews 10 (1). https://doi.org/10.1186/S13643-021-01626-4.
Smith, Valerie, Declan Devane, Cecily M Begley, and Mike Clarke. 2011. “Methodology in Conducting a Systematic Review of Systematic Reviews of Healthcare Interventions.” BMC Medical Research Methodology 11 (1): 15. https://doi.org/10.1186/1471-2288-11-15.
Wagner, Gerit. 2024. “BibDedupe: An Open-Source Python Library for Bibliographic Record Deduplication.” Journal of Open Source Software 9 (97): 6318. https://doi.org/10.21105/JOSS.06318.
Wagner, Gerit, and Julian Prester. 2025. “CoLRev: An Open-Source Environment for Collaborative Reviews.” https://github.com/CoLRev-Environment/colrev.
Wagner, Gerit, Julian Prester, Roman Lukyanenko, and Guy Paré. 2026. “Data Management in Literature Reviews: The C5-DM Framework.” Research Synthesis Methods (Under Review).
Wagner, Gerit, Julian Prester, and Guy Paré. 2021. “Exploring the Boundaries and Processes of Digital Platforms for Knowledge Work: A Review of Information Systems Research.” The Journal of Strategic Information Systems 30 (4): 101694. https://doi.org/10.1016/j.jsis.2021.101694.