Online labor markets and worker selection: A systematic review (vignette)

Building on the C5-DM framework for data management in literature reviews (Wagner et al. 2026), this vignette illustrates how data management principles can be implemented in a literature review. The framework foregrounds data conceptualization, collection, curation, control, and consumption as foundational activities that shape the transparency, reliability, and reuse of literature review outcomes. The vignette is organized into two complementary parts. The middle column presents a systematic literature review following established reporting conventions. The right column explains how the manuscript is internally grounded in explicit data management decisions aligned with the C5-DM framework, adding an interactive layer of annotations that makes these decisions visible. The vignette thus serves as a concrete illustration of good data management practice in literature reviews—one that readers can follow directly in their own work while also sharpening their understanding of what to look for when evaluating other software solutions and data management approaches.

Note5C-DM Framework

This column explains how the data management principles are implemented. ⬇️

Plan

This review focuses on worker selection decisions in online labor markets. In line with qualitative systematic reviews (Higgins and Green 2008; Smith et al. 2011), it aims at collecting evidence from prior empirical studies and aggregating it.

The review is conducted using a shared GitHub repository, which was synchronized locally by the team.

⧉ Rely on a version-controlled repository to organize and trace review data (Curate).

Search

We specified search strategies for the Crossref and DBLP application programming interfaces (APIs)¹ using the core keyword microsourcing and a set of semantically related synonyms. We also reused samples from prior reviews (Wagner, Prester, and Paré 2021; Fiers 2023). The resulting query formulations were systematically tabulated to document the conceptual scope of the search and to enable consistent execution across data sources (see Table 1).

⧉ Select standard file formats to facilitate access and contribution by humans and software (Control).

⧉ Store search results separately from the primary data, using identifiers as links (Conceptualize).

⧉ Reuse data from prior literature review papers and curated repositories (Consume).

import json
from pathlib import Path
from urllib.parse import quote

SEARCH_DIR = Path("data/search")

def md_link(text: str, target_path: str) -> str:
    target_path_posix = Path(target_path).as_posix()
    url = target_path_posix
    return f"[{text}]({url})"


json_files = sorted(SEARCH_DIR.glob("*.json"))

rows = []
for jf in json_files:
    try:
        data = json.loads(jf.read_text(encoding="utf-8"))
    except Exception:
        continue

    label = (data.get("label") or jf.stem).strip()
    results_path = (data.get("search_results_path") or "").strip()

    # Render links in the same "data/search/..." style as in your manual table
    json_link_target = f"data/search/{jf.name}"
    search_strategy_cell = md_link(jf.name, json_link_target)

    search_results_cell = (
        md_link(Path(results_path).name, results_path)
        if results_path
        else ""
    )

    rows.append((label, search_strategy_cell, search_results_cell))

print("::: {#tbl-search-overview}")
print("Table: Overview of search strategies and results.\n")
print("| Source | Search strategy | Search results |")
print("|:--------|:-----------------|:----------------|")
for s, strat, res in rows:
  print(f"| {s} | {strat} | {res} |")
print("\n:::")

Table 1: Overview of search strategies and results.

Source	Search strategy	Search results
Prior review: Fiers (2023)	Fiers2023_search_history.json	Fiers2023.csv
Prior review: Wagner, Prester, and Paré (2021)	WagnerPresterPare2021_search_history.json	WagnerPresterPare2021.bib
Crossref (API search)	crossref_search_history.json	crossref.bib
DBLP (API search)	dblp_search_history.json	dblp.bib

The search strategies are stored in JSON format together with the raw data files in the data/search directory, in line with the standard of Haddaway et al. (2022).

⧉ Document database coverage, especially when using LLM-based tools (Collect).

Dedupe

Metadata was prepared using CoLRev and extensions (Wagner and Prester 2025). Deduplication was done using BibDedupe (Wagner 2024).

NoteManually supervised task — Deduplicate

Trigger: Records in data/records.bib are in colrev_status = {md_prepared}
Responsible: Review coordinator
Command:

colrev dedupe

Output (in data/records.bib): Duplicates resolved and records set to md_processed (ready for prescreen)
Validation: Run colrev validate after committing
History filter: dedupe

⧉ Set up data preparation procedures to ensure high-quality and machine-readable data (Collect).

Prescreen

For prescreening, we tested a new LLM-based prescreened. Comparison with prescreening decisions of GW showed low reliability with the llm-prescreener. Results were therefore reverted and a fully manual prescreen was implemented.

⧉ Assess all manual and computational changes and roll back unreliable contributions (Control).

NoteManually supervised task — Prescreen records

Trigger: data/records.bib updated
Responsible: Two independent coders (GW and JP)
Protocol: screening procedures and criteria (see protocol/screening.md)
Output: records_screened.bib
History filter: prescreen

For full-text screening, PDF documents were retrieved and prepared.

NoteManually supervised task — Retrieve and prepare full texts

Trigger: Records in data/records.bib with colrev_status = {rev_prescreen_included}
Responsible: Review coordinator (pdf-get/pdf-prep) and individual reviewer (pdf-get-man/pdf-prep-man)

Commands:

# 1) pdf-get: automated retrieval and linking of PDFs
colrev pdf-get

# 2) pdf-get-man: manual retrieval for remaining missing PDFs
colrev pdf-get-man

# 3) pdf-prep: automated PDF preparation (OCR/text extraction/checks)
colrev pdf-prep

# 4) pdf-prep-man: manual fixes for remaining problematic PDFs
colrev pdf-prep-man

Output (in data/records.bib):

• After retrieval, records are updated with file = {data/pdfs/<...>.pdf} and advanced to colrev_status = {pdf_imported}.
• After preparation, records are advanced to colrev_status = {pdf_prepared}.
• Failures are flagged as pdf_needs_manual_preparation and must be resolved via colrev pdf-prep-man before they are set to pdf_prepared.

Notes: PDF retrieval depends on institutional access, licensing constraints, and rate limits.
History filters: pdf-get · pdf-get-man · pdf-prep · pdf-prep-man

Records were screened independently, as described in the following.

NoteManual task — Screen full texts

Trigger: Records in data/records.bib with colrev_status = {pdf_prepared}
Responsible: Two independent coders (GW and JP)
Protocol: full-text screening procedures and criteria (see protocol/fulltext_screening.md)

Process:

• Each coder screens the full text independently and records an inclusion/exclusion decision.
• Disagreements are resolved through discussion (or third-coder arbitration, if needed).
• Decisions and reasons are logged in the record fields (e.g., screening_criteria, screening_decision, screening_reason) according to the protocol.

Output (in data/records.bib):

• Included full texts: colrev_status = {rev_screen_included}
• Excluded full texts: colrev_status = {rev_screen_excluded} (with documented exclusion reason)

History filter: screen

⧉ Treat manual data work as primary analytical procedures by documenting parameters and tracing changes (Conceptualize).

Data extraction

In line with the methodology of systematic reviews (Higgins and Green 2008; Smith et al. 2011), we selected structured data forms to extract evidence from the studies.

NoteManual task — Extract evidence

Trigger: data/records.bib updated
Responsible: Two independent coders (GW and JP)
Protocol: data extraction form and protocol (see protocol/data_extraction.md)
Output: extracted_evidence.yaml
History filter: data

⧉ Align data structures with the specific review type (Curate).

Synthesis

The narrative synthesis is in the paper document in Markdown format, allowing for larger teams to work on the same document (similar to the covid19-review). The current status of the project is automatically updated with every change and reflected in the PRISMA flow diagram (Figure 1, in line with the recommendations of Page et al. (2021)).

⧉ Adopt non-redundant data structures, supported by identifiers and automated validation procedures (Control).

In [2]:

from prisma_flow_diagram import plot_prisma_from_records

plot_prisma_from_records(records_path="/home/gerit/ownCloud/data/literature_reviews/LRDM/C5-DM-vignette/data/records.bib", show=True)

Figure 1: PRISMA Flow Diagram (generated by prisma-flow-diagram)

Table 2 provides summary of extracted evidence.

In [3]:

from pathlib import Path
import yaml
import pandas as pd
import matplotlib.pyplot as plt

yaml_path = Path("data/evidence_platform_work_biases.yml")

with yaml_path.open("r", encoding="utf-8") as f:
    doc = yaml.safe_load(f)

df = pd.DataFrame(doc.get("papers", []))

cols = [
    "study_id", "citation_key", "year", "platform", "platform_type",
    "method", "data", "sample",
    "bias_type", "bias_mechanism",
    "outcome_affected", "evidence_level", "direction_of_bias",
    "key_result", "notes",
]
df = df[[c for c in cols if c in df.columns]]

df["year"] = pd.to_numeric(df.get("year"), errors="coerce").astype("Int64")
df = df.sort_values(["bias_type", "evidence_level", "year"], na_position="last").reset_index(drop=True)

compact_cols = [
    "citation_key", "platform_type", "bias_type",
    "outcome_affected", "key_result"
]
df_compact = df[compact_cols].rename(columns={"citation_key": "study"})
df_compact

In [3]:

Table 2: Table of evidence (generated from extracted-evidence.yaml)

	study	platform_type	bias_type	outcome_affected	key_result
0	RosenblatEtAl2018	ride_hailing	algorithmic_visibility_bias	task_access	Algorithmic control mechanisms differentially ...
1	KuehlEtAl2022	online_freelance	experience_lock_in_bias	matching_probability	Early experience thresholds significantly redu...
2	WoodEtAl2019	online_labor	geographic_bias	wages	Workers in low-income countries receive system...
3	HannakEtAl2017	online_marketplaces	rating_bias	reputation_scores	Online reputation systems encode demographic b...
4	CookEtAl2018	accommodation	rating_bias	booking_probability	Hosts with African-American–sounding names rec...

Data availability

To make the review reusable, the data was published on GitHub under the CC BY 4.0 license².

⧉ Share review data via open data platforms and use open licenses to enable reuse (Consume).

References

Fiers, Fien. 2023. “Inequality and Discrimination in the Online Labor Market: A Scoping Review.” New Media & Society 25 (12): 3714–34. https://doi.org/10.1177/14614448221128379.

Haddaway, Neal R., Melissa L. Rethlefsen, Melinda Davies, Julie Glanville, Bethany McGowan, Kate Nyhan, and Sarah Young. 2022. “A Suggested Data Structure for Transparent and Repeatable Reporting of Bibliographic Searching.” Campbell Systematic Reviews 18 (4): 1–12. https://doi.org/10.1002/CL2.1288.

Higgins, Julian PT, and Sally Green. 2008. Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series. John Wiley & Sons, Ltd., Chichester, UK.

Page, Matthew J., Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, et al. 2021. “The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews.” Systematic Reviews 10 (1). https://doi.org/10.1186/S13643-021-01626-4.

Smith, Valerie, Declan Devane, Cecily M Begley, and Mike Clarke. 2011. “Methodology in Conducting a Systematic Review of Systematic Reviews of Healthcare Interventions.” BMC Medical Research Methodology 11 (1): 15. https://doi.org/10.1186/1471-2288-11-15.

Wagner, Gerit. 2024. “BibDedupe: An Open-Source Python Library for Bibliographic Record Deduplication.” Journal of Open Source Software 9 (97): 6318. https://doi.org/10.21105/JOSS.06318.

Wagner, Gerit, and Julian Prester. 2025. “CoLRev: An Open-Source Environment for Collaborative Reviews.” https://github.com/CoLRev-Environment/colrev.

Wagner, Gerit, Julian Prester, Roman Lukyanenko, and Guy Paré. 2026. “Data Management in Literature Reviews: The C5-DM Framework.” Research Synthesis Methods (Under Review).

Wagner, Gerit, Julian Prester, and Guy Paré. 2021. “Exploring the Boundaries and Processes of Digital Platforms for Knowledge Work: A Review of Information Systems Research.” The Journal of Strategic Information Systems 30 (4): 101694. https://doi.org/10.1016/j.jsis.2021.101694.

---

1. For the illustration, we relied on open-access API-searches, because licensing issues do not allow for publication of raw data exported from databases like WOS or EBSCO.

2. Indexing in SYNERGY, SearchRXiv is planned once the review progresses beyond the illustration stages.