Assessing data quality – A probability-based metric for semantic consistency

Keywords

data-management, student-paper

Summary

We present a probability-based metric for semantic consistency using a set of uncertain rules. As opposed to existing metrics for semantic consistency, our metric allows to consider rules that are expected to be fulfilled with specific probabilities. The resulting metric values represent the probability that the assessed dataset is free of internal contradictions with regard to the uncertain rules and thus have a clear interpretation. The theoretical basis for determining the metric values are statistical tests and the concept of the p-value, allowing the interpretation of the metric value as a probability. We demonstrate the practical applicability and effectiveness of the metric in a real-world setting by analyzing a customer dataset of an insurance company. Here, the metric was applied to identify semantic consistency problems in the data and to support decision-making, for instance, when offering individual products to customers.

Article / DOI link

Citation (APA style)

Heinrich, B., Klier, M., Schiller, A., & Wagner, G. (2018). Assessing data quality – A probability-based metric for semantic consistency. Decision Support Systems 110, 95–106. https://doi.org/10.1016/J.DSS.2018.03.011

Citation: BibTeX

@article{HeinrichKlierSchillerEtAl2018,
  doi        = {10.1016/J.DSS.2018.03.011},
  author     = {Heinrich, Bernd and Klier, Mathias and Schiller, Alexander and Wagner, Gerit},
  journal    = {Decision Support Systems},
  title      = {Assessing data quality – A probability-based metric for semantic consistency},
  year       = {2018},
  volume     = {110},
  pages      = {95--106},
  url        = {https://www.sciencedirect.com/science/article/pii/S0167923618300599},
  abstract   = {We present a probability-based metric for semantic consistency using a set of uncertain rules. As opposed to existing metrics for semantic consistency, our metric allows to consider rules that are expected to be fulfilled with specific probabilities. The resulting metric values represent the probability that the assessed dataset is free of internal contradictions with regard to the uncertain rules and thus have a clear interpretation. The theoretical basis for determining the metric values are statistical tests and the concept of the p-value, allowing the interpretation of the metric value as a probability. We demonstrate the practical applicability and effectiveness of the metric in a real-world setting by analyzing a customer dataset of an insurance company. Here, the metric was applied to identify semantic consistency problems in the data and to support decision-making, for instance, when offering individual products to customers.}
}

Citation: RIS

TY  - JOUR
AU  - Heinrich, Bernd
AU  - Klier, Mathias
AU  - Schiller, Alexander
AU  - Wagner, Gerit
TI  - Assessing data quality – A probability-based metric for semantic consistency
T2  - Decision Support Systems
PY  - 2018
VL  - 110
SP  - 95
EP  - 106
DO  - 10.1016/J.DSS.2018.03.011
UR  - https://www.sciencedirect.com/science/article/pii/S0167923618300599
ER  -