--- session: "Session 5" title: "Machine Learning Baselines: Classification" session_id: session-05 format: html: default --- In this exercise, we cover: | Part | Topic | Time | |---|---|---:| | 1 | Case understanding and first model | 15 min | | 2 | Probabilities and thresholds | 10 min | | 3 | Confusion matrix and metrics | 10 min | | 4 | ROC and AUC | 10 min | | 5 | Comparing customer categories | 10 min | | Wrap-up | Reflection | 5 min | | **Overall** | | **60 min** | # Case: Rising defaults at LendWise You are a business analyst working for LendWise, a fintech company that offers fast digital consumer loans. Over the last six months, LendWise has seen a noticeable spike in credit defaults in two customer categories: - **Early-career urban borrowers:** typically younger customers, often with lower savings, moderate income, and irregular repayment histories. - **Self-employed platform workers:** customers with fluctuating income, often higher short-term debt exposure, and less predictable monthly cash flows. The risk team wants to understand whether default risk follows the same logic in both categories, or whether the predictors and model quality differ substantially. You first focus on early-career urban borrowers and build a logistic regression model to predict default. Afterward, you compare the results with platform workers. LendWise is interested in questions such as: - Which factors are associated with a higher probability of default in early-career urban borrowers? - How can predicted probabilities be translated into default/non-default classifications? - How well does the model perform, and how informative are the ROC curve and AUC? - Does the model perform equally well across customer categories? **Reflection:** Think about how you would answer these questions based on your analysis. ## Data description - `customer_category`: customer segment - `income`: monthly income in EUR - `debt_to_income`: debt-to-income ratio - `missed_payments_12m`: number of missed payments in the last 12 months - `credit_utilization`: share of available credit currently used (between 0 and 1) - `months_with_company`: months since first activity with LendWise - `default`: whether the customer defaulted Throughout the exercise, **`default = 1` means default** and **`default = 0` means no default**. This convention determines how predicted classes, confusion matrices, and business decisions are interpreted. The dataset was curated by the company's Risk Analytics Department and has already been cleaned, validated, and prepared for modeling. ### Package setup Before starting this exercise, make sure the required R packages are installed. Run this command once in the R console. You do not need to run it every time you open the exercise. You can skip this step if these packages are already installed. ```{r} #| eval: false install.packages(c( "tidyverse", "pROC" )) ``` ```{r} library(tidyverse) library(pROC) df <- read_csv( "https://raw.githubusercontent.com/fs-ise/machine-learning-for-big-data/main/exercises/data/credit_default.csv", show_col_types = FALSE ) glimpse(df) ``` # Part 1 — Analyze early-career urban borrowers with logistic regression In this first part, focus only on early-career urban borrowers. ## Task 1.1 — Prepare the subset Create a filtered data frame containing only early-career urban borrowers. ```{r} # YOUR CODE HERE ``` ## Task 1.2 — Estimate the model Estimate a logistic regression model for credit default. - Use `default` as the outcome. - Use `income`, `debt_to_income`, `missed_payments_12m`, `credit_utilization`, and `months_with_company` as predictors. - Fit the model with `glm(..., family = binomial)`. - Inspect `summary(model)`. ```{r} # YOUR CODE HERE ``` ## Task 1.3 — Interpret the model Logistic-regression coefficients describe changes in **log-odds**, holding the other predictors constant. Exponentiating them produces **odds ratios**: the factor by which a one-unit increase multiplies the odds of default. - Which variables have positive associations with default? - Which variables have negative associations with default? - Can the largest raw coefficient automatically be called the most important predictor? ## Task 1.4 — From coefficients to classification Explain how the model moves from a linear combination of predictors to a predicted probability and then a default/non-default class. # Part 2 — Probabilities and thresholds Logistic regression produces probabilities. A classification is a separate decision created by comparing those probabilities with a threshold. Because the modeled event is `default = 1`, the returned values estimate **P(default = 1)**. ## Task 2.1 — Understanding outputs Run the code and compare the probabilities with the classes. ```{r} y_prob <- predict( model, type = "response" ) threshold <- 0.5 y_pred <- if_else( y_prob >= threshold, 1L, 0L ) head(y_prob) head(y_pred) ``` - What is the difference between `y_prob` and `y_pred`? - Why is it useful to define the threshold explicitly? ## Task 2.2 — Manual thresholding Try thresholds of 0.2, 0.4, 0.6, and 0.7: ```{r} t <- 0.4 y_pred_t <- if_else( y_prob >= t, 1L, 0L ) sum(y_pred_t) ``` - How does the number classified as high default risk change? **Your answer:** *Write your answer here.* - What pattern appears as the threshold increases? **Your answer:** *Write your answer here.* Then identify the 20% riskiest and 20% safest customers using `quantile()`. ```{r} # YOUR CODE HERE ``` ::: {.callout-important title="Threshold and approval logic"} Assume customers classified as likely defaults are not automatically approved: - **Lower risk threshold** → more customers flagged as risky → fewer approvals and lower default exposure. - **Higher risk threshold** → fewer customers flagged as risky → more approvals and potentially more defaults. Changing the threshold does not change estimated probabilities; it changes how the organization acts on them. ::: # Part 3 — Confusion matrix and metrics ::: {.callout-note} For simplicity, predictions are evaluated on the same observations used for estimation. This deliberately optimistic in-sample evaluation is sufficient for introducing metrics. Train/test splits, cross-validation, and out-of-sample evaluation follow in the machine-learning workflow session. ::: ## Task 3.1 — Create and interpret the matrix Create a labeled $2 \times 2$ table. Rows should be actual outcomes and columns predicted outcomes. ```{r} # YOUR CODE HERE ``` Identify true negatives, false positives, false negatives, and true positives. ## Task 3.2 — Compute evaluation metrics Compute and interpret accuracy, precision, and recall. Recall is also called sensitivity or the true-positive rate. ## Task 3.3 — Business reflection LendWise is trying to grow its customer base aggressively. - Why might the company accept more false negatives temporarily? - Why might LendWise lower the threshold during an economic downturn? # Part 4 — ROC curve and AUC Use `pROC` to evaluate discrimination across all possible thresholds. Here, higher predicted probabilities should correspond to `default = 1`, so `direction = "<"` states that controls tend to have lower scores than cases. ```{r} # YOUR CODE HERE ``` ## Task 4.1 — Understanding ROC - What does the ROC curve represent? - What are TPR and FPR? ## Task 4.2 — Interpreting AUC - What does AUC represent? - What would values near 0.5, 0.7, and 1.0 suggest? ## Task 4.3 — Threshold perspective - Why is ROC/AUC useful before the company fixes one decision threshold? # Part 5 — Compare early-career urban borrowers and platform workers Repeat the same model specification for platform workers, then compare coefficients and discrimination. ## Task 5.1 — Fit the second model Fit the same logistic-regression specification for platform workers and inspect `summary(model_platform)`. ```{r} # YOUR CODE HERE ``` ## Task 5.2 — Compare coefficients - Are directions and statistical evidence the same in both groups? - How do meaningful unit changes or odds ratios differ? ```{r} # YOUR CODE HERE ``` ## Task 5.3 — Compare ROC and AUC - Which category has the higher in-sample AUC? - What might explain the difference? ```{r} # YOUR CODE HERE ``` # 🧩 Wrap-up {#wrap-up} 🎉🎈 You have completed the exercise—good work! 🎈🎉 In this exercise, we have learned to: - Fit and interpret logistic regression models in R with `glm()` - Interpret log-odds coefficients and odds ratios - Generate predicted probabilities with `predict()` - Convert probabilities into decisions using explicit thresholds - Understand threshold trade-offs and their business costs - Interpret confusion matrices and compute accuracy, precision, and recall - Understand ROC curves and AUC - Compare model behavior across customer segments