--- session: "Session 6" title: "Machine Learning Foundations: Workflow" session_id: session-06 format: html: default --- In this exercise, we cover: | Part | Topic | Time (min) | |-------------|------------------------------------------|-----------:| | 1 | Define the problem and split the data | 12 | | 2 | Specify preprocessing without leakage | 10 | | 3–4 | Specify models and workflows | 10 | | 5–6 | Cross-validation and baseline evaluation | 15 | | 7 | Tune a decision tree | 15 | | 8 | Compare and select models | 8 | | 9 | Final held-out test evaluation | 12 | | 10 | Generalization reflection | 8 | | **Overall** | | **90** | ## Learning objective Implement a supervised machine-learning workflow in R and explain how splitting, resampling, preprocessing, tuning, and final testing support **generalization**. We continue the LendWise credit-default case from Session 5. There, the focus was how classification probabilities and metrics work. Here, the focus is whether a model will perform on **unseen customers** and how to compare model classes without contaminating the final evaluation. A short reminder: - supervised learning estimates a relationship between features and a known target; - unsupervised learning searches for structure without a predefined target; - our supervised target is `default`. ### Package setup Before starting this exercise, make sure the required R packages are installed. Run this command once in the R console. You do not need to run it every time you open the exercise. You can skip this step if these packages are already installed. ```{r} #| eval: false install.packages(c( "tidyverse", "tidymodels" )) ``` ```{r} library(tidyverse) library(tidymodels) ``` We use the `tidymodels` ecosystem to implement the machine-learning workflow. It includes `yardstick`, which provides functions for evaluating model performance (e.g., accuracy, sensitivity, ROC AUC, and confusion matrices). In Session 5, we calculated some of these metrics manually. Here, we use standardized functions to evaluate models consistently across cross-validation and final testing. # Part 1 — Define the problem and split the data {#split} We use the complete `credit_default.csv` dataset and retain `customer_category` as a predictor. The event of interest is **default**. We therefore define `default` as the first factor level, following the convention used by our evaluation functions. ## Task 1.1 — Load and inspect the data 1. Load the csv from `https://raw.githubusercontent.com/fs-ise/machine-learning-for-big-data/main/exercises/data/credit_default.csv`. 2. Convert the numeric outcome to a factor whose levels are ordered `default`, then `no_default`. 3. Inspect the data, row count, and outcome proportions. ```{r} # YOUR CODE HERE ``` ## Task 1.2 — Create a held-out test set Create a reproducible 80/20 split, stratified by `default`. Store the partitions as `train_data` and `test_data`, then compare row counts and class proportions. ```{r} # YOUR CODE HERE ``` Why must `test_data` remain untouched until model selection is complete? **Your answer:** *Write your answer here.* # Part 2 — Specify preprocessing without leakage {#recipe} A [recipe](https://recipes.tidymodels.org/reference/recipe.html) records what transformations should happen. It does not estimate normalization statistics until a workflow is fitted. During cross-validation, those statistics are learned separately from each analysis fold and then applied to its assessment fold. ## Task 2.1 — Create the recipe Use `train_data` to specify a recipe that: 1. predicts `default` from all remaining variables; 2. creates indicator columns for nominal predictors such as `customer_category`; 3. normalizes numeric predictors. ```{r} # YOUR CODE HERE ``` Normalization is useful for many models, including logistic regression when predictors have very different units. Trees generally do not require it, but using one common recipe makes the comparison concise; the tree's split ordering is unchanged by a monotonic rescaling. # Part 3 — Specify two model classes {#models} We deliberately compare two different model classes: - logistic regression is relatively constrained and represents effects through a linear log-odds function; - a decision tree can represent nonlinear relationships and interactions, but greater flexibility can increase overfitting risk. ## Task 3.1 — Specify the models Create: 1. an untuned logistic regression using the `glm` engine; 2. a classification tree using the `rpart` engine, with `cost_complexity`, `tree_depth`, and `min_n` marked for tuning. ```{r} # YOUR CODE HERE ``` # Part 4 — Bundle preprocessing and models {#workflows} A [workflow](https://workflows.tidymodels.org/) bundles a recipe and model specification. This ensures the same transformations are applied consistently in resampling, final fitting, and prediction. ## Task 4.1 — Create two workflows Create `logistic_workflow` and `tree_workflow` by combining the shared recipe with each model specification. ```{r} # YOUR CODE HERE ``` # Part 5 — Cross-validation {#cross-validation} Cross-validation creates temporary analysis and assessment partitions **inside the training data**. These folds support model comparison and tuning; they do not replace the final held-out test set. ## Task 5.1 — Create stratified folds Create five reproducible folds from `train_data`, stratified by `default`. Confirm that the object contains five splits. ```{r} # YOUR CODE HERE ``` Discuss: 1. What role do the assessment portions of the folds play? **Your answer:** *Write your answer here.* 2. How do they differ from `test_data`? **Your answer:** *Write your answer here.* 3. Why would repeated model selection using test performance bias the reported final result? **Your answer:** *Write your answer here.* # Part 6 — Evaluate the logistic baseline {#baseline} We now evaluate the logistic regression model using `yardstick`. Instead of calculating metrics manually, `metric_set()` combines ROC AUC, accuracy, and sensitivity into a reusable collection of evaluation metrics. We apply the same metrics during cross-validation, model tuning, and final testing. ## Task 6.1 — Resample the baseline workflow 1. Create `classification_metrics` with `roc_auc`, `accuracy`, and `sens`. 2. Fit the logistic workflow across `folds`. 3. Collect the mean and standard error for each metric. ```{r} # YOUR CODE HERE ``` # Part 7 — Tune the decision tree {#tuning} The tree's hyperparameters control its flexibility: - **`cost_complexity`:** Penalizes complex trees. Higher values encourage smaller trees. - **`tree_depth`:** Maximum number of splits along a path. Higher values allow deeper trees. - **`min_n`:** Minimum number of observations required in a node to attempt a split. Higher values restrict splitting. We will examine the decision-tree algorithm in the next session. For now, we use a small tuning grid to keep computation time short. ## Task 7.1 — Create and evaluate a tuning grid Use [expand_grid](https://tidyr.tidyverse.org/reference/expand_grid.html) to create all combinations of: - `cost_complexity`: 0.0001, 0.001, 0.01; - `tree_depth`: 2, 4, 6; - `min_n`: 10, 25. Evaluate the grid using the **same training folds and metrics**. Select the configuration with the best mean ROC AUC. ```{r} # YOUR CODE HERE ``` At any point in this task, did the tree see `test_data`? **Your answer:** *Write your answer here.* ## Task 7.2 — Visualize tuning results Visualize how maximum tree depth affects predictive performance. Use `collect_metrics(tree_results)` to retrieve the results and: - Keep only ROC AUC results. - Fix `min_n = 25` and `cost_complexity = 0.001`. - Plot `tree_depth` on the X-axis and mean ROC AUC on the Y-axis. - Connect the points to illustrate changes in performance. ```{r} # YOUR CODE HERE ``` # Part 8 — Compare and select models {#comparison} We now compare the logistic regression baseline with the tuned decision tree. Model selection uses cross-validation results, **not** the held-out test set. We compare mean ROC AUC and consider the uncertainty of the estimates. ## Task 8.1 — Compare validation ROC AUC Work through the provided code step by step. Inspect the intermediate results and focus on what the comparison means rather than writing another transformation pipeline from scratch. **Step 1 — Prepare the logistic regression results** Inspect `logistic_cv_metrics`. Then keep ROC AUC with `filter()` and use `transmute()` to create consistently named columns. ```{r} logistic_cv_metrics logistic_summary <- logistic_cv_metrics |> filter(.metric == "roc_auc") |> transmute( model = "Logistic regression", mean_roc_auc = mean, std_err = std_err ) logistic_summary ``` **Step 2 — Retrieve the best decision tree** `show_best()` retrieves the configuration with the highest mean validation ROC AUC across the entire tuning grid. The column names must match Step 1. ```{r} tree_summary <- show_best( tree_results, metric = "roc_auc", n = 1 ) |> transmute( model = "Decision tree", mean_roc_auc = mean, std_err = std_err ) tree_summary ``` **Step 3 — Compare and select** `bind_rows()` combines the summaries and `arrange()` sorts the models. The selection is stored in `selected_model` for Part 9. ```{r} model_comparison <- bind_rows( logistic_summary, tree_summary ) |> arrange(desc(mean_roc_auc)) model_comparison selected_model <- if_else( model_comparison$model[1] == "Decision tree", "decision_tree", "logistic_regression" ) selected_model ``` **Interpret the results:** 1. Which model has the higher mean validation ROC AUC, and how large is the difference? 2. What do the standard errors tell us about the precision of the estimated mean ROC AUC values? 3. Does the observed improvement justify the additional complexity of the tree, or could logistic regression be sufficient? Remember: the reported standard errors describe uncertainty in the individual estimated means, **not directly in their difference**. # Part 9 — Finalize and evaluate once on the test set {#final-test} Only now do we use the held-out test observations. We finalize the selected workflow, refit it on all training observations, and evaluate exactly once on `test_data` through `last_fit()`. ## Task 9.1 — Finalize the selected workflow If the tree won, insert `best_tree` into `tree_workflow`. Otherwise, use `logistic_workflow` unchanged. Then call `last_fit()` with the original `data_split`. ```{r} # YOUR CODE HERE ``` ## Task 9.2 — Inspect final predictions Use yardstick to create a test confusion matrix and ROC curve. # Part 10 — Reflect on generalization {#reflection} Answer the following questions concisely. 1. Why could training accuracy be misleading? 2. Why must preprocessing be inside the workflow and resampling process? 3. Why should the test set be used only after model choices are finalized? 4. Why can a more flexible model perform worse on unseen observations? 5. What is the role of cross-validation? 6. What should happen after final test evaluation and before deployment? ## Wrap-up 🎉🎈 You have completed the exercise—good work! 🎈🎉 You implemented the standard supervised workflow: ```text split → recipe → workflows → training-data cross-validation → model selection → finalization → one held-out test evaluation ``` You also connected each step to generalization, overfitting, leakage, selection bias, and the bias–variance trade-off.