--- session: "Session 7" title: "Machine Learning Foundations: Algorithms" session_id: session-07 format: html: default --- In this exercise, we cover: | Part | Topic | Time (min) | |---|---|---:| | 1 | Common evaluation setup | 10 | | 2 | Regularized logistic regression | 15 | | 3 | k-nearest neighbors | 11 | | 4 | Decision tree | 12 | | 5 | Random forest | 12 | | 6 | RBF support vector machine | 12 | | 7 | Validation comparison and recommendation | 8 | | 8 | Final held-out evaluation | 7 | | Wrap-up | Reflection | 5 | | **Overall** | | **90** | ## Learning objectives After this exercise, you should be able to: - apply selected machine-learning algorithms in R; - explain how their main tuning parameters control model flexibility; - compare regularized logistic regression, k-NN, decision trees, random forests, and SVMs; - compare algorithms using a common validation procedure; - consider interpretability, computational cost, preprocessing requirements, and predictive performance; - recommend an appropriate model for a business prediction task. > In Session 6, we learned that models must be compared using the same training data and validation procedure. Here, we keep that workflow fixed and vary the algorithm. ### Package setup Before starting this exercise, make sure the required R packages are installed. Run this command once in the R console. You do not need to run it every time you open the exercise. You can skip this step if these packages are already installed. ```{r} #| eval: false install.packages(c( "tidyverse", "tidymodels", "glmnet", "kknn", "ranger", "kernlab", "rpart.plot" )) ``` ```{r} library(tidyverse) library(tidymodels) ``` # Part 1 — Establish the common evaluation setup {#setup} We continue the LendWise default-classification task from Sessions 5 and 6. Every algorithm receives the same training observations and five cross-validation folds. The held-out test set remains unused until a recommendation has been made from validation results. ## Task 1.1 — Re-establish the Session 6 setup Run the common setup and verify: - `default` is the first factor level and therefore the event for sensitivity and ROC AUC; - the split is stratified and reproducible; - all five folds come only from `train_data`. ```{r} df <- read_csv( "https://raw.githubusercontent.com/fs-ise/machine-learning-for-big-data/main/exercises/data/credit_default.csv", show_col_types = FALSE ) |> mutate( default = factor( default, levels = c(1, 0), labels = c("default", "no_default") ) ) set.seed(42) data_split <- initial_split(df, prop = 0.8, strata = default) train_data <- training(data_split) test_data <- testing(data_split) set.seed(42) folds <- vfold_cv(train_data, v = 5, strata = default) classification_metrics <- metric_set( roc_auc, accuracy, sens ) levels(train_data$default) nrow(train_data) nrow(test_data) folds ``` Why must every algorithm use these same folds? Why must `test_data` remain outside model comparison? **Your answer:** *Write your answer here.* ## Task 1.2 — Define algorithm-specific preprocessing Some algorithms depend directly on distances or coefficient penalties, while tree splits are largely insensitive to predictor scale. Complete two recipes: - `scaled_recipe` dummy-encodes nominal predictors and normalizes numeric predictors; - `tree_recipe` keeps the original predictors without normalization. Which model families should use each recipe? **Your answer:** *Write your answer here.* ```{r} # YOUR CODE HERE ``` # Part 2 — Regularized logistic regression {#regularized} Regularization balances fit and simplicity. `penalty` controls its strength: larger values shrink coefficients more. `mixture = 0` gives ridge, `mixture = 1` gives lasso, and intermediate values give elastic net. ## Task 2.1 — Tune the model Run the supplied specification and workflow to tune the 18 explicit combinations in `regularized_grid` on the shared folds. The wider penalty range makes both mild and substantial shrinkage available for comparison. ```{r} regularized_grid <- expand_grid( penalty = c(0.0001, 0.001, 0.01, 0.1, 1, 10), mixture = c(0, 0.5, 1) ) regularized_spec <- logistic_reg( penalty = tune(), mixture = tune() ) |> set_engine("glmnet") |> set_mode("classification") regularized_workflow <- workflow() |> add_recipe(scaled_recipe) |> add_model(regularized_spec) set.seed(42) regularized_results <- tune_grid( regularized_workflow, resamples = folds, grid = regularized_grid, metrics = classification_metrics ) best_regularized <- select_best( regularized_results, metric = "roc_auc" ) best_regularized show_best(regularized_results, metric = "roc_auc", n = 5) ``` How does increasing `penalty` change model flexibility? Why is normalization appropriate here? **Your answer:** *Write your answer here.* ## Task 2.2 — Compare unregularized, ridge, and lasso coefficients Task 2.1 used cross-validation to choose tuning parameters for predictive model selection. Here, instead, use one shared, moderate penalty chosen for a clear conceptual comparison. Keeping the penalty strength fixed lets you attribute differences between ridge and lasso coefficients to the type of penalty rather than to two different CV-selected penalty values. The supplied code uses `comparison_penalty <- 0.05` for both ridge and lasso and fits all three models to `train_data` with `scaled_recipe`. This value gives more instructive contrast than the initial value of `0.01` without collapsing the ridge coefficients. Complete the plot by mapping coefficient `term` to the y-axis and `estimate` to the x-axis. ```{r} comparison_penalty <- 0.05 penalized_coefficients <- tibble( model = c("Ridge", "Lasso"), mixture = c(0, 1) ) |> mutate( fitted_workflow = map( mixture, \(mixture_value) { workflow() |> add_recipe(scaled_recipe) |> add_model( logistic_reg( penalty = comparison_penalty, mixture = mixture_value ) |> set_engine("glmnet") |> set_mode("classification") ) |> fit(data = train_data) } ), coefficients = map( fitted_workflow, \(fitted) tidy(extract_fit_parsnip(fitted)) ) ) |> select(model, coefficients) |> unnest(coefficients) |> filter(term != "(Intercept)") unregularized_workflow <- workflow() |> add_recipe(scaled_recipe) |> add_model( logistic_reg() |> set_engine("glm") |> set_mode("classification") ) unregularized_coefficients <- unregularized_workflow |> fit(data = train_data) |> extract_fit_parsnip() |> tidy() |> filter(term != "(Intercept)") |> transmute( model = "Unregularized", term, estimate ) coefficient_comparison <- bind_rows( unregularized_coefficients, penalized_coefficients |> select(model, term, estimate) ) |> mutate( model = factor( model, levels = c("Unregularized", "Ridge", "Lasso") ) ) |> complete( model, term = unregularized_coefficients$term, fill = list(estimate = 0) ) ``` ```{r} # YOUR CODE HERE: plot estimate by term, colored by model. ``` Compare ridge and lasso with the unregularized baseline. How does ridge shrink the coefficients while retaining meaningful contributions across predictors? Under the **same penalty strength**, how does lasso differ from ridge, and does it set any coefficients exactly to zero? Why does ridge usually keep all coefficients nonzero while lasso can perform variable selection? Finally, why is this teaching penalty appropriate for conceptual comparison but not a replacement for the cross-validation used for predictive model selection in Task 2.1? **Your answer:** *Write your answer here.* # Part 3 — k-nearest neighbors {#knn} k-NN predicts from nearby labeled observations. Small `k` produces flexible, high-variance boundaries; large `k` smooths over more observations and increases bias. ## Task 3.1 — Tune the number of neighbors Use Euclidean distance, rectangular neighbor weights, and candidate values 3, 5, 9, 15, and 25. ::: {.callout-tip title="Useful documentation"} See the [`nearest_neighbor()` documentation](https://parsnip.tidymodels.org/reference/nearest_neighbor.html) for its model-specific arguments and engine choices. ::: ```{r} knn_grid <- tibble( neighbors = c(3L, 5L, 9L, 15L, 25L, 30L, 40L, 50L) ) # YOUR CODE HERE: complete knn_spec, knn_workflow, and knn_results. ``` Why would unscaled `income` dominate predictors that range between 0 and 1? **Your answer:** *Write your answer here.* ## Task 3.2 — Interpret model flexibility Inspect the ROC AUC plot. How do very small and larger values of `k` change the smoothness and flexibility of the decision boundary? Describe the observed validation pattern and explain it using bias, variance, and generalization. Does the most flexible candidate perform best? **Your answer:** *Write your answer here.* # Part 4 — Decision tree {#tree} A tree recursively partitions predictor space. Greater depth and smaller terminal-node requirements increase flexibility, while a larger `cost_complexity` encourages pruning. ## Task 4.1 — Tune tree complexity Use eight combinations: cost complexity 0.001 or 0.01, depth 3 or 6, and minimum node size 10 or 25. ```{r} # YOUR CODE HERE: complete the tree specification, workflow, grid, and tuning call. ``` ## Task 4.2 — Fit and interpret the selected tree Run the supplied code to finalize the ROC-AUC-optimal tree, fit it to all training observations, and plot it. Interpret one or two important splits in substantive LendWise terms: which applicants are separated, and how do their predicted outcomes differ? Then explain how `tree_depth` and pruning through `cost_complexity` control flexibility. ```{r} #| fig-width: 14 #| fig-height: 9 #| out-width: 100% final_tree_workflow <- finalize_workflow( tree_workflow, best_tree ) fitted_tree_workflow <- fit( final_tree_workflow, data = train_data ) fitted_tree <- extract_fit_engine(fitted_tree_workflow) rpart.plot::rpart.plot( fitted_tree, type = 4, extra = 104, fallen.leaves = TRUE, tweak = 0.75, varlen = 0, faclen = 0, digits = 3, branch = 0.4, box.palette = "GnBu" ) ``` **Your answer:** *Write your answer here.* # Part 5 — Random forest {#forest} A random forest aggregates trees trained on bootstrap samples while considering random predictor subsets. Aggregation generally reduces the instability of one tree, but the ensemble is harder to explain as a single decision path. ## Task 5.1 — Tune the forest Keep `trees = 500` fixed. Tune `mtry` over 2, 4, and 6 predictors and `min_n` over 10 and 25. ::: {.callout-tip title="Useful documentation"} See the [`rand_forest()` documentation](https://parsnip.tidymodels.org/reference/rand_forest.html) for its model-specific arguments, including `mtry`, `trees`, and `min_n`. ::: ```{r} # YOUR CODE HERE: complete rf_spec, rf_workflow, rf_grid, and rf_results. ``` Why might a forest predict better than one tree? What interpretability is lost? **Your answer:** *Write your answer here.* # Part 6 — RBF support vector machine {#svm} An SVM seeks a decision boundary with a wide margin. `cost` controls how strongly classification errors are penalized. The RBF kernel permits nonlinear boundaries, while `rbf_sigma` controls how locally the kernel responds and therefore affects flexibility. ## Task 6.1 — Tune cost and kernel scale Tune six combinations: cost 0.25, 1, or 4 and RBF sigma 0.01 or 0.1. Use normalized predictors. ::: {.callout-tip title="Useful documentation"} See the [`svm_rbf()` documentation](https://parsnip.tidymodels.org/reference/svm_rbf.html) for model-specific guidance on `cost` and `rbf_sigma`. ::: ```{r} # YOUR CODE HERE: complete svm_spec, svm_workflow, svm_grid, and svm_results. ``` ## Task 6.2 — Interpret the SVM tuning surface Use the validation plot to answer: which `cost` and `rbf_sigma` combinations allow more local, flexible decision boundaries, and which encourage smoother boundaries? In this grid, does greater flexibility consistently improve validation ROC AUC? Support your answer with the plotted pattern. **Your answer:** *Write your answer here.* # Part 7 — Compare validation performance and recommend a model {#comparison} All five algorithms have now been evaluated using the same folds. ROC AUC is the primary selection metric, while accuracy, sensitivity, uncertainty, and organizational requirements provide additional context. ## Task 7.1 — Collect each model's best metrics Run the supplied helper and comparison code. It retrieves ROC AUC, accuracy, and sensitivity at each model's ROC-AUC-optimal configuration. ```{r} collect_best_metrics <- function(results, best_parameters, model_name) { all_metrics <- collect_metrics(results) parameter_names <- intersect( names(best_parameters), names(all_metrics) ) |> setdiff(".config") all_metrics |> inner_join( best_parameters, by = parameter_names ) |> mutate(model = model_name) |> select( model, .metric, mean, std_err, any_of(parameter_names) ) } validation_comparison <- bind_rows( collect_best_metrics( regularized_results, best_regularized, "Regularized logistic" ), collect_best_metrics( knn_results, best_knn, "k-NN" ), collect_best_metrics( tree_results, best_tree, "Decision tree" ), collect_best_metrics( rf_results, best_rf, "Random forest" ), collect_best_metrics( svm_results, best_svm, "RBF SVM" ) ) validation_comparison |> arrange(.metric, desc(mean)) ``` ## Task 7.2 — Visualize the comparison Create a plot that compares the mean validation metrics across algorithms and includes uncertainty based on the cross-validation standard errors. ::: {.callout-tip title="Plotting hint"} Use `geom_errorbar()` for intervals such as `mean ± 1.96 * std_err`. `position_dodge()` can keep metrics from overlapping, and `coord_flip()` can make model names easier to read. ::: ## Task 7.3 — Interpretability and predictive power Typical properties—not absolute rules—include: | Model | Typical strength | Typical limitation | Scaling needed here? | |---|---|---|---| | Regularized logistic | interpretable and stable | mostly linear decision boundary | yes | | k-NN | simple and locally flexible | scaling and dimensionality sensitive | yes | | Decision tree | transparent decision paths | unstable and can overfit | no | | Random forest | strong flexible predictions | less directly interpretable | no | | RBF SVM | powerful nonlinear boundaries | harder to interpret and tune | yes | Answer: 1. Inspect the full comparison table—not only the ROC AUC ranking. Which model has the strongest mean validation ROC AUC? Is there a model with a competitive ROC AUC but clearly problematic accuracy or sensitivity? **Your answer:** *Write your answer here.* 2. What does `sens = 0` mean substantively for LendWise? **Your answer:** *Write your answer here.* 3. How can a model obtain a reasonably strong ROC AUC while predicting no defaults at the standard classification threshold? **Your answer:** *Write your answer here.* 4. Are ROC AUC differences large relative to their standard errors? Which models naturally capture nonlinear relationships, and is the most flexible model necessarily best here? **Your answer:** *Write your answer here.* 5. What matters for LendWise besides the highest score when choosing a model and its operating threshold? **Your answer:** *Write your answer here.* ## Task 7.4 — Lock the recommendation using validation only The code below identifies the highest mean validation ROC AUC. Before running the final evaluation, write a short recommendation that also considers stability and interpretability. # Part 8 — Final held-out test evaluation {#final-test} We now use the test set once. This is not another opportunity to tune the winner. ## Task 8.1 — Finalize the selected workflow Run the selection map, finalize the chosen workflow with its best parameters, and evaluate it with `last_fit()`. ## Task 8.2 — Compare validation and test performance Compare the selected model's cross-validation metrics with its held-out test metrics. Then inspect the confusion matrix and ROC curve. Does the held-out result broadly support the validation-based recommendation? **Your answer:** *Write your answer here.* ## Wrap-up 🎉🎈 You have completed the exercise—good work! 🎈🎉 You have learned to: - apply regularized logistic regression, k-NN, decision trees, random forests, and RBF SVMs; - connect tuning parameters to model flexibility; - recognize algorithm-specific preprocessing requirements; - compare algorithms under one cross-validation framework; - balance predictive performance, stability, computation, and interpretability; - select and evaluate a final model on unseen data.