---
session: "Session 9"
title: "Deployment in Organizations: Explainable AI"
session_id: session-09
format: html
---


In this exercise, we return to the **LendWise loan-default case** from previous sessions. The bank has developed several predictive models. Before deploying them, we need to understand how their predictions are produced and how they can be explained to different stakeholders.

We first examine models that are **inherently interpretable**, where the model structure itself provides an explanation. We then treat a random forest as a black box and apply **post-hoc explainability methods**.

| Part | Topic | Time (min) |
|---|---|---:|
| 1 | Three models, one applicant | 15 |
| 2 | Logistic regression: coefficients as explanations | 15 |
| 3 | Decision tree: rules as explanations | 10 |
| 4 | Global post-hoc explanation: permutation importance | 15 |
| 5 | From average effects to individual what-if profiles | 15 |
| 6 | Local post-hoc explanation with SHAP | 10 |
| Wrap-up | Match the explanation to the question | 10 |
| **Overall** | | **90** |

### Learning objectives

After this exercise, you should be able to:

- distinguish **inherently interpretable models** from **post-hoc explanations**;
- distinguish **global** from **local** explanations;
- interpret logistic-regression coefficients and shallow decision trees;
- use permutation importance to investigate which variables a black-box model relies on;
- use partial-dependence and individual profiles to investigate how predictions change;
- use SHAP values to explain an individual prediction;
- identify important limitations of model explanations;
- select explanation methods for different organizational stakeholders.

### Package setup

Run this command once in the R console. You can skip this step if the packages are already installed.

```{r}
#| eval: false
install.packages(c(
  "tidyverse",
  "tidymodels",
  "DALEX",
  "DALEXtra",
  "ranger",
  "rpart",
  "rpart.plot"
))
```


```{r}
library(tidyverse)
library(tidymodels)

tidymodels_prefer()

df <- read_csv(
  "https://raw.githubusercontent.com/fs-ise/machine-learning-for-big-data/main/exercises/data/credit_default.csv",
  show_col_types = FALSE
) |>
  mutate(
    customer_category = factor(customer_category),
    # The event of interest ("default") is the SECOND level.
    # R's glm() models the probability of the second level,
    # so raw glm coefficients then refer directly to default.
    # tidymodels and yardstick treat the FIRST level as the event
    # by default, so metric calls need event_level = "second".
    default = factor(
      default,
      levels = c(0, 1),
      labels = c("no_default", "default")
    )
  )

set.seed(42)

data_split <- initial_split(
  df,
  prop = 0.8,
  strata = default
)

train_data <- training(data_split)
test_data <- testing(data_split)

logistic_recipe <- recipe(default ~ ., data = train_data) |>
  step_dummy(all_nominal_predictors())

tree_recipe <- recipe(default ~ ., data = train_data)

logistic_spec <- logistic_reg() |>
  set_engine("glm")

tree_spec <- decision_tree(
  tree_depth = 3,
  min_n = 20,
  cost_complexity = 0
) |>
  set_engine("rpart") |>
  set_mode("classification")

rf_spec <- rand_forest(
  trees = 500,
  mtry = 3,
  min_n = 10
) |>
  set_engine("ranger") |>
  set_mode("classification")

logistic_fit <- workflow() |>
  add_recipe(logistic_recipe) |>
  add_model(logistic_spec) |>
  fit(data = train_data)

tree_fit <- workflow() |>
  add_recipe(tree_recipe) |>
  add_model(tree_spec) |>
  fit(data = train_data)

set.seed(42)

rf_fit <- workflow() |>
  add_recipe(tree_recipe) |>
  add_model(rf_spec) |>
  fit(data = train_data)

predict_default_probability <- function(model, new_data) {
  predict(
    model,
    new_data = new_data,
    type = "prob"
  ) |>
    pull(.pred_default)
}

rf_test_probability <- predict_default_probability(
  rf_fit,
  test_data
)

applicant_index <- which.min(
  abs(rf_test_probability - 0.5)
)

applicant <- test_data |>
  slice(applicant_index) |>
  select(-default)

applicant_outcome <- test_data$default[[applicant_index]]

set.seed(42)

explanation_data <- test_data |>
  slice(-applicant_index) |>
  slice_sample(
    n = min(500, nrow(test_data) - 1)
  )

explainer_rf <- DALEXtra::explain_tidymodels(
  rf_fit,
  data = explanation_data |> select(-default),
  y = as.numeric(explanation_data$default == "default"),
  predict_function = predict_default_probability,
  label = "Random forest",
  type = "classification",
  verbose = FALSE
)
```


Inspect the variables to familiarize yourself with the predictors and outcome.

```{r}
glimpse(df)
```


# Part 1 — Three models, one applicant {#models}

The three models solve exactly the same prediction problem:

- **logistic regression** uses one additive equation;
- the **decision tree** uses explicit decision rules;
- the **random forest** combines predictions from 500 trees.

Throughout the exercise, we explain the same applicant. It was selected because the random-forest probability is closest to 0.5, making it a borderline case rather than an obviously easy prediction. The shallow tree uses depth 3 and `min_n = 20`.

An **inherently interpretable** model exposes understandable prediction logic; a **post-hoc** method explains a fitted model after training. **Global** explanations describe behavior across cases, while **local** explanations describe one case. **Model-specific** methods use a particular model structure; **model-agnostic** methods use inputs and predictions and can be applied across model types.

## Task 1.1 — Compare the predictions


Inspect `applicant`.

Then calculate the predicted probability of default for the applicant using:

- `logistic_fit`;
- `tree_fit`;
- `rf_fit`.

The helper function

```r
predict_default_probability(model, applicant)
```

returns the probability of default.

```{r}
applicant

# YOUR CODE HERE
```

At a 0.5 threshold, which class does each model predict? Even when predictions agree, could the reasons behind them differ?

**Your answer:** *Write your answer here.*

Can the predicted probability by itself explain **why** the model produced the prediction?

**Your answer:** *Write your answer here.*



## Task 1.2 — Classify the models


Classify each model before continuing.

| Model | Inherently interpretable or black box? | Why? |
|--|----|---|
| Logistic regression | | |
| Shallow decision tree | | |
| Random forest | | |



## Task 1.3 — Check predictive quality


Compute test-set ROC AUC for all three models with yardstick. Why does model quality matter before explaining a model, and what would a much weaker tree imply about the fidelity of its explanation to the random forest?

```{r}
model_fits <- list(logistic = logistic_fit, tree = tree_fit, random_forest = rf_fit)

# Hint: `default` is the second factor level, so pass `event_level = "second"`
# to yardstick metrics such as roc_auc().
# YOUR CODE HERE
```

**Your answer:** *Write your answer here.*



# Part 2 — Logistic regression: coefficients as explanations {#logistic}

A logistic regression exposes its internal structure directly. Each predictor has a coefficient determining how it changes the **log-odds** of default, holding other predictors constant.

## Task 2.1 — Inspect the coefficients


Extract the fitted `glm` model with `extract_fit_engine()` and inspect its coefficients using `broom::tidy()`.

Because `default` is the second factor level, raw `glm` models the log-odds of **default** directly, so the coefficients need no sign adjustment. Add `odds_ratio = exp(estimate)` to express each effect as a multiplicative change in the odds of default.

Ignore `(Intercept)`, inspect `p.value`, and remember that coefficient magnitudes are not comparable across predictors on different scales unless predictors are standardized. Check the reference level with `levels(train_data$customer_category)`.

::: {.callout-tip title="Useful documentation"}
See the [`logistic_reg()` documentation](https://parsnip.tidymodels.org/reference/logistic_reg.html).
:::

```{r}
coef_tbl <- logistic_fit |>
  extract_fit_engine() |>
  broom::tidy() |>
  mutate(odds_ratio = exp(estimate))

# YOUR CODE HERE
```

Identify one predictor associated with **higher** predicted default risk and one associated with **lower** predicted default risk.

**Your answer:** *Write your answer here.*

As a sanity check, does each direction agree with the tree or partial-dependence profile?

**Your answer:** *Write your answer here.*

What does an odds ratio greater than `1` mean?

**Your answer:** *Write your answer here.*



## Task 2.2 — Explain the applicant


Use the applicant's encoded values (and reference categories) together with `coef_tbl`. Here, the **contribution** of a predictor is its coefficient times the difference between the applicant's value and the training average of that predictor, $\beta_j \times (x_j^{\text{applicant}} - \bar{x}_j^{\text{train}})$. Coefficient × raw value is usually uninformative because it depends on where the predictor's zero lies.

```{r}
# Bake the applicant and the training data with the fitted recipe.
# For each predictor, compute: coefficient * (applicant value - training mean).
# Sort by absolute contribution.
# YOUR CODE HERE
```

Which characteristics of this applicant appear to push the logistic-regression prediction toward **higher** default risk, and which push it toward **lower** risk?

**Your answer:** *Write your answer here.*

Why is this explanation **not** evidence that changing one of these variables would causally change the applicant's default risk?

**Your answer:** *Write your answer here.*



# Part 3 — Decision tree: rules as explanations {#tree}

A shallow decision tree provides another form of inherent interpretability. Instead of coefficients, it represents the prediction as a sequence of **if/then rules**.

## Task 3.1 — Visualize the tree


Extract the fitted `rpart` model and visualize it. With `extra = 104`, each node shows the predicted class, class probabilities, and percentage of observations; probabilities are listed in factor-level order, so `no_default` comes first and `default` second. Use `nn = TRUE` to number nodes, and `rpart.plot::rpart.predict(..., nn = TRUE)` to check the applicant's leaf.

::: {.callout-tip title="Useful documentation"}
See the [`decision_tree()` documentation](https://parsnip.tidymodels.org/reference/decision_tree.html) and [`rpart.plot`](https://cran.r-project.org/package=rpart.plot).
:::

```{r}
tree_engine <- extract_fit_engine(tree_fit)
applicant

# YOUR CODE HERE
```

Trace the applicant manually through the tree.

Which variables determine the path?

**Your answer:** *Write your answer here.*

A shallow tree may predict `no_default` in most leaves when defaults are rare.

Which terminal node does the applicant reach, and what does that node tell you?

**Your answer:** *Write your answer here.*



## Task 3.2 — Compare two forms of inherent interpretability


Compare the logistic regression and decision tree.

| Question | Logistic regression | Decision tree |
|---|---|---|
| How is the model represented? | | |
| How can we obtain a global understanding? | | |
| How can we explain one prediction? | | |
| What could make the model difficult to interpret? | | |



# Part 4 — Global post-hoc explanation: permutation importance {#importance}

The random forest contains hundreds of decision trees. Instead of inspecting the complete model, we now use **model-agnostic post-hoc explanations**.

DALEX provides a common interface for explaining different predictive models. An `explainer` connects a fitted model, data, observed outcomes, and its prediction function; `DALEXtra` integrates this interface with tidymodels. The random-forest explainer `explainer_rf` has already been prepared for you. Explanations use a held-out test sample so their summaries are not based on the same cases used to fit the model.

| Method | Scope | Main question | DALEX function |
|----|--|-----|---|
| Permutation importance | Global | Which variables does the model rely on? | `model_parts()` |
| Partial dependence | Global | How does the average prediction change with X? | `model_profile()` |
| Individual profile | Local | How would this prediction change if X changed? | `predict_profile()` |
| SHAP | Local | Which variables contributed to this prediction? | `predict_parts()` |

Permutation feature importance asks:

> How much worse does the model become when the information in one predictor is destroyed?

The predictor is randomly shuffled while the fitted model remains unchanged.
For classification, DALEX uses `1 - AUC` as its default loss; with `type = "difference"`, importance is the AUC lost relative to the full model. `_full_model_` is the unpermuted loss, `_baseline_` is the loss after all predictors are removed, and the boxplots show variation across repeated permutations.

## Task 4.1 — Calculate permutation importance


Use `DALEX::model_parts()` with the prepared `explainer_rf`.

Use `B = 10` permutations. Then plot the result.

::: {.callout-tip title="Useful documentation"}
See the [DALEX model-parts documentation](https://modeloriented.github.io/DALEX/reference/model_parts.html).
:::

```{r}
set.seed(42)

# YOUR CODE HERE
```

Which predictors does the random forest rely on most strongly?

**Your answer:** *Write your answer here.*



## Task 4.2 — Interpret importance carefully


Does permutation importance tell you whether increasing a variable increases or decreases default risk?

**Your answer:** *Write your answer here.*

Does it show that the variable causes default?

**Your answer:** *Write your answer here.*

Does it show that the variable is important for this particular applicant?

**Your answer:** *Write your answer here.*

What could happen to importance estimates when two predictors contain very similar information?

**Your answer:** *Write your answer here.*



# Part 5 — From average effects to individual what-if profiles {#profiles}

Feature importance tells us **whether** a variable matters, but not **how predictions change** as the variable changes.

We investigate two complementary views:

- a **partial-dependence profile (PDP)** summarizes average model predictions;
- an **individual profile** changes one predictor for one applicant while holding the applicant's other characteristics constant.

A PDP averages many individual what-if profiles and is therefore global. An individual profile instead keeps one observation fixed and varies one predictor at a time.

## Task 5.1 — Partial-dependence profiles


Create partial-dependence profiles for `debt_to_income` and `credit_utilization`.

Use `N = 200` observations to keep computation fast.

::: {.callout-tip title="Useful documentation"}
See [Partial-dependence profiles](https://ema.drwhy.ai/partialDependenceProfiles.html). A minimal call is `DALEX::model_profile(explainer_rf, variables = "debt_to_income", type = "partial")`; use `plot(..., geom = "profiles")` to reveal individual lines under the average.
:::

```{r}
set.seed(42)

# YOUR CODE HERE
```

For each predictor, describe the overall direction of the relationship, whether it appears approximately linear, and any regions where predictions change particularly strongly.

**Your answer:** *Write your answer here.*



## Task 5.2 — Explain the same applicant with an individual profile


Now calculate individual profiles for the same two variables and the selected `applicant`.

A **counterfactual explanation** seeks the smallest feasible change that flips a decision; a **ceteris-paribus profile** varies one variable while holding the others fixed. Here we create the latter.

::: {.callout-tip title="Useful documentation"}
See [Ceteris-paribus / individual profiles](https://ema.drwhy.ai/ceterisParibus.html). Use `predict_profile()` with arguments `new_observation` and `variables`.
:::

```{r}
# YOUR CODE HERE
```

Overlay the applicant profile and the PDP when comparing them. Where do they answer different questions?

**Your answer:** *Write your answer here.*

Suppose the individual profile shows that lower credit utilization would reduce the model's predicted default probability. Why should this **not automatically be presented as advice** to the applicant?

**Your answer:** *Write your answer here.*



# Part 6 — Local post-hoc explanation with SHAP {#shap}

We now ask:

> Why did the random forest assign this particular probability to this applicant?

SHAP explains one prediction relative to a **baseline prediction**. Each feature receives a contribution showing how much it pushes the prediction above or below that baseline. Together, the contributions explain the difference between the baseline and the prediction.


Suppose the model's **baseline probability of default is 20%**, while it predicts **55%** for one applicant.

SHAP distributes this difference across the applicant's feature values:

| Component | Contribution |
|---|---:|
| **Baseline prediction** | **20%** |
| High debt-to-income ratio | +18 pp |
| High credit utilization | +12 pp |
| Long customer relationship | −4 pp |
| Other features | +9 pp |
| **Prediction** | **55%** |

$$\text{Prediction} = \text{Baseline} + \sum_{j=1}^{p} \text{SHAP}_j$$

In this example:

$$55\% = 20\% + 18\% + 12\% - 4\% + 9\%$$

**Interpretation:** Positive SHAP values push this prediction **above** the baseline; negative values push it **below** the baseline.

SHAP explains the **model's prediction**, not the causal effect of changing a feature.



## Task 6.1 — Calculate a SHAP explanation


DALEX approximates Shapley values by sampling feature orderings.

Use `DALEX::predict_parts()` with:

- `new_observation = applicant`;
- `type = "shap"`;
- `B = 25`.

Here, `B = 25` controls the number of sampled orderings used in the approximation. Plot the result and print its baseline. The baseline is the mean prediction over `explanation_data`; check that baseline plus the summed contributions is approximately the applicant probability. Boxplots show variation across the `B` sampled orderings, and results can change with the seed.

::: {.callout-tip title="Useful documentation"}
See the [conceptual Shapley-value documentation](https://ema.drwhy.ai/shapley.html) and the DALEX [`predict_parts()` reference](https://modeloriented.github.io/DALEX/reference/predict_parts.html).
:::

```{r}
set.seed(42)

# YOUR CODE HERE
```

Which applicant characteristics push the prediction most strongly toward default and away from default?

**Your answer:** *Write your answer here.*



## Task 6.2 — Compare the local explanations


Compare the explanations produced for this applicant by logistic regression, the decision tree, and SHAP applied to the random forest. Logistic-regression contributions use the log-odds scale while SHAP contributions use the probability scale, so their numerical values are not directly comparable.

| Method | Top three drivers for this applicant |
|---|---|
| Logistic regression | |
| Decision tree | |
| SHAP | |

Do the three approaches emphasize broadly similar characteristics?

**Your answer:** *Write your answer here.*

Which explanation would be easiest to communicate to a nontechnical applicant?

**Your answer:** *Write your answer here.*

Does the SHAP explanation tell us what **caused** the applicant's predicted default risk?

**Your answer:** *Write your answer here.*



# Wrap-up — Match the explanation to the question {#wrap-up}


**Model-specific** methods use a model's internal structure; **model-agnostic** methods explain it through inputs and predictions.

Complete the comparison.

| Method | Global or local? | Model-specific? | Main question answered | Important limitation |
|---|---|---|---|---|
| Logistic coefficients | | | | |
| Decision tree | | | | |
| Permutation importance | | | | |
| Partial dependence | | | | |
| Individual profile | | | | |
| SHAP | | | | |

Finally, select the explanation you would prioritize for each stakeholder.

| Stakeholder | Most useful explanation(s) | Why? |
|---|---|---|
| Credit analyst reviewing one case | | |
| Applicant affected by a decision | | |
| Model-governance team | | |

Compare `applicant_outcome` with the three predictions from Task 1.1. What does this case suggest, and why can one case prove little about model quality?

**Your answer:** *Write your answer here.*

Optional discussion: How should regulation, contestability, and governance shape which explanations the organization records and communicates?

**Your answer:** *Write your answer here.*