You are a business analyst working for MetaCorp. The company wants to understand which factors are associated with employee performance. A central question in this notebook is whether remote work is associated with employee performance once we also account for other relevant variables.
MetaCorp is interested in questions such as:
Does remote work seem to be associated with higher or lower performance?
Does this relationship remain once we control for experience, training, salary, and management quality?
Is the relationship between remote work and performance similar across departments?
When should we be careful not to overinterpret regression results?
The target variable is: performance, a quantitative employee performance score
Potential predictors include:
experience: years of work experience
training_hours: annual training hours
salary: annual salary in EUR
remote_share: share of working time spent remotely (between 0 and 1)
team_size: size of the employee’s team
manager_quality: manager quality on a 1–5 scale
department: department (Sales, IT, HR)
Setup and dataset
Load the dataset
import pandas as pddf = pd.read_csv("data/employee_performance_data.csv")df.head()
employee_id
name
role
department
performance
experience
training_hours
salary
remote_share
team_size
manager_quality
0
E100000
Aleksandr Weihmann
Sales Representative
Sales
198.242841
18
45.381330
59096.739149
0.227314
11
5
1
E100001
Eleni Hauffer
HR Specialist
HR
144.281750
1
48.305080
34060.355685
0.419772
5
3
2
E100002
Paul Dobes-Stey
Software Engineer
IT
191.788132
16
29.344653
64625.846566
0.782596
9
4
3
E100003
Klemens Löchel
System Administrator
IT
190.445249
10
51.931709
38057.342688
0.763128
13
5
4
E100004
Alexandre Davids
Business Development Manager
Sales
173.408707
10
32.776373
45828.309067
0.499737
10
2
Variable overview
Variable
Meaning
Type
performance
employee performance score
target variable
experience
years of experience
numeric
training_hours
annual training hours
numeric
salary
annual salary in EUR
numeric
remote_share
share of remote work between 0 and 1
numeric
team_size
size of employee team
numeric
manager_quality
manager quality rating from 1 to 5
numeric
department
employee department
categorical
Quick inspection
df.describe(include="all")
employee_id
name
role
department
performance
experience
training_hours
salary
remote_share
team_size
manager_quality
count
1400
1400
1400
1400
1400.000000
1400.000000
1400.000000
1400.000000
1400.000000
1400.000000
1400.000000
unique
1400
1400
16
3
NaN
NaN
NaN
NaN
NaN
NaN
NaN
top
E100000
Aleksandr Weihmann
Sales Representative
Sales
NaN
NaN
NaN
NaN
NaN
NaN
NaN
freq
1
1
180
573
NaN
NaN
NaN
NaN
NaN
NaN
NaN
mean
NaN
NaN
NaN
NaN
160.967192
10.228571
40.086960
50479.856722
0.517162
8.485000
3.257857
std
NaN
NaN
NaN
NaN
23.252447
6.014805
9.891894
12912.846130
0.216851
3.473548
1.267032
min
NaN
NaN
NaN
NaN
85.270833
0.000000
7.605623
17394.427222
0.024597
3.000000
1.000000
25%
NaN
NaN
NaN
NaN
144.141334
5.000000
33.571152
40088.060513
0.350686
5.000000
2.000000
50%
NaN
NaN
NaN
NaN
161.435817
10.000000
39.876371
50937.579218
0.526263
9.000000
3.000000
75%
NaN
NaN
NaN
NaN
177.921722
16.000000
46.828952
60827.411076
0.686289
12.000000
4.000000
max
NaN
NaN
NaN
NaN
232.193662
20.000000
76.577018
82726.231450
0.994014
14.000000
5.000000
Part A — Baseline regression
We begin with one regression model for the full company. This gives us a first answer to the question:
How is remote_share associated with performance when we control for other variables at the same time?
This is useful because simple bivariate comparisons can be misleading. For example, employees with higher remote work shares may also differ in their department, training patterns, or salaries.
Task A1: Fit a regression model with scikit-learn
Use the full dataset to estimate a linear regression model for performance — use concepts from the lecture and consult the LinearRegression documentation if needed.
Your tasks:
One-hot encode nominal variables.
Define X and y.
Fit a LinearRegression() model.
Print the coefficients and intercept.
Pay special attention to the coefficient of remote_share.
from sklearn.linear_model import LinearRegression# Write your own code here.# Suggested objects: df_model, X, y, model, coef_df
Note. When initial versions raise errors, remember that regression models work with numerical predictors exclusively. Find a way to address this and run the model.
Task A2: Interpret the coefficients
Answer the following questions:
Which variables have a positive association with performance?
Which variables have a negative association with performance?
How would you interpret the coefficient of remote_share?
Is the remote_share effect large or small in practical terms?
How should dummy variables such as department_IT or department_Sales be interpreted?
NoteInterpretation reminder
A coefficient expresses the expected change in the dependent variable when the predictor increases by one unit, holding the other variables constant.
ImportantImportant note on remote_share
Because remote_share ranges from 0 to 1, a one-unit increase means moving from fully on-site to fully remote.
In practice, you may also want to interpret smaller changes, such as an increase of 0.10 or 10 percentage points.
Part B — Regression tables with statsmodels
In this notebook, we use statsmodels to complement scikit-learn—not for better predictions, but to gain a deeper understanding of how regression results are constructed, tested, and interpreted.
NoteWhy use statsmodels as an alternative library?
When working with regression models, we often want to interpret results in a structured, test-oriented way—for example:
testing whether coefficients are statistically significant
examining standard errors and confidence intervals
evaluating overall model fit (e.g., R², F-tests)
interpreting regression tables in a familiar format
While scikit-learn is excellent for fitting models and making predictions, it does not focus on detailed statistical inference or provide rich regression summaries.
This is where statsmodels becomes particularly useful.
statsmodels can be seen as an alternative library with a workflow similar to R, especially for those familiar with classical statistical modeling. It provides:
comprehensive regression output (tables with coefficients, standard errors, p-values, confidence intervals)
a strong focus on statistical testing and interpretation
a convenient formula-based interface (using strings like y ~ x1 + x2), which is widely used in R
The formula syntax is especially important to understand, as it allows you to specify models in a clear and expressive way.
Task B1: Estimate the same model with statsmodels
Use statsmodels to create a regression table for the same model.
import statsmodels.api as sm# Write your own code here.
Task B2: Interpret the regression table
In Part A, you estimated the same model using scikit-learn. Now, use the statsmodels output to deepen your understanding of regression results and modeling workflows.
Focus on both the results and the differences between the two approaches.
Questions:
Compare coefficients Are the coefficient estimates (e.g., for remote_share) similar to those from scikit-learn?
Understanding statistical significance Which variables are statistically significant based on the p-values? How does this additional information change (or not change) your interpretation from Part A?
Interpreting remote_share more formally What do the p-value and confidence interval suggest about the reliability of the remote_share effect?
Model fit and explanatory power What do R² and adjusted R² tell you about how well the model explains performance? Why should we be cautious when interpreting these values?
Comparing workflows: scikit-learn vs. statsmodels Reflect on how the model was specified in both parts:
How did you define variables and transformations in scikit-learn?
How does the formula interface in statsmodels simplify (or change) this process?
What are the advantages of using a formula like performance ~ remote_share + ...?
Part C — Regression diagnostics
Regression coefficients are only part of the story. Before relying on model predictions, it is important to assess whether the model provides a reasonable representation of the data.
A key assumption of linear regression is linearity: the model assumes that the relationship between the predictors and the outcome can be approximated by a straight line (conditional on the other variables). If this assumption is violated, the estimated coefficients may be misleading—even if they appear statistically significant.
To assess this, we focus on residual plots. Residuals capture what the model cannot explain. If the model is appropriate, these unexplained parts should look like random noise. Systematic patterns in the residuals, however, indicate that the model is missing important structure (e.g., non-linear relationships).
At the same time, remember that linearity is only one of several assumptions we discussed in the lecture.
This is why residual plots are one of the most important and intuitive diagnostic tools in regression analysis.
A key point is that even if remote_share has a positive or negative coefficient, that does not automatically mean that the model is perfectly specified or that the relationship should be interpreted causally.
Task C1: Residual plot
Create a simple residual plot for the fitted scikit-learn model from part A.
import matplotlib.pyplot as plt# residuals = y - model.predict(X_without_constant_or_matching_feature_matrix)# plt.scatter(model.predict(X_without_constant_or_matching_feature_matrix), residuals)# plt.axhline(0, linestyle="--")# plt.xlabel("Predicted performance")# plt.ylabel("Residuals")# plt.title("Residual plot")# plt.show()
Questions:
Do the residuals look roughly random?
Do you see patterns that may indicate model problems?
What would a clear funnel shape suggest?
Part D — Subgroup analysis: does one model fit all departments?
Sometimes, a single regression model for the whole company may hide important differences. The relationship between predictors and performance may not be the same for all employees. For example, internal feedback at MetaCorp suggests that complaints about remote work are more frequent in some departments than in others. In departments with high coordination needs, employees may report communication challenges or delays, while in others, remote work may support concentration and flexibility.
This suggests that the effect of remote_share may depend on the work context—even if our initial model treats all employees the same. Instead of assuming that one model fits everyone equally well, we can explore this idea by estimating separate models for different groups.
This helps us answer questions such as:
Do relationships look similar across departments?
Are some variables more important in certain contexts?
Does the effect of remote_share change depending on the work environment?
In this task, we take a first step toward understanding such differences by comparing models across departments.
Task D1: Department-specific regressions
Estimate separate regressions for Sales, IT, and HR.
# Write your own code here, using the following scaffold# import statsmodels.api as sm# for dept in df["department"].unique():# df_sub = ...# X_sub = ...# y_sub = ...# TODO: fit OLS# model_sub = ...# print(f"\n=== Regression results for {dept} ===")# print(model_sub.summary())
Task D2: Compare the subgroup models
Use the results to answer the following questions.
Do the coefficients differ across departments?
Is remote_share associated with performance in the same way in every department?
Does manager_quality seem equally important in all departments?
What does this tell us about heterogeneity in company data?
Why might a single model be too simplistic?
NoteKey idea
If relationships differ strongly between groups, a single regression model can hide important structure.
Part E — Deployment
Saving and loading a model is useful in practice, but it is not the main conceptual focus of this notebook.
Here, it is included as an optional extension after you have already worked through model estimation, interpretation, diagnostics, and subgroup analysis.
Task E1 (optional): Save a fitted model
import joblib# TODO: save the fitted sklearn model# joblib.dump(model, "data/employee_performance_model.joblib")
Task E2 (optional): Load the model and predict a new case
⚠️ Important note In this example, we only save the trained model, but not the preprocessing steps (e.g., one-hot encoding of the department variable).
The prediction works only because we manually created a new data point that exactly matches the structure of the training data (X). In real-world applications, you should always save the full preprocessing pipeline together with the model to ensure consistent predictions.
Task E3: Reflect on practical and organizational considerations
In practice, deploying a model like this involves more than just saving and loading it. Think about how such a model would be used in a real company like MetaCorp.
Questions:
What practical challenges might arise when using this model to support decisions about employees?
What risks could emerge if the model is used without careful interpretation or oversight?
What steps and safeguards would be important to ensure that the model is used fairly, transparently, and responsibly, taking into account legal and organizational requirements (e.g., employee representation, data protection)?
Wrap-up
🎉🎈 You have completed the notebook - good work! 🎈🎉
In this notebook, we have learned to
Fit and interpret linear regression models using scikit-learn
Read and understand regression tables with statsmodels
Interpret coefficients, including the effect of remote_share
Reflect on statistical significance, model fit, and practical relevance
Diagnose regression models using residual plots and assumptions
Compare models across departments
Save, load, and apply a trained model for prediction