flowchart
subgraph AFTER_BREAK["<b>After break: Filter and group data</b>"]
direction TB
B1["<b>Boolean conditions and row filtering</b> (20 min)"]
B2["<b>Combined conditions</b> (15 min)"]
B3["<b>π» Exercise: Conditional filtering</b> (20 min) Β· countries.csv, countries5.py"]
B4["<b>Grouping by one or more columns</b> (15 min)"]
B5["<b>π» Exercise: Group countries and insurance</b> (17 min) Β· countries.csv, countries6.py, insurance.xlsx, insurance1.py"]
B1 --> B2 --> B3 --> B4 --> B5
end
subgraph BEFORE_BREAK["<b>Before break: Summarize and visualize data</b>"]
direction TB
A0["<b>Solve homework</b> (10 min)"]
A1["<b>Sums, means, standard deviation, and correlation</b> (20 min)"]
A2["<b>Summaries, unique values, and frequencies</b> (15 min)"]
A3["<b>π» Exercise: Countries descriptive statistics</b> (15 min) Β· countries.csv, countries3.py"]
A4["<b>Line, bar, and pie charts</b> (30 min)"]
A5["<b>π» Exercise: Countries visualizations</b> (10 min) Β· countries.csv, countries4.py"]
A0 --> A1 --> A2 --> A3 --> A4 --> A5
end
Session 06/Python 3: Statistics, charts, and grouping
Focus on descriptive statistics, basic charts (line, bar), filtering, and grouping. Other materials are complementary.
NoteSession 4 datasets
The files underlying the data and exped DataFrames are available in Canvas in data.zip.
Homework
import pandas as pd
countries = pd.read_csv('data/countries.csv')
countries.loc[len(countries)] = ['Belgium',11.5,11.849,803.166,'Europe']
countries.sort_values(by='Country')
countries['GDP_per_capita'] = countries.GDP / countries.Population
countries.sort_values(by='GDP_per_capita', ascending=False)
countries = countries.drop('GDP_per_capita', axis=1)
countries = countries.drop(20)Slide 37 β Summing DataFrame values
The latter two expressions calculate the same sums using label-based and position-based selection.
import pandas as pd
exped = pd.read_excel("../materials/data/Exped.xlsx")
exped.loc[:, ['QUANTITY', 'REVENUE']].sum()
exped.iloc[:, 5:7].sum()QUANTITY 2959
REVENUE 483465
dtype: int64Slide 39 β Mean, standard deviation, and correlation
exped.loc[:, ['QUANTITY', 'REVENUE']].mean()
exped['REVENUE'].std()
exped.loc[:, ['QUANTITY', 'REVENUE']].corr()| QUANTITY | REVENUE | |
|---|---|---|
| QUANTITY | 1.00000 | 0.29915 |
| REVENUE | 0.29915 | 1.00000 |
Slide 40 β Statistical summaries
describe() is type-aware, so its summary depends on whether the selected data are numerical or categorical.
exped.describe()
exped['REVENUE'].describe()
exped['WHERE'].describe()count 1000
unique 5
top London
freq 411
Name: WHERE, dtype: objectSlide 41 β Unique values and frequencies
exped['WHERE'].unique()
exped['WHERE'].value_counts()WHERE
London 411
Paris 300
New York 103
Tokyo 102
Sydney 84
Name: count, dtype: int64Slide 42 β Descriptive statistics
Mean measures the average value of a dataset.
\[ \bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i \]
- \(x_i\): Individual observation
- \(n\): Number of observations
The mean is sensitive to outliers.
Variance measures how spread out values are around their mean. A higher variance indicates greater variability, while a variance of zero means all values are identical. It is calculated as:
\[ s^2 = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2 \]
where:
- \(x_i\): Individual observation
- \(\bar{x}\): Sample mean
- \(n\): Number of observations
- \(n-1\): Degrees of freedom (corrects for estimating the population mean from the sample)
Variance is expressed in squared units of the original data. The standard deviation (\(s = \sqrt{s^2}\)) converts it back to the original units.
import pandas as pd
countries = pd.read_csv('data/countries.csv')
countries['Area'].sum()
countries['GDP'].mean()
countries['GDP'].var()
countries.loc[:, ['Population', 'Area', 'GDP']].mean()
countries.loc[[2, 10, 11, 13], ['Population', 'Area', 'GDP']].mean()
countries['Area'].min()
countries.loc[:, ['Population', 'Area', 'GDP']].corr()
countries.describe() # or: countries[['Population', 'Area', 'GDP']].describe()
countries.loc[:, ['Country', 'Continent']].describe()
len(countries['Continent'].unique())
countries['Continent'].value_counts()Slide 43 β Creating a basic chart
With no explicit column selection, plot() includes all numerical columns by default.
In-class demo ../materials/data/temperatures.csv
import pandas as pd
temps = pd.read_csv("../materials/data/temperatures.csv")
temps.head()
temps.plot()
Slide 44 β Selecting variables for a chart
temps.iloc[:, 1:4].plot()
temps.plot(
x='Month',
y=['Maximum', 'Average', 'Minimum'],
)Slide 45 β Formatting the x-axis
temps.plot(
x='Month',
y=['Maximum', 'Average', 'Minimum'],
xticks=range(len(temps['Month'])),
rot=75,
)Slide 47 β Creating a bar chart
df = exped['WHERE'].value_counts()
print(df)
df.dtypes
df.plot(kind='bar')Slide 48 β Creating pie charts
df.plot(kind='pie', y='WHERE')
df.plot(kind='pie', subplots=True)
df.plot(kind='pie', subplots=True, legend=True)Slide 49 β Basic visualizations
import pandas as pd
countries = pd.read_csv('data/countries.csv')
countries.plot(x='Country', y=['Population', 'Area', 'GDP'],
xticks=range(len(countries['Country'])), rot=75)
df = countries['Continent'].value_counts()
df.plot(kind='bar')
df.plot(kind='pie')
countries.plot(kind='density', y='GDP')
countries.plot(kind='box', y='GDP')
ImportantExam relevance
Only line and bar plots are relevant to the exam; see the document listing exam-relevant topics.
Break β 10 minutes
Slide 50 β Creating a Boolean condition
The comparison produces a Boolean Series with one value per row.
exped.WHERE == 'London'Slide 51 β Selecting rows based on a condition
exped.loc[exped.WHERE == 'London']Slide 52 β Combining conditions
Use & for βandβ and | for βorβ. Each individual pandas condition needs parentheses when conditions are combined.
exped.loc[
(exped.WHERE == 'London')
& (exped.REVENUE >= 2500)
]
exped.loc[
(exped.QUANTITY >= 5)
| (exped.REVENUE >= 2500)
]Slide 53 β Conditional filtering
import pandas as pd
countries = pd.read_csv('data/countries.csv')
countries.loc[countries.Area > 8000]
countries.loc[countries.Area > 8000, 'GDP'].mean()
countries.loc[countries.Area < 800, 'GDP'].mean()
countries.loc[(countries.Area > 8000) & (countries.Population < 400)]
countries.loc[(countries.Area > 8000) & (countries.Population < 400), ['Country', 'GDP']]
df = countries.loc[(countries.Area > 8000) & (countries.Population < 400), ['Country', 'GDP']]
df.plot(x='Country', kind='bar', rot=0)
df = countries.loc[countries.Population < 100, 'Continent'].value_counts()
df.plot(kind='pie', y='Continent')
df = countries.loc[countries.Continent == 'Europe', 'GDP']
df.plot(kind='density')- Connect filtering conditions to the resulting summaries and visualizations.
Slide 54 β Grouping by one column
exped.groupby('HOW')['HOW'].count()
exped.groupby('HOW')['REVENUE'].mean()Slide 55 β Grouping by multiple columns
a = exped.groupby(['HOW', 'WHERE'])['REVENUE'].mean()
aSlide 56 β Grouping and aggregation
import pandas as pd
countries = pd.read_csv('data/countries.csv')
countries.groupby('Continent')['Continent'].count()
countries.groupby('Continent')['GDP'].mean()
countries['GDP_by_Pop'] = countries['GDP']/countries['Population']
countries.groupby('Continent')['GDP_by_Pop'].mean()import pandas as pd
insurance = pd.read_excel('data/insurance.xlsx')
insurance.groupby('State')['InsuredValue'].mean()
pd.options.display.float_format = '{:.2f}'.format
insurance.groupby('State')['InsuredValue'].mean()
insurance.groupby('State')['InsuredValue'].count()
insurance.loc[insurance['Flood'] == 'Y'].groupby('State')['InsuredValue'].count()
insurance.loc[(insurance['Flood'] == 'Y') | (insurance['Earthquake'] == 'Y')].groupby('State')['InsuredValue'].count()- Demonstrate grouping with both the country and insurance datasets.
NoteExam relevance
Machine learning and prediction are not relevant to the exam.
Summary and announcements
- TODO
TipNotes for improvement
Take notes on improvements and common questions during the session and add them to feedback.qmd afterwards.