Notes S-04/Python 1: Python and pandas basics

NotePreparation TODOs

• TODO: Add the Spyder setup.

• TODO: Use Anaconda and be prepared for questions.

• TODO: Walk through spytertest.py from Canvas with students at the start of the Python session. Retain the original filename spelling; its repository path is not known.

• TODO: Clarify the note “use os.chdir() for Spyder.”

• TODO: Check whether pandas 3.0 is installed automatically.

• TODO: check with PR: no explicit take-home/homework exercises in this session?

The existing notes also identify these materials, but do not establish an exact page mapping:

• Versicherungsvertreter.py
• Versicherungsvertreter.docx

TODO: Confirm where these materials should be used. The original note referred to the second file as Versicherungsvertreter.xslx; the supplied inventory contains the Word document above instead.

Notes:

• Rule: TBD?? in-class demo: directly hard-code solution; in-class exercise: include script !??
• Note: solutions (of take-home exercises) are discussed by the tutors

Prepare the Python environment before the session and plan to complete as many exercises as time permits.

Slide 1 - Introductory discussion

• Ask about prior programming experience.
• Discuss why Python complements Excel for advanced analytics, regression, prediction, machine learning, and AI.
• Contrast interpreted and compiled languages.
• Motivate Python through reusable libraries and the “don’t repeat yourself” principle. Don’t repeat yourself principle of programming: by using Python libraries (modularization of code), we benefit from the code written by others. Illustration (whiteboard): the Python code that is involved in a typical ML studyis roughly 2 million LOC. 99.99% is imported (Pandas, scikitlearn, …) and a few hundred LOC are our code (tip of the iceberg, we orchestrate existing libraries). For the sake of the argument, we simply ignore all of the other code, the Python core or the code that runs the OS and operates the hardware. It is not just LOC, but the 99% of the code (the libraries) are often very well designed and maintained (if you pick the right libraries). Also, they are constantly updated so we get performance improvements, remaining bugs are fixed, and the documentation gets better. This means we can really “stand on the shoulders of giants”. Pretty amazing.
• On the whiteboard, illustrate an ML study as a small amount of course code built on a much larger body of imported, maintained library code. Emphasize that students orchestrate existing libraries and can “stand on the shoulders of giants.”
• Ask why students should not simply use ChatGPT: effective prompting still requires validation and improvement.

Slides 2-4 - Shell, Scripts, IDE

TODO: include Spyder setup here.

Slide 5 — Basic arithmetic and string operations

Run these expressions individually in Spyder so that each result is visible.

3*5
16-5
16/4
16/5
16%5              # modulo/remainder after integer division (1)
16//5             # floor division: returns whole-number qotient (3)
3**2              # exponent
"Hello " + "World"

Note: operators explained on the following slide.

Slide 8 — Variables and reassignment

Refer to a variable as names bound to an object

Python allows a variable to be reassigned to values of different types (dynamic typing), while multiple assignment sets several variables in one statement.

x = 4
x

x = 3.12
x

x = 'hello'
x

a, b = 1, 2
a
b
a + b

Slide 9 - Variable memory and CPU

Explain step by step how Python approaches this statement.

• Assignment: right side to the variable z.
• Right side is a complex statement. Needs to be evaluated first….

Slide 10 — Output with print()

These examples contrast comma-separated output and string concatenation, demonstrate repetition, use str() for conversion, and use end to control the line ending.

print('hello')
print('hello', 'there')
print('hello' + ' ' + 'there')
print('haha' * 4)
print(5 * 4)

a = 54
print(a)
print('The number is ' + str(a) + '!')
print('The number is', a, '!')

print('hello ')
print('Peter')
print('hello', end=' ')
print('Peter')

Slide 11 — Predicting program output

Predict the result of each block before running it.

x = 15
x = x + 5
print(x)

20

Note: for the x = x + 5, the following slide provides an animated explanation.

x = 4
y = x + 1
x = 2
print(x, y)

2 5

a = 4.5
b = 2
print(a // b)

2.0

x = 17 / 2 % 2 * 3**3
print(x)

13.5

x, y = 2, 6
x, y = y, x + 2
print(x, y)

6 4

a, b = 2, 3
c, b = a, c + 1
print(a, b, c)

The final block intentionally raises a NameError because c is read before it has been assigned.

Slide 13 — Basic formula programs

Depending on available time, complete this exercise in class or assign it as homework.

In-class exercise

radius = 20
area = 3.14159 * radius**2
print ('The area of the circle with radius', radius, 'is:', area)

In-class exercise

investment = 1000
interest_rate = 0.06
investment_period = 5
future_value = investment * (1 + interest_rate)**investment_period
print("The future value is:", future_value)

☕ Break — 10 minutes

Slide 15 — Importing pandas

pandas DataFrames are widely used to load, inspect, clean, and prepare tabular data for machine learning. Models commonly receive DataFrames (or similar formats) depending on the library.

The short alias pd is the conventional name used for pandas (programmers are often lazy and prefer short names).

import pandas as pd

Slide 16 — Reading an Excel file into a DataFrame

TODO: get and link Bikesales.xlsx file

The final expression displays the DataFrame in an interactive environment.

data = pd.read_excel('Bikesales.xlsx')
data

Slide 17 — Inspecting a DataFrame

These expressions show the DataFrame dimensions, column names, and column data types.

data.shape
list(data)
data.dtypes

Slide 18 — Working directory and file paths

This path is machine-specific. The raw-string prefix r prevents Windows backslashes from being interpreted as escape sequences.

TODO: select a suitable path here:

import os
os.chdir(r'D:\Dropbox\Lehre\Introduction to Programming\Presentation')

Note:

import os

print(os.getcwd())

TODO: where is it in spyder?

Slide 19 — Reading CSV files

The relative path is resolved against the active working directory.

data = pd.read_csv('Bikesales.csv')

Slide 20 — Loading, previewing, and exporting data

../materials/Exped_Table.xlsx

head() previews the first rows; index=False avoids writing the DataFrame index as an extra CSV column.

os.chdir(r'D:\Dropbox\Lehre\Introduction to Programming\Presentation')
exped = pd.read_excel('Exped.xlsx')
exped.head()

exped.to_csv('Exped.csv', index=False)

Slide 21 — Setting a DataFrame index

TODO: think: should Transid be an incrementing index? in this case, it does not differ from the default/artificial index… It may be a more reasonable example if the new index differs from the one created by default.

After set_index(), Transid supplies the row labels.

exped1 = exped
exped1 = exped1.set_index('Transid')
exped1.head()

Slide 22 — Importing and exporting data

In-class exercise ../materials/data/countries.csv

import pandas as pd

countries = pd.read_csv('../materials/data/countries.csv')
print(countries.shape)
print(countries.columns) # or print(list(countries))
countries.head()

(20, 5)
Index(['Country', 'Population', 'Area', 'GDP', 'Continent'], dtype='str')

	Country	Population	Area	GDP	Continent
0	China	1398.72	9596.96	12234.78	Asia
1	India	1351.16	3287.26	2575.67	Asia
2	US	329.74	9833.52	19485.39	N.America
3	Indonesia	268.07	1910.93	1015.54	Asia
4	Brazil	210.32	8515.77	2055.51	S.America

In-class exercise ../materials/data/insurance.xlsx

import pandas as pd

insurance = pd.read_excel('../materials/data/insurance.xlsx')
print(insurance.shape)
print(insurance.head())
print(list(insurance))
# insurance.to_csv('insurance.csv', index=False)

(500, 10)
   Policy     Expiry Location State   Region  InsuredValue Construction  \
0  100242 2021-01-02    Urban    NY     East       1617630        Frame   
1  100314 2021-01-02    Urban    NY     East       8678500  Fire Resist   
2  100359 2021-01-02    Rural    WI  Midwest       2052660        Frame   
3  100315 2021-01-03    Urban    NY     East      17580000        Frame   
4  100385 2021-01-03    Urban    NY     East       1925000      Masonry   

  BusinessType Earthquake Flood  
0       Retail          N     N  
1    Apartment          Y     Y  
2      Farming          N     N  
3    Apartment          Y     Y  
4  Hospitality          N     N  
['Policy', 'Expiry', 'Location', 'State', 'Region', 'InsuredValue', 'Construction', 'BusinessType', 'Earthquake', 'Flood']

• Complete all exercises as time permits.
• If time is short, assign unfinished exercises as homework.
• Python solutions will be discussed by the tutors.

Summary and announcements

TipNotes for improvement

Take notes on improvements and common questions during the session and add them to feedback.qmd afterwards.