Exercise 9: Big data and neural networks

Part 1: Backpropagation step by step

Solution

See http://hmkcode.github.io/ai/backpropagation-step-by-step.

Part 2: Interactive neural networks

Solution

There is not one single correct solution for this task. The TensorFlow Playground uses random initialization, and the final test loss can vary slightly across runs. The goal is therefore not to reproduce one exact value, but to find a model that is as simple as possible while still capturing the structure of the dataset.

Dataset	Input features	Hidden layers	Neurons per layer	Activation function	Example final test loss
Two clusters	\(x_1\), \(x_2\)	0	none	Linear output only	0.00
Circle	\(x_1\), \(x_2\), \(x_1^2\), \(x_2^2\)	1	3–4	Tanh or ReLU	0.00
Spiral	\(x_1\), \(x_2\)	3–4	6–8 each	ReLU	approx. 0.03–0.10

For the two-clusters dataset, the classes are almost linearly separable. This means that the model does not need hidden layers. A simple linear classifier using \(x_1\) and \(x_2\) can already draw an appropriate decision boundary. Adding more layers or many neurons may still work, but it is unnecessary and makes the model more complex than needed.

For the circle dataset, a small non-linear model already works well. The pattern cannot be separated by a straight line, so a model without hidden layers is usually not sufficient when only \(x_1\) and \(x_2\) are used. Adding squared input features such as \(x_1^2\) and \(x_2^2\) makes the circular structure easier to learn. With these features, one hidden layer with only a few neurons is often enough.

For the spiral dataset, the decision boundary is much more complex. The classes are intertwined in a curved, non-linear pattern, so a simple linear model or a very small hidden layer cannot capture the structure well. Several hidden layers with multiple neurons are usually needed (e.g., all features, one layer with 7 neurons, one layer with 5 neurons). The ReLU activation function works particularly well because it allowed the network to combine several local linear boundaries into a more flexible non-linear decision surface.

Overall, the experiments show that the required network architecture depends strongly on the complexity of the dataset. Simple, linearly separable patterns can be learned with very small models, while strongly non-linear patterns require additional hidden layers, more neurons, and suitable activation functions. The aim is not to maximize model size, but to choose the smallest architecture that captures the relevant pattern with low test loss.

Part 3: Code understanding — PyTorch model objects

Solution

Task 1:

Input: 10 features
Linear: 10 -> 8
ReLU
Linear: 8 -> 4
ReLU
Linear: 4 -> 1
Output: 1 value

Part 4: Use Cases Quiz — NovaStream

Solution

#	Summary Statement	Likely Suitable Approaches
1	Predict customers likely to cancel soon.	Logistic Regression, Random Forest, Neural Networks
2	Recommend personalized media content to users.	Recommendation Systems, Neural Networks, Transformers
3	Generate multilingual subtitles from spoken content.	Transformers, RNN/LSTM
4	Detect suspicious or fraudulent account behavior.	Anomaly Detection, Neural Networks
5	Optimize subscription prices and discounts dynamically.	Reinforcement Learning, Dynamic Pricing Models, Optimization Approaches
6	Classify uploaded images into content categories.	CNN
7	Optimize same-day delivery routes under constraints.	Vehicle Routing / Operations Research / Optimization Algorithms
8	Predict the next item a user will consume.	RNN/LSTM, Time series, Transformer
9	Identify media items frequently consumed together.	k-Means, Association Rule Mining
10	Detect toxic or abusive chat messages in real time.	Transformer-based NLP models, Neural Networks
11	Segment customers into behavior-based groups.	Clustering, e.g., K-Means
12	Forecast future server and bandwidth demand.	Linear Regression, Time-Series Models, RNN/LSTM