Exercise 8: Big data and text analytics

Part 1: Case and dataset

Task 1.1 — Load and inspect the dataset

import pandas as pd

df = pd.read_csv("data/imdb_reviews_sample.csv")
df.head()

df["sentiment"].value_counts()

sentiment
negative    2532
positive    2468
Name: count, dtype: int64

Task 1.2 — Prepare text, labels, and a train-test split

Solution

from sklearn.model_selection import train_test_split

df = df.dropna(subset=["review", "sentiment"]).copy()

# Many IMDB review exports contain HTML line breaks such as <br />.
# This light cleaning keeps the task simple but removes a common artifact.
df["clean_review"] = (
    df["review"]
    .str.replace(r"<br\s*/?>", " ", regex=True)
    .str.replace(r"\s+", " ", regex=True)
    .str.strip()
)

df["label"] = df["sentiment"].map({"negative": 0, "positive": 1})

X = df["clean_review"]
y = df["label"]

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=42,
    stratify=y,
)

print(X_train.shape, X_test.shape)
print(y_train.value_counts(normalize=True))
print(y_test.value_counts(normalize=True))

(3750,) (1250,)
label
0    0.5064
1    0.4936
Name: proportion, dtype: float64
label
0    0.5064
1    0.4936
Name: proportion, dtype: float64

Part 2: Sparse text representations

Task 2.1 — Create Bag-of-Words and TF-IDF matrices

Solution

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

count_vectorizer = CountVectorizer(
    stop_words="english",
    max_features=5000,
)
X_train_bow = count_vectorizer.fit_transform(X_train)
X_test_bow = count_vectorizer.transform(X_test)

tfidf_vectorizer = TfidfVectorizer(
    stop_words="english",
    max_features=5000,
    min_df=2,
    ngram_range=(1, 2),
)
X_train_tfidf = tfidf_vectorizer.fit_transform(X_train)
X_test_tfidf = tfidf_vectorizer.transform(X_test)

sparsity = 1 - (X_train_tfidf.nnz / (X_train_tfidf.shape[0] * X_train_tfidf.shape[1]))

print("Bag-of-Words shape:", X_train_bow.shape)
print("TF-IDF shape:", X_train_tfidf.shape)
print("TF-IDF sparsity:", round(sparsity, 4))

Bag-of-Words shape: (3750, 5000)
TF-IDF shape: (3750, 5000)
TF-IDF sparsity: 0.9854

Expected interpretation:

• The number of rows corresponds to the number of training reviews.
• The number of columns corresponds to the vocabulary retained by the vectorizer.
• A high sparsity value means that most document-feature combinations are zero. This is typical for classical text representations.

Task 2.2 — Inspect sparse features for one review

Solution

import numpy as np

review_id = 0
feature_names = np.array(tfidf_vectorizer.get_feature_names_out())
row = X_train_tfidf[review_id].toarray().ravel()

top_positions = row.argsort()[::-1][:15]

top_features = pd.DataFrame({
    "feature": feature_names[top_positions],
    "tfidf": row[top_positions],
})

display(X_train.iloc[review_id])
display(top_features)

'Worst Movie I Have Ever Seen! 90 Minutes of excruciating film-making. All the ingredients to make this movie a true work of CRAP. Bad acting, bad directing, bad storytelling, bad makeup, bad dialogue, bad effects, and bad reasoning behind certain actions taken by the characters. It also threw in a terrible naked shot of a dumb blond, and a breast shot of a stupid Asian girl, and both attempts were just scary, since these girls are ugly. Some good horror movies came out of the 80s, but this could never be considered one of them. Kevin Tenney also committed one of the greatest sins in storytelling: he introduced characters at the end of the movie (an Old Man and Old Woman). I would vote for it below a 1 out of 10 but the voting system doesn\'t work that way apparently. Right from the title sequence I knew it would suck and I would return my DVD but Best Buy doesn\'t refund DVDs, or consumable products as they call it, or so my receipt says. I have "The Dunwich Horror" and that was truly god-awful, but I still feel that "Night of the Demons" (an obvious Evil Dead RIP-OFF) was far worse than "Dunwich Horror." This is just like "The Howling," how in the hell could sequels get milked out of this anorexic cow??? Save your money and get the "Texas Chainsaw Massacre" (just don\'t get any of its sequels though) or "The Evil Dead" or "Dawn of the Dead." "Night of the Demons" is a very, very, very bad investment. Every second of it was just maddening, excruciating pain for the audience, because the whole movie all-around was horrible! Do yourself a favor, DON\'T SEE IT! You\'ll be saving some brain cells.'

	feature	tfidf
0	bad	0.318879
1	evil dead	0.217983
2	storytelling	0.197931
3	demons	0.197931
4	dead	0.190483
5	sequels	0.188522
6	horror	0.168756
7	evil	0.132952
8	night	0.121025
9	shot	0.120767
10	just	0.120738
11	blond	0.110300
12	breast	0.106651
13	chainsaw	0.105596
14	ingredients	0.105596

A good answer should recognize that TF-IDF features are interpretable because they correspond to words or short phrases. However, they mainly show lexical signals. They do not fully capture context, irony, negation, or meaning beyond word occurrence.

Task 2.3 — Find similar reviews using TF-IDF

Solution

from sklearn.metrics.pairwise import cosine_similarity

query_id = 0
similarities = cosine_similarity(X_train_tfidf[query_id], X_train_tfidf).ravel()

top_ids = similarities.argsort()[::-1][1:6]

similar_reviews_tfidf = pd.DataFrame({
    "similarity": similarities[top_ids],
    "sentiment": y_train.iloc[top_ids].map({0: "negative", 1: "positive"}).values,
    "review": X_train.iloc[top_ids].values,
})

print("Query review:")
print(X_train.iloc[query_id])
print("Query sentiment:", "positive" if y_train.iloc[query_id] == 1 else "negative")

display(similar_reviews_tfidf)

Query review:
Worst Movie I Have Ever Seen! 90 Minutes of excruciating film-making. All the ingredients to make this movie a true work of CRAP. Bad acting, bad directing, bad storytelling, bad makeup, bad dialogue, bad effects, and bad reasoning behind certain actions taken by the characters. It also threw in a terrible naked shot of a dumb blond, and a breast shot of a stupid Asian girl, and both attempts were just scary, since these girls are ugly. Some good horror movies came out of the 80s, but this could never be considered one of them. Kevin Tenney also committed one of the greatest sins in storytelling: he introduced characters at the end of the movie (an Old Man and Old Woman). I would vote for it below a 1 out of 10 but the voting system doesn't work that way apparently. Right from the title sequence I knew it would suck and I would return my DVD but Best Buy doesn't refund DVDs, or consumable products as they call it, or so my receipt says. I have "The Dunwich Horror" and that was truly god-awful, but I still feel that "Night of the Demons" (an obvious Evil Dead RIP-OFF) was far worse than "Dunwich Horror." This is just like "The Howling," how in the hell could sequels get milked out of this anorexic cow??? Save your money and get the "Texas Chainsaw Massacre" (just don't get any of its sequels though) or "The Evil Dead" or "Dawn of the Dead." "Night of the Demons" is a very, very, very bad investment. Every second of it was just maddening, excruciating pain for the audience, because the whole movie all-around was horrible! Do yourself a favor, DON'T SEE IT! You'll be saving some brain cells.
Query sentiment: negative

	similarity	sentiment	review
0	0.262307	positive	The 80's is largely considered the decade in w...
1	0.211370	negative	Why is it that Canada can turn out decent to g...
2	0.207395	negative	Avoid this crap at all costs. Bad script, bad ...
3	0.202169	positive	most of the bad reviews on this website blame ...
4	0.202030	negative	This movie has to be one of the most boring an...

Expected interpretation:

• TF-IDF similarity is often driven by shared words and phrases.
• Retrieved reviews may be similar in topic or wording, but not necessarily in deeper meaning.
• Sparse similarity can be useful, but it is limited when two texts express similar ideas with different words.

Part 3: Dense representations

Task 3.1 — Create sentence embeddings

Solution

from sentence_transformers import SentenceTransformer

embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

X_train_emb = embedding_model.encode(
    X_train.tolist(),
    show_progress_bar=True,
    normalize_embeddings=True,
)
X_test_emb = embedding_model.encode(
    X_test.tolist(),
    show_progress_bar=True,
    normalize_embeddings=True,
)

print("Training embeddings:", X_train_emb.shape)
print("Test embeddings:", X_test_emb.shape)

Expected interpretation:

• With all-MiniLM-L6-v2, each review is represented by a 384-dimensional vector.
• The representation is dense because most values are non-zero.
• The dimensions are learned features rather than directly named words, so they are less transparent than TF-IDF columns.

Task 3.2 — Find similar reviews using dense embeddings

Solution

from sklearn.metrics.pairwise import cosine_similarity

query_id = 0
similarities_dense = cosine_similarity(
    X_train_emb[query_id].reshape(1, -1),
    X_train_emb,
).ravel()

top_ids_dense = similarities_dense.argsort()[::-1][1:6]

similar_reviews_dense = pd.DataFrame({
    "similarity": similarities_dense[top_ids_dense],
    "sentiment": y_train.iloc[top_ids_dense].map({0: "negative", 1: "positive"}).values,
    "review": X_train.iloc[top_ids_dense].values,
})

print("Query review:")
print(X_train.iloc[query_id])
print("Query sentiment:", "positive" if y_train.iloc[query_id] == 1 else "negative")

display(similar_reviews_dense)

Expected interpretation:

Dense embeddings often retrieve reviews that are similar in overall meaning even if they do not share many exact words. However, dense similarity is not identical to sentiment similarity. Two reviews can be semantically close because they discuss similar topics, genres, actors, or plot elements, while still expressing different evaluations.

Task 3.3 — Visualize dense embeddings

Solution

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

sample_size = min(500, len(df))
sample_df = df.sample(n=sample_size, random_state=42).copy()

sample_embeddings = embedding_model.encode(
    sample_df["review"].tolist(),
    show_progress_bar=True,
    normalize_embeddings=True,
)

pca = PCA(n_components=2, random_state=42)
coords = pca.fit_transform(sample_embeddings)

sample_df["dim_1"] = coords[:, 0]
sample_df["dim_2"] = coords[:, 1]

plt.figure(figsize=(8, 6))
plt.scatter(
    sample_df["dim_1"],
    sample_df["dim_2"],
    c=sample_df["label"],
    alpha=0.65,
)
plt.xlabel("PCA dimension 1")
plt.ylabel("PCA dimension 2")
plt.title("Dense review embeddings reduced to two dimensions")
plt.show()

Expected interpretation:

• There may be some separation, but do not expect a perfect positive/negative split.
• The plot may reveal topic-based or style-based neighborhoods as well as sentiment-related patterns.
• PCA compresses hundreds of dimensions into two dimensions, so the plot is only a simplified view of the embedding space.

Part 4: Classification

Task 4.1 — Train a classifier on TF-IDF features

Solution

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
import time

start_time = time.time()

tfidf_clf = LogisticRegression(max_iter=1000)
tfidf_clf.fit(X_train_tfidf, y_train)

y_pred_tfidf = tfidf_clf.predict(X_test_tfidf)

tfidf_train_time = time.time() - start_time

tfidf_accuracy = accuracy_score(y_test, y_pred_tfidf)
print("Accuracy:", round(tfidf_accuracy, 3))
print("Training time:", round(tfidf_train_time, 3), "seconds")
print(classification_report(y_test, y_pred_tfidf, target_names=["negative", "positive"]))
print(confusion_matrix(y_test, y_pred_tfidf))

A good interpretation should note that TF-IDF combined with logistic regression is often a strong baseline for sentiment classification. It is relatively interpretable because features correspond to words or phrases, and coefficients can be inspected.

Task 4.2 — Train a classifier on dense embeddings

Solution

start_time = time.time()

dense_clf = LogisticRegression(max_iter=1000)
dense_clf.fit(X_train_emb, y_train)

y_pred_dense = dense_clf.predict(X_test_emb)

dense_train_time = time.time() - start_time

dense_accuracy = accuracy_score(y_test, y_pred_dense)
print("Accuracy:", round(dense_accuracy, 3))
print("Training time:", round(dense_train_time, 3), "seconds")
print(classification_report(y_test, y_pred_dense, target_names=["negative", "positive"]))
print(confusion_matrix(y_test, y_pred_dense))

Expected interpretation:

The dense model may perform better, worse, or similarly depending on the dataset size, model, and preprocessing. Generic pretrained embeddings are useful because they encode semantic information, but TF-IDF can be very competitive for supervised sentiment classification. The important learning outcome is not that dense vectors always win, but that they represent texts differently.

Task 4.3 — Compare sparse and dense models

Solution

results = pd.DataFrame(
    {
        "representation": ["TF-IDF", "Dense embeddings"],
        "accuracy": [tfidf_accuracy, dense_accuracy],
        "training_time_seconds": [tfidf_train_time, dense_train_time],
    }
)

results

A possible interpretation:

Criterion	TF-IDF + logistic regression	Dense embeddings + logistic regression
Accuracy	Often strong for sentiment	Often strong, but not automatically better
Training time	Usually fast	Embedding creation may be slower
Feature interpretability	High	Lower
Captures exact words	Yes	Less directly
Captures semantic similarity	Limited	Stronger

The comparison should emphasize trade-offs rather than a single winner.

Part 5: Edge cases and reflection

Solution

edge_cases = pd.DataFrame(
    {
        "review": [
            "This movie was sick in the best possible way.",
            "This movie made me sick.",
            "Not bad at all.",
            "A masterpiece? Not really.",
            "The acting was fine, but the story went nowhere.",
            "I expected more.",
            "The plot was unpredictable and surprisingly moving.",
            "I would not say it is a bad movie.",
        ],
        "expected_sentiment": [
            "positive",
            "negative",
            "positive",
            "negative",
            "negative",
            "negative",
            "positive",
            "positive_or_ambiguous",
        ],
    }
)

label_map = {0: "negative", 1: "positive"}

edge_tfidf = tfidf_vectorizer.transform(edge_cases["review"])
edge_cases["tfidf_prediction"] = pd.Series(tfidf_clf.predict(edge_tfidf)).map(label_map)

edge_emb = embedding_model.encode(
    edge_cases["review"].tolist(),
    normalize_embeddings=True,
)
edge_cases["dense_prediction"] = pd.Series(dense_clf.predict(edge_emb)).map(label_map)

edge_cases

Expected interpretation:

• Negation, irony, slang, and understatement can be difficult.
• TF-IDF may rely heavily on visible sentiment words such as bad, sick, or masterpiece.
• Dense embeddings may better capture some paraphrases, but they can also fail when the sentence is short, ironic, or context-dependent.
• The synthetic examples are useful for qualitative failure analysis, not for measuring general performance.