Analytics & Big Data

Session 8: Big data 1

Prof. Dr. Gerit Wagner

(2026-05-04)

Explain the characteristics of big data and their implications for analytics workflows.
Compare data warehouse, data lake, and logical data warehouse architectures.
Describe the text analytics pipeline from preprocessing to representation and modeling.

Big data

4V’s of big data

Big data matters when the properties of the data change what managers can decide, automate, or compete on.

Volume: UPS route optimization

Managerial dilemma

How do you optimize millions of delivery decisions when every route has thousands of possible variants?

Key data facts

250M+ address data points
200,000+ route options per typical route
Parcels, drivers, fleet, road network, customer constraints
Real-time updates from operations and telematics

Models: vehicle routing · scheduling · ETA prediction · dynamic re-optimization
Outcomes: fewer miles · lower fuel costs · better on-time delivery · operational data advantage

UPS delivery truck or route optimization image

Rationale and key points to cover:

This slide illustrates volume as operational complexity. The point is not merely that UPS has “a lot of data.” The deeper point is that parcel logistics creates an enormous number of possible decisions: which parcels go into which vehicle, in which order, driven by which driver, through which road network, under which delivery and pickup constraints.

Emphasize that a route is not simply a map from A to B. It includes today’s parcels, promised delivery windows, pickup commitments, driver constraints, fleet capacity, customer requirements, historical travel times, current network conditions, road restrictions, and operational updates during the day.

The analytical problem is therefore combinatorial. A human dispatcher cannot compare hundreds of thousands of possible stop sequences. The firm needs optimization models, routing heuristics, scheduling systems, and ETA prediction. More advanced versions also support dynamic re-optimization when traffic, delays, or operational disruptions occur.

The competitive implication is that small improvements per route become enormous at system scale. Saving a few minutes or a few kilometers per driver per day can translate into major annual savings in fuel, labor, emissions, and service reliability. This creates an operational data advantage: firms with richer route histories, denser delivery networks, and better optimization infrastructure can operate at lower cost and higher reliability.

References for data and claims:

UPS ORION materials describe large-scale route optimization using more than 250 million address data points and evaluating more than 200,000 route options for a typical route.
UPS has reported ORION-related savings on the order of 100 million miles and around 10 million gallons of fuel per year.
Useful source search terms: “UPS ORION 250 million address data points”, “UPS ORION 200,000 route options”, “UPS ORION 100 million miles 10 million gallons”.

Velocity: Visa fraud detection

Managerial dilemma

Can fraud be detected before the transaction is approved?

Key data facts

Up to 83,000 transactions / second
≈ 7.17B decisions / day at peak capacity
500+ data points per transaction
Fraud is rare, but losses are massive
Visa reported $40B+ fraud blocked in 2023

Models: real-time risk scoring · anomaly detection · streaming ML · network analysis
Outcomes: approve · decline · challenge · escalate — all within milliseconds

Digital payment or fraud detection image

Rationale and key points to cover:

This slide illustrates velocity as the need to decide while the business event is still happening. In payment fraud detection, the decision must be made before the customer leaves the checkout page or terminal. The system cannot wait for an overnight batch report.

The key teaching point is the millisecond decision window. The model must ingest transaction data, compare it against historical patterns, evaluate location, merchant, amount, device, account history, and behavioral signals, and then decide whether the transaction should be approved, declined, challenged, or escalated.

This is also a useful example of rare-event analytics. Fraudulent transactions may be a tiny share of all transactions, but at Visa scale even a very small percentage can imply very large losses. False positives also matter: blocking genuine customers damages trust, merchant revenue, and user experience. The analytical challenge is therefore not simply “detect fraud”; it is “detect fraud while minimizing false declines, in milliseconds, at massive scale.”

The models typically include supervised machine learning, anomaly detection, streaming analytics, graph and network analysis, and real-time decision rules. The models are embedded directly in the transaction infrastructure.

The competitive implication is trust. Payment networks compete partly on reliability, fraud protection, authorization quality, and customer experience. Better fraud analytics can reduce losses, increase legitimate approval rates, and make merchants and banks more willing to rely on the network.

References for data and claims:

Visa reports infrastructure capacity of up to 83,000 transactions per second.
83,000 × 86,400 seconds per day = 7,171,200,000 theoretical peak transaction decisions per day.
Visa describes using 500+ data points in fraud detection and reported blocking over $40 billion in fraud in 2023.
ECB/EBA payment fraud reports can be used to illustrate that fraud rates are small as a percentage of transaction value but still correspond to billions in absolute losses.
Useful source search terms: “Visa 83,000 transactions per second”, “Visa 500 data points fraud”, “Visa blocked 40 billion fraud 2023”, “ECB EBA payment fraud 2024 0.002 percent”.

Variety: Walmart ambient IoT supply chain

Managerial dilemma

How do you coordinate inventory when the relevant data comes from many different worlds?

Key data facts

Target: 90M pallets by end of 2026
Active in 500 Walmart locations
Potential scale: 4,600 stores + 40+ distribution centers
Sensor data: location, movement, temperature, humidity, dwell time
Combined with inventory, supplier, store, and replenishment data

Models: sensor fusion · demand forecasting · replenishment optimization · exception detection
Outcomes: fewer stockouts · fresher products · less waste · tighter supplier coordination

Retail supply chain or IoT pallet tracking image

Rationale and key points to cover:

This slide illustrates variety as the integration of heterogeneous data worlds. Walmart’s supply-chain example is useful because the relevant data are not all in one system. They come from physical pallets, ambient IoT sensors, store inventory systems, supplier systems, distribution centers, transport flows, and AI-based replenishment systems.

The important teaching point is that the same business object—the pallet—becomes a digital object. It can produce information about where it is, how long it has been there, whether it was exposed to temperature or humidity deviations, and whether it is moving through the supply chain as expected.

This is especially important for food and grocery supply chains, where freshness, waste, stockouts, and timing matter. Data variety allows Walmart to connect physical product flow with digital planning and replenishment decisions.

Typical analytical models include sensor fusion, demand forecasting, replenishment optimization, dwell-time analytics, exception detection, and cold-chain monitoring. More advanced versions can be described as supply-chain control towers or digital twins.

The competitive implication is coordination. Walmart can use these data not only internally but also to structure supplier behavior. The firm gains better visibility over inbound flows, supplier performance, inventory accuracy, and store-level availability. This can reduce waste, improve freshness, reduce stockouts, and strengthen Walmart’s cost and availability advantage.

References for data and claims:

Wiliot announced collaboration with Walmart to scale ambient IoT tracking toward 90 million pallets by the end of 2026.
Wiliot/Walmart materials mention deployment across hundreds of Walmart locations and potential expansion across Walmart’s store and distribution-center network.
Wiliot describes ambient IoT sensors collecting location and condition data such as temperature and humidity.
Useful source search terms: “Wiliot Walmart 90 million pallets 2026”, “Walmart ambient IoT pallets 500 locations”, “Wiliot Walmart temperature humidity pallets”.

Veracity: Digital advertising fraud

Managerial dilemma

The dashboard says there were clicks and impressions — but were they real?

Key data facts

Fraudulent impressions, clicks, conversions, or data events
DoubleVerify measures 8.3T+ media transactions / year
Ad fraud losses projected at $172B by 2028
One fraud operation fell from 2.5B to 100M daily bid requests after mitigation

Models: bot detection · invalid traffic scoring · anomaly detection · graph analysis
Outcomes: block fake traffic · clean attribution · reallocate spend · protect ROI

Digital advertising fraud or bot traffic image

Rationale and key points to cover:

This slide illustrates veracity: whether the data can be trusted. Digital advertising is a strong example because the data may look extremely precise. A dashboard can report impressions, clicks, conversions, engagement rates, and cost per acquisition. But the central managerial question is whether these events reflect real human attention and real customer intent.

This is a better teaching example than a simple sensor failure because the ambiguity is managerial and economic. A click may exist in the data but still be generated by a bot. An impression may be counted but never actually seen by a human. A conversion may be attributed to a campaign but be manipulated by fraud or poor measurement.

The analytical models include bot detection, invalid-traffic classification, anomaly detection, device and behavioral fingerprinting, graph analysis, publisher-quality scoring, and campaign-level traffic-quality models.

The decision consequences are significant. Advertisers may block traffic, exclude publishers, shift budgets, reprice impressions, revise attribution models, or stop campaigns. Without veracity, the organization may optimize toward fake engagement. This is a powerful teaching point: bad data can make analytics actively harmful because the model learns from signals that do not correspond to real business value.

The competitive implication is marketing efficiency and trust. Firms with better traffic-quality analytics can spend advertising budgets more effectively, avoid waste, protect brands, and build more reliable attribution systems.

References for data and claims:

IAB defines ad fraud and invalid traffic as fraudulent representation of impressions, clicks, conversions, or other data events.
DoubleVerify reports measuring over 8.3 trillion media transactions annually.
DoubleVerify cites Juniper estimates that global ad fraud losses could reach $172 billion by 2028.
HUMAN Security reported mitigation of a fraud operation that reduced activity from 2.5 billion daily bid requests to 100 million daily bid requests.
Useful source search terms: “IAB ad fraud invalid traffic impressions clicks conversions”, “DoubleVerify 8.3 trillion media transactions”, “Juniper ad fraud 172 billion 2028”, “HUMAN 2.5 billion daily bid requests 100 million”.

Turning big data into value

Architecture

Limitations of the data warehouse approach

Traditional DWH solutions are designed to provide a single point of truth. Important aspects are:

Merge and unify data from multiple data sources
High data quality
Proper historization of the data
Data Governance and Compliance

This results in the following problem areas in today’s world:

Lack of flexibility due to high effort for changes
Time expenditure due to transfer from the operational sources and the aggregations
Past orientation of data (snapshot of the past); there is an increasing need for ad hoc and real-time analyses
Partial knowledge, as only structured data is stored, with increasing need for social media data, etc.
Patchwork: due to gradual introduction of data warehouses, many isolated solutions exist and are operated separately both technically and methodologically

The four layers of big data (I)

Data Source Layer

This is where the data arrives at the organization. It includes everything from sales records, customer database, feedback, social media channels, marketing list, email archives etc.

Identify and prioritize data sources

The four layers of big data (II)

The four layers of big data (III)

The four layers of big data (IV)

From data warehouse to data lake

Instead of recording millions of transactions, today’s organizations are recording billions of interactions. Companies are capturing more and more data that can open business opportunities and unlock new sources of value for organizations.

Companies are not able to store this data in data warehouses because it is of high volume, mostly raw and often not structured. As consequence, data lakes have emerged as an alternative approach. The intent is to capture enterprise data and load it in its raw form into a centralized, large, and inexpensive storage system.

In shifting from data warehouses to data lakes, it became important to decouple data movement from data transformation. Data movement (the “E” and “L” of ETL) is an operational task. Data transformation (the “T” of ETL) is a content-based, analytic-facing task that requires an understanding both of the data and how it’s to be used.

A clean separation between data movement and data transformation has the benefits of less friction because the instance loading the data isn’t responsible for transforming it.

The data lake

A data lake is a method of storing data within a system in its natural format, that facilitates the collocation of data in various schemata and structural forms. The idea of data lake is to have a single store of all data in the enterprise ranging from raw data to transformed data which is used for various tasks including reporting, visualization, analytics and machine learning.

Modern data analytics architecture

Logical data warehouse

A “logical data warehouse” provides analytical company data without first physically moving it to a physical data warehouse.

As in a classic data warehouse, uniform views are provided for analysis purposes.

While the data in the classic data warehouse comes from a “well-defined” physically uniform database, the “logical data warehouse” pulls data together from the data lake at the time of the query.

Aggregation is done just in time. Thus, the schema of the data warehouse is just virtual.

Text analytics

The vast majority of global data is stored in unstructured formats. For example:

emails and internal communication
customer support tickets
meeting notes and reports
product reviews and social media posts
annual reports and corporate filings
contracts and legal documents
job descriptions and résumés
scientific and market research reports
CRM notes and sales reports
helpdesk and incident reports

Text analytics combines methods from natural language processing, machine learning, information retrieval, and statistics to systematically analyze large volumes of textual data and derive actionable insights.

Application areas

Text preprocessing

Typical sources of noise

punctuation, HTML tags, URLs, emojis
inconsistent casing and formatting
very frequent but low-information words (stopwords)
morphological variants (user, users, using)

Preprocessing options (choose what fits the analytical approach)

Text cleanup
remove punctuation / HTML / special characters
Tokenization
split text into words or n-grams
Stopword removal
remove frequent low-information words
Stemming / lemmatization
reduce related word forms to a common base
Advanced linguistic processing
POS tagging, parsing, word sense disambiguation, synonym / pronoun normalization

➜

Text processing: Examples

Stopword removal:

from sklearn.feature_extraction.text import CountVectorizer

documents = ["Jupiter is the largest gas planet."]

vectorizer = CountVectorizer(stop_words="english")
matrix = vectorizer.fit_transform(documents)
print(vectorizer.get_feature_names_out())

['gas' 'jupiter' 'largest' 'planet']

➜

Feature generation: Sparse representations (classical NLP)

A document is treated as a bag of words or terms.

each word or term becomes a feature
word order is ignored
each document is represented as a vector
the result is a structured document-term matrix

Key idea: Text is transformed into numerical features that can be used by classical machine learning models.

Ways to create vectors

Technique	Feature value
Binary term occurrence	term appears: yes / no
Term occurrence	number of times a term appears
Term frequency	occurrences divided by document length
TF-IDF	frequency weighted by how distinctive the term is

➜

Feature generation: TF-IDF example

TF-IDF (term frequency-inverse document frequency) measures how important a term is to a document relative to a corpus

$tf$ (term frequency): how often a term appears in a document
$idf$ (inverse document frequency): how rare a term is across the corpus

\[idf_i = log(\frac{N}{df_i})\]

with $N$: total number of documents
$df_i$: the number of documents in which $t_i$ appears.

TF-IDF weight:

\[w_{ij} = tf_{ij} \times idf_i\]

Gives more weight to rare words
Gives less weight to common words (domain-specific stopwords)

➜

Feature generation: Dense representations (embeddings)

Words are represented as dense vectors of real numbers, e.g., 50–300 values per word
Dense vectors capture semantic meaning, i.e., similar words get similar vector representations
Well-known methods: Word2Vec, GloVe, fastText, BERT

➜

Feature generation: Dense representations (embeddings)

The sentence_transformers library offers pre-trained models for dense vectors:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

text = "Customer reviews mention fast delivery and helpful support."

dense_vector = model.encode(text)

print(type(dense_vector))
print(dense_vector.shape)
print(dense_vector[:10])

<class ‘numpy.ndarray’>
(384,)
[-0.042 0.018 0.067 …]

➜

Feature generation: Transformer-based analytics

Transformer models combine dense vectors with positional encoding:

➜

Feature selection

In one-hot encoded documents, or in large corpora, feature selection can help to facilitate computation and avoid overfitting. Two major feature selection approaches in the context of text analytics are:

Pruning methods: removing words that are too infrequent or to frequent, based on absolute or percental thresholds.

Token filtering based on part-of-speech (POS): sometimes, the analyses focus on particular classes of words. For example:

For sentiment analysis, adjectives often carry evaluative meaning
- good, bad, great
For clustering, nouns often define the topic or object
- red cars and blue cars are about cars
- red trousers and blue trousers are about trousers

➜

Text analytics: Clustering

Typical goal: find groups of documents that are similar to each other

Start with a set of documents
Represent each document as a vector
- sparse vectors: Bag-of-Words, TF-IDF
- dense vectors: embeddings
Compare documents using a similarity measure
Use a clustering algorithm to group similar documents

Similarity measures for text data

Representation	Typical similarity measure	Useful for
Bag-of-Words / TF-IDF	Cosine similarity	comparing word-use patterns
Sets of words	Jaccard similarity	overlap between vocabularies
Dense embeddings	Cosine similarity / Euclidean distance	semantic similarity

➜

Summary

Big data is not just “more data”; it changes analytics through volume, velocity, variety, and veracity. Each dimension creates a different managerial challenge: scaling decisions, acting in real time, integrating heterogeneous data, and deciding whether data can be trusted.
Big data architectures extend traditional data warehouses by separating data storage, data movement, and data transformation. Data warehouses emphasize structured, governed “single truth,” while data lakes store raw heterogeneous data; logical data warehouses provide virtual, query-time integration.
Text analytics turns unstructured text into analyzable data through a pipeline of preprocessing, feature generation, feature selection, and modeling. Classical approaches rely on sparse document-term matrices such as Bag-of-Words and TF-IDF.
Dense embeddings and Transformer-based models shift text analytics from manually engineered features toward semantic representations and integrated AI workflows. These representations support retrieval, similarity analysis, clustering, classification, summarization, information extraction, and RAG-based systems.

Survey: Session 8

https://forms.gle/hNuAD8UZvC8KYo4a8

References

Manning, C., & Schutze, H. (1999). Foundations of statistical natural language processing. MIT press. https://web.stanford.edu/~jurafsky/slp3/

Schmarzo, B. (2016). Big data MBA. Wiley. https://doi.org/10.1002/9781119238881