Statistical & Mathematical Methods for Data Science · Chapter 1
Everything in PPT 1, rebuilt so you can see it: flip the definition cards, tap through both pyramids, work the case scenarios like exam questions, and run the calculations yourself. The professor tests in three formats — this guide is organised in exactly those three.
The single most likely definition question: "Distinguish data science, machine learning, data mining and statistics." Each field below has one essence line — memorise those four lines and you can reconstruct the rest.
The goal: better decisions from big data.
Improves decision-making by basing decisions on insights extracted from large datasets. It pulls out actionable insights — principles, problem definitions, algorithms and processes that reveal nonobvious, useful patterns.
Exam trap: Data science is broader in scope than both machine learning and data mining — it contains them, it isn't a synonym for either.
The algorithms that find the patterns.
Focuses on the design and evaluation of algorithms for extracting patterns from data. A pattern only counts if it passes two tests: it must be useful and nonobvious — otherwise it isn't worth the effort.
Exam trap: If a scenario says "designing / evaluating an algorithm," the answer is ML — even if the word "data" appears everywhere.
Patterns in structured, stored data.
Generally deals with the analysis of structured data, which usually lives in relational databases. It often carries an emphasis on commercial applications, powered by data warehouses and analytical databases.
Exam trap: "Structured data + relational database + business use" is the data-mining fingerprint.
The oldest science of data.
The branch of science dealing with the collection and analysis of data. Its simplest form, descriptive (summary) statistics, uses measures of central tendency (e.g. the arithmetic mean) and measures of variation (e.g. the range).
Exam trap: Classic statistics asks why the data looks the way it does (the hidden model that generated it). ML/DS ask what predicts best. Scenario 9 in the practice set tests exactly this.
| Field | Core question it answers | Works with | One-line identity |
|---|---|---|---|
| Data Science | "What decision should we make?" | Large datasets, any form | Broadest scope — decisions from insights |
| Machine Learning | "Which algorithm extracts the pattern best?" | Data fed to algorithms | Design + evaluation of algorithms |
| Data Mining | "What patterns hide in our stored data?" | Structured data in relational databases | Commercial focus, warehouse-powered |
| Statistics | "How was this data generated, and what does it summarise to?" | Collected samples & populations | Collection + analysis; mean & range at its simplest |
Read the term, say the definition out loud, then tap to check. The highlighted phrases on the back are the exact wording markers are scanning for.
Both pyramids say the same thing two ways: raw material at the bottom, value at the top, and most of the work (and most of the marks) is in knowing what each layer adds. Tap any layer to see what it means and an example.
How raw signals become wisdom. Each layer adds context and meaning.
Memory hook: every step up answers a bigger question — What? → So what? → How? → Why / what now?
How an organisation climbs from raw records to harvested value.
Exam favourite: a company that "only makes dashboards" is stuck at layer 2 — Charts. No predictions, no actions, no value harvested.
Big Data is defined by three properties. If a scenario mentions data that is huge, messy in format, or arriving too fast to store first — it's pointing at one of these.
Data at a scale too large for ordinary tools — terabytes to exabytes and beyond.
Many forms at once: neat tables (structured) plus text, images, audio and social posts (unstructured).
Data arrives as a continuous, high-speed stream that must be processed as it flows, not later.
The slide's timeline, drawn out. Notice how the "type" of data shifts from one-off records to continuous generation.
~3200 BCE
Mesopotamia. A sale is recorded — transactional data: one record of one exchange.
~3000 BCE
Egypt. A census counts people for tax collection — non-transactional (demographic) data.
Writing → 2013
5 exabytes total created in roughly 5,000 years of recorded history.
2013 → today
5 exabytes every 2 days. The same amount now arrives before your weekend ends.
5 exabytes ÷ 2 days — the current daily creation rate
created in 30 days at that rate (2.5 × 30) — a likely exam calculation
2.5 × 365 — nearly a zettabyte of new data every year
The professor's favourite format for Big Data: a short story, and you must say whether the core problem is Volume, Variety or Velocity — or all three at once. Watch for the traps: a problem can be tiny in size and still be Big Data, and a huge dataset can be a problem of only one V.
The chapter's timeline quietly introduces the two oldest kinds of data there are. Transactional data records events — things that happened between parties. Non-transactional data describes entities — what people and things are like. Almost every database you will ever meet holds both, and exam scenarios ask you to tell them apart.
Records of events — things that happened.
Captured at the moment an exchange or activity occurs. Each record is one event and carries the same anatomy: who was involved, what happened, when (a timestamp), and usually how much. It is generated continuously and grows fast.
Examples: a sale receipt, an ATM withdrawal, a flight booking, a card payment, a hospital admission, a click on a website.
Chapter anchor: Mesopotamia, ~3200 BCE — a merchant recording a sale. The first data in history was transactional.
Descriptions of entities — what things are.
Data about people, products and places that records no exchange: attributes and characteristics. It changes slowly (an address, a birth date, a product's category) and exists to give transactional data its context — it is the reference, or master, data of a system.
Examples: census demographics, a customer profile (name, address, occupation), a product catalogue, a student record, a price list.
Chapter anchor: Egypt, ~3000 BCE — a census counting people for tax collection. Demographic data, not exchanges.
| Transactional | Non-transactional | |
|---|---|---|
| Records | An event / exchange between parties | Facts and attributes of an entity |
| Always has | A timestamp — it happened at a moment | No moment — it simply is |
| How it changes | Constantly appended, record by record | Rarely updated, field by field |
| Volume behaviour | Huge and growing every second | Comparatively small and stable |
| Main job | Run operations; reveal behaviour over time | Provide context — who/what the events involve |
| Ancient example | Mesopotamian sale record (~3200 BCE) | Egyptian census (~3000 BCE) |
When the chapter says ML and data mining "extract patterns", these six are the shapes those patterns take. Learn each by its picture and its trigger words — the scenario cards in the next section are all one of these six in disguise. And remember the chapter's quality test: a pattern only counts if it is useful and nonobvious.
Items or events that co-occur more often than chance. The market-basket pattern: customers who buy nappies on Friday evenings also buy snacks — so the store shelves them together.
Patterns learned from labelled examples become rules that assign new items to categories decided in advance: this email is spam, that transaction is fraud, this tumour is benign.
Natural groupings that emerge from similarity alone — no labels are given beforehand. The algorithm discovers that customers fall into usage tiers; naming the groups is the human's job afterwards.
A rare case that deviates sharply from the normal pattern — valuable precisely because it is rare: the $50,000 transfer from a $200 account, the sensor reading ten times its usual value.
Patterns where order and time carry the meaning: browse → add to cart → buy; demand rising every winter; a patient's readings drifting steadily worse. The same events shuffled would say nothing.
As one variable rises, another reliably rises (or falls) with it: temperature with ice-cream sales, study hours with marks. The workhorse pattern behind charts and reports — and behind most bad conclusions.
The professor's scenarios hide the concept inside a story. The highlighted phrases are the trigger words that give the answer away — train your eye to spot them before you reveal.
A storage constraint means the data you have — or keep generating — no longer fits the storage you own, or costs more to keep than it is worth. Chapter 1 sets this up directly: data is now created at 2.5 exabytes a day, but disks, servers and budgets are finite. Every scenario below is that collision wearing a different costume.
Storage capacity grows in steps — you buy a disk, it fills, you buy another. Data creation grows continuously. Sooner or later the curves cross, and someone has to decide what gets kept, summarised, sampled or thrown away.
Past the crossing point, "keep everything forever" stops being a plan. The six moves below are what replace it — and each practice scenario that follows is solved by one of them.
Almost every storage-constraint answer is one of these six moves. Learn the menu first, then spot which one each scenario is asking for.
A representative sample carries the population's patterns at a fraction of the size — you rarely need all the data to learn from it.
Replace raw rows with means, totals and hourly aggregates. A summary keeps the meaning and drops the bulk.
Squeeze files and move cold data to cheaper, slower tiers; keep only hot, recent data on fast storage.
With high Velocity you can't store first and think later — analyse as data flows, keep events and exceptions, not everything.
Spread data across many machines that grow by adding more — the data-warehouse infrastructure data mining is built on.
Data nobody uses is a cost, not an asset. Set a retention policy and delete on schedule — GIGO's cousin: junk kept is junk paid for.
A classic exam discussion question, and the answer is hiding in this chapter's own definitions: a pattern only counts if it is useful and nonobvious, and the goal of data science is better decisions — not a bigger percentage. Run the demo first; the six reasons underneath will then feel obvious.
Each square stands for 10 emails. Blue is legitimate mail, red is spam. Run two different spam filters over the same inbox and compare how the user feels — not just the score.
headline accuracy
spam actually caught
good emails lost to junk
Any two or three of these, well explained, earn full marks on this question. Learn all six and pick the ones that fit the scenario you're given.
95% can mean doing literally nothing.
Accuracy counts every case equally. When 95% of the data belongs to one class (not-spam, no-fraud, healthy), a model that always predicts that class scores 95% while learning nothing at all — exactly what Filter A did above.
Exam phrasing: accuracy is misleading on imbalanced data. Always ask: "95% of what — and what is the base rate?"
The wrong 5% can be the only 5% that matters.
The model is perfect on the easy, harmless majority and wrong on the rare cases the user actually cares about: the fraud that slipped through, the cancer that was missed, the spam still flooding the inbox. One expensive miss outweighs hundreds of cheap correct calls.
Exam phrasing: judge a model by the cost of its errors, not just the count of them.
It breaks the chapter's own rule for patterns.
A pattern counts only if it is useful AND nonobvious. A model that learns "most emails aren't spam" or "umbrella sales rise when it rains" can be highly accurate while telling the user nothing they didn't already know — no insight, no value. Hamming: insight, not numbers.
Exam phrasing: high accuracy on a trivial or already-known pattern is not a data-science success.
A score is not an outcome.
The goal of data science is improved decision-making. If the model's output never reaches the Actions layer of the pyramid — no process changes, no one acts differently — the user receives a number, not a benefit, however accurate it is.
Exam phrasing: value is only harvested at the Actions layer; accuracy stops at Predictions.
The 95% was measured in the lab.
The score usually comes from clean, historical test data. Live data is messier — missing values, new behaviour, shifting patterns — so real-world accuracy drops, sometimes sharply. The user judges the model on the live version, not the lab report. That's GIGO working in reverse.
Exam phrasing: accuracy on test data overstates performance on unseen, real data.
95% overall can be 60% for someone.
One headline number averages over everyone. The model might be near-perfect for most users and terrible for one group — a region, a language, a customer type. That group's users experience a broken product, and their unhappiness is correct, not irrational.
Exam phrasing: always check accuracy per group, not only in aggregate.
"Accuracy alone does not measure value. First, if the data is imbalanced — say 95% of cases in one class — a model can reach 95% by always predicting that class, learning nothing useful or nonobvious. Second, errors are not equal: the missed 5% are usually the costly cases (fraud, disease, spam), so the user's actual problem remains unsolved. Third, the goal of data science is improved decision-making — a model creates value only when it changes an action. Finally, the 95% is measured on test data; on messy real-world data, performance is lower. A model should therefore be judged by the value of what it gets right and the cost of what it gets wrong, not by accuracy alone."
Don't just read the formulas — move the sliders and type your own numbers until the method is automatic. These are the three calculation types this chapter can produce.
Assignments & quizzes 40% · Midterm 30% · Final 30%. Drag each score and watch the weighted total build up segment by segment.
The two descriptive statistics named in this chapter. Type any list of numbers, separated by commas.
Method: always convert to a per-day rate first, then multiply by the period asked. The same pattern answers "per week" (×7 = 17.5 EB) or "per year" (×365 ≈ 912 EB).
Scenario questions love this set: they describe a job ("draw a histogram in two lines of code") and you name the library. Learn each one by its job, not its name.
The numerical foundation — fast multi-dimensional arrays that almost everything else is built on.
Builds on top of NumPy — optimisation, integration, and scientific algorithms.
Advanced 2D data structures (rows × columns) for cleaning and wrangling tabular data.
The original plotting library — full control over every pixel of a chart.
Attractive statistical plots — e.g. a histogram in minimal code, built on Matplotlib.
Fitting and testing statistical models — regression, hypothesis tests, formal inference.
The short-answer material that doesn't fit anywhere else: two quotable lines, the five stages where humans stay in charge, and the lab rules that could appear as easy-mark questions.
"The purpose of computing is insight, not numbers."
— Richard Hamming, 1962 · Why we analyse at all: charts and totals are means, insight is the end.
"A data scientist is a statistician who works at a startup — who knows more statistics than a computer scientist and more computer science than a statistician."
— The classic job description · The role sits exactly between the two disciplines.
Algorithms do the heavy lifting, but a human stays responsible at every one of these checkpoints:
Decide what question is actually worth answering.
Choose, gather and clean the data the model will learn from.
Pick the method that fits the problem and the data.
Critically judge whether the output makes sense.
Turn findings into decisions — the top of both pyramids.