Statistical & Mathematical Methods for Data Science · Chapter 2

Data, Big Data & the Data Scientist, taught visually

Everything in PPT 2, rebuilt so you can see it: where data comes from, how much of it exists (with the slide's real numbers), what makes it "big", the five types of data, and who data scientists are. Every definition comes with a worked example, and the case scenarios are longer this time — each one asks two or three questions, the way a real exam case does.

Definitions — each with an example Case scenarios — descriptive, multi-part Calculations — read time & storage cost
01

Data all around — where it is collected and warehoused

The chapter opens with a simple observation: lots of data is being collected and warehoused — not by scientists in labs, but by ordinary systems doing ordinary business. Four sources are named on the slide. Learn each with its example, because scenario questions describe one of these and ask you to name it.

Web data & e-commerce

Every click, search and cart leaves a record.

Websites log page views, searches, clicks and purchases automatically. An online store doesn't just sell a shirt — it records that you viewed three shirts, compared prices for six minutes, read two reviews, and bought the blue one at 9:42 pm.

Example: Daraz or Amazon knowing you browsed running shoes — and showing you shoe ads for a week — is web/e-commerce data at work.

Why it matters: this data is mostly unstructured or semi-structured (logs, text, images), which is exactly why it's hard to analyse with old tools.

Financial & bank / credit transactions

Money moving = data moving.

Every ATM withdrawal, card swipe, mobile-wallet transfer and loan payment is captured by banks in real time. This is transactional data (Chapter 1) at industrial scale — structured, timestamped, and extremely sensitive.

Example: your bank blocking a card used in Karachi and Dubai within the same hour is possible only because both transactions were recorded and compared instantly.

Exam angle: financial data is the classic home of fraud / anomaly detection scenarios — the data exists precisely so unusual events can be spotted.

Online trading & purchasing

Markets generate data faster than humans can watch.

Stock exchanges and trading apps produce price ticks, buy/sell orders and order-book changes thousands of times per second. Purchasing platforms add orders, bids and auctions on top.

Example: a share price changing 40 times in one second is 40 data points nobody typed — machines generated them for other machines.

Exam angle: trading is the textbook Velocity example — the value of the data expires in milliseconds, so it must be processed as a stream.

Social networks

People, posts — and the connections between them.

Platforms collect posts, photos, likes, comments, follows and friendships. The unique part is not the posts; it's the network of relationships — who is connected to whom — which later becomes graph data (Section 05).

Example: Facebook suggesting "People you may know" is mining the friendship graph, not reading your posts.

Chapter link: the Obama campaign case (Section 09) is entirely about this source — 1 million app installs opened up the friends graph of millions more.

02

How much data do we have? The slide's hard numbers

These six figures are the chapter's most quotable facts — and the raw material for its calculations. Notice the two "physics" numbers at the end: disk is cheap ($35/TB) but reading is slow (3 hours per TB). That tension — storage is affordable, scanning is not — drives almost everything else in the chapter.

Google · 2008

20 PB / day

Data processed by Google every single day in 2008. One day's worth would take a single disk reader over 6 years to scan (Section 12 does this maths).

Facebook

60 TB / day

Of daily logs alone — the exhaust trail of clicks and actions, before counting photos or posts. Over a year that's ≈ 21,900 TB (21.9 PB) of logs.

eBay · May 2009

6.5 PB

Of user data already stored, plus 50 TB arriving every day. Reading the stored 6.5 PB once, end to end, would take about 2 years at the slide's disk speed.

1000 Genomes Project

200 TB

Of human genome data from a single research project — proof that science, not just commerce, produces big data. Even this "small" dataset takes ≈ 23 days to read fully at 100 MB/s.

Cost of storage

$35 / TB

The price of 1 TB of disk. Cheap enough that keeping data is rarely the problem — eBay's entire 6.5 PB would cost only about $227,500 in raw disks.

Time to read

3 hrs / TB

Reading 1 TB at 100 MB/s takes roughly 3 hours (the exact figure is 2.8 hrs — Section 12 shows the working). Data is cheap to keep, expensive to touch.

The unit ladder — every step is ×1,000

Examiners test whether "petabyte" is a feeling or a number to you. Each rung holds a thousand of the rung below it. The bar heights below grow the same way your intuition should: a PB is not "a big TB" — it is a million GB.

KB
kilobyte · a short text file
MB
1,000 KB · one photo
GB
1,000 MB · a film
TB
1,000 GB · a big disk — $35
PB
1,000 TB · Google's daily diet × 20
The calculation pattern hiding here: every number in this section combines with the two physics numbers. Storage cost = size in TB × $35. Read time = size in MB ÷ 100 MB/s. If the exam gives you any dataset from this slide, one of those two multiplications is the expected answer — practise both in Section 12.
03

Big Data — the definition, and the three V's done properly

Chapter 2 gives its own definition, and it's sharper than "a lot of data": Big Data is any data that is expensive to manage and hard to extract value from. A dataset earns the label by the pain it causes, not by crossing a magic size. The three V's then explain where the pain comes from — and note carefully: this chapter defines Velocity differently from Chapter 1.

V1

Volume

Slide definition: "The size of the data."

The sheer amount — terabytes and petabytes that no single machine can hold comfortably or scan in reasonable time.

Example: eBay's 6.5 PB of user data. At 100 MB/s, merely reading it once takes ~2 years — so "just scan everything" stops being a strategy.
V2

Velocity

Slide definition: "The latency of data processing relative to the growing demand for interactivity."

Read that again — in this chapter velocity is about how fast answers must come back compared to how long processing takes. Users expect interactive, instant responses; big data makes processing slow. The gap is the velocity problem.

Example: a fraud check on a card payment must answer in milliseconds while the customer waits at the till — but it has to reason over billions of past transactions. High demand for interactivity + slow processing = velocity pain.
V3

Variety & Complexity

Slide definition: "The diversity of sources, formats, quality, structures."

Data arrives as tables, free text, XML, graphs and streams, from many sources, at many quality levels. This chapter pairs variety with complexity — the mess itself is part of the definition.

Example: a hospital holds lab results in tables, doctors' notes as text, X-rays as images and monitor feeds as streams — four formats, four tools, one patient.

The Big Data wheel, flattened — how each V grows

The slide's wheel diagram shows the three V's as rings around a small, calm centre. In the middle sits old-fashioned data: a table, read in batch, measured in KB. Each ring outward is more "big data" than the last. Read each row left → right and watch one V escalate:

Volumehow much
KBMBGBTBPB
Velocityhow immediate
BatchPeriodicNear real timeReal time
Varietywhat forms
TableReportsDatabasePhoto · WebAudio · VideoCMSSMS · Social · Mobile apps

The exam point of the wheel: data becomes "big" by moving outward on any ring — a tiny but real-time stream (velocity) or a modest pile of mixed formats (variety) can be Big Data even at small volume. And the centre never disappears: ordinary tables still exist inside every big-data system.

04

Definition flip-cards — now with an example on every card

Say the definition out loud, then tap. The back of each card gives the slide's wording (highlighted — that's what markers scan for) plus a concrete example, because this chapter's exam pairs them: "Define X and give an example" is the standard two-mark format.

05

Types of data we have — five shapes, five different problems

"What type of data is this?" is a favourite scenario question, and the chapter lists exactly five types. Each card below gives the definition, a concrete example, and the trigger words that identify it in an exam story. The last one — streaming — carries the chapter's most quoted line.

Relational data
Tables · transactions · legacy data

Definition: data organised into tables of rows and columns with fixed structure, stored in relational databases — including old legacy systems still running core business.

Example: a bank's account table (account no., name, balance) and its transaction table — every ATM withdrawal is one neat row.

Triggers: "table", "rows and columns", "SQL", "legacy system", "transaction records"
Text data
The web's native format

Definition: unstructured natural language — web pages, articles, emails, reviews, posts. No rows, no columns; meaning lives in the words themselves.

Example: 40,000 product reviews on an e-commerce site. The insight ("customers hate the battery") is inside sentences, not fields.

Triggers: "web pages", "reviews", "emails", "articles", "free text", "comments"
Semi-structured data
XML — structure without a table

Definition: data that is not in tables but carries tags and markers that separate elements and describe what they are — self-describing, flexible, nested.

Example: an XML product feed: <book><title>…</title><price>…</price></book> — you can read its structure without a schema document.

Triggers: "XML", "tags", "self-describing", "nested", "no fixed schema but organised"
Graph data
Social networks · Semantic Web (RDF)

Definition: data modelled as nodes and the relationships (edges) between them, where the connections are as important as the things themselves.

Example: Facebook's friendship network — people are nodes, friendships are edges. In the Semantic Web, RDF stores facts as triples: (Karachi) —isIn→ (Pakistan).

Triggers: "network", "connections", "friends of friends", "RDF", "linked data", "relationships"
Streaming data
Flows past — scan it once, or lose it

Definition: data that arrives as a continuous, high-speed flow and cannot all be stored first. The slide's exact warning: "You can afford to scan the data once."

Example: live card transactions or stock ticks. Each item gets one look as it passes — your algorithm must decide on the spot, because the full stream will never sit still in a database.

Triggers: "live feed", "real time", "sensor stream", "scan once", "cannot store everything"
How to tell them apart
The 10-second test

Ask three questions of any dataset in a scenario: (1) Does it sit in tables? → relational. (2) Is it words, or tagged/nested? → text or semi-structured. (3) Is it about connections — or does it never stop moving? → graph or streaming.

Exam habit: name the type AND the clue that proves it — one extra line, full marks.
"Scan once" is the streaming trap: if a scenario says a system cannot store the data and analyse it later — the feed is too fast or too large, so each record is examined a single time as it flows past — the answer is streaming data, and the consequence is that algorithms must work in one pass, keeping summaries instead of the raw data.
06

What to do with these data — the five jobs

Collecting data is pointless without operations on it. The slide lists five, and they form a rough ladder: from summarising what you have, through finding things in it, up to discovering knowledge nobody asked for explicitly. Each job below has a definition and an example — scenarios usually describe the job and ask you to name it.

JOB 1

Aggregation & Statistics

Summarise masses of records into totals, averages and counts. Example: 2 million receipts → "average basket = Rs 1,850, peak hour = 8 pm."

JOB 2

Data Warehousing & OLAP

Store history in a warehouse built for analysis, then slice it interactively with OLAP (Online Analytical Processing). Example: sales by region → by city → by month, drilled down live in a meeting.

JOB 3

Indexing, Searching & Querying

Build indexes so specific items can be found instantly instead of scanning everything; includes keyword-based search. Example: Google finding one page among billions in 0.4 seconds.

JOB 4

Pattern Matching

Find data that matches a structural pattern — the slide names XML / RDF. Example: "find every product (node) connected to supplier X within two hops" in linked data.

JOB 5

Knowledge Discovery

The top of the ladder: data mining and statistical modeling extract patterns and models nobody specified in advance. Example: discovering that diaper buyers also buy snacks on Friday evenings.

JobYou already know…You get back…Works best on
Aggregation & StatisticsWhat summary you want (total, mean)Descriptive numbersRelational / transactional data
Warehousing & OLAPThe dimensions you want to exploreInteractive multi-dimensional views of historyRelational data, accumulated over time
Indexing, Searching, QueryingThe keywords or item you're looking forThe matching records, fastText data (web), any large collection
Pattern MatchingThe shape of what you're looking forAll sub-structures that fit the shapeSemi-structured (XML), graph (RDF)
Knowledge DiscoveryOnly that something interesting may existNew patterns & predictive modelsAny type — this is data mining's home

The pattern in the table: as you go down the rows, you specify less and the system discovers more. Search answers the question you asked; knowledge discovery answers questions you didn't know to ask. That contrast is a ready-made exam essay point.

07

What is Data Science? — Chapter 2's own definitions

This chapter defines data science twice, and both wordings are examinable. Learn them as a pair: the first is about what it does to data, the second about where its methods come from.

"An area that manages, manipulates, extracts, and interprets knowledge from tremendous amount of data."

— Definition 1 (the verbs definition) · Notice: the object is knowledge, not data — DS starts with data but its product is knowledge. Four verbs worth memorising in order: manages → manipulates → extracts → interprets.

"A multidisciplinary field of study with goal to address the challenges in big data."

— Definition 2 (the fields definition) · And the slide's crucial footnote: "Data science principles apply to all data — big and small." Big data is the motivation, not a entry requirement.

Eight ingredients feed into Data Science

The slide's hub diagram: eight disciplines and mindsets pour into one centre. A scenario that describes a person or a course is usually pointing at one of these eight.

Data EngineeringBuilding the pipelines and storage that move and hold the data.
Scientific MethodHypothesis → experiment → evidence. Keeps analysis honest.
MathThe language underneath: modeling, algebra, optimisation.
Domain ExpertiseKnowing the field the data is about — medicine, finance, retail…
Data Sciencewhere all eight meet
StatisticsProbability, statistical & stochastic modeling, inference.
Hacker MindsetCurious, hands-on problem solving — get the data, make it work.
VisualizationTurning results into pictures people can reason about.
Advanced ComputingHigh-performance computing, databases, AI, pattern recognition, data warehousing.

Who uses it, and what each parent field contributes

The slide stresses that theories and techniques from many fields are used to investigate and analyse large amounts of data — to help decision makers in science, engineering, economics, politics, finance and education. The three parent disciplines split the work like this:

Computer Science
The machinery

Contributes pattern recognition, visualization, data warehousing, high-performance computing, databases and AI — everything that makes analysis possible at scale.

Mathematics
The models

Contributes mathematical modeling — turning real problems into formal structures that can be computed, optimised and proved things about.

Statistics
The uncertainty

Contributes statistical and stochastic modeling and probability — the tools for reasoning honestly about randomness, samples and confidence.

The "multidisciplinary" trap: the slide's Venn diagram (by Brendan Tierney, 2012) overlaps Statistics, Pattern Recognition, Neurocomputing, AI, Machine Learning, Data Mining, KDD, Databases & Data Processing and Visualisations — all orbiting one centre: Data Science. If a question asks "why is data science called multidisciplinary?", the answer is this overlap: no single parent field covers it, and the outer ring adds business skills — strategy, domain knowledge, communication, presentation, inquisitiveness, problem solving.
08

Why Data Science — the "sexy job" evidence

The chapter answers "why data science?" with labour-market evidence, not philosophy. Three exhibits: a famous quote, a famous shortage, and a famous curve. All three are short-answer gold.

"…the sexy job in the next 10 years will be statisticians."

— Hal Varian, Google's Chief Economist · The quote that launched a thousand degree programmes. Note he said statisticians — the field's statistical roots showing.

"Data Scientist: The Sexiest Job of the 21st Century"

— Harvard Business Review, October 2012 (linked on the slide) · The article title that made "data scientist" a job people actually applied for.

McKinsey Global Institute · June 2011

140,000–190,000

Predictive analysts the U.S. alone would need by 2018 — deep analytical talent that did not exist yet in sufficient numbers.

McKinsey Global Institute · June 2011

1.5 million

Managers and analysts additionally needed by 2018 who could understand and make decisions based on big data analysis. The shortage was at both levels: the people who build models, and the people who must act on them.

The supply side responded: new data science institutes were created or repurposed at NYU, Columbia, Washington and UC Berkeley, and new degree programmes, courses and boot-camps appeared — at Berkeley spanning Stats, the I-School, CS and Astronomy. One proposal even floated an MS in "Big Data Science".

Gartner's 2014 Hype Cycle — where Data Science sat

The slide asks "Why is it sexy?" and answers with this curve. Every technology rides the same five phases. In July 2014, Data Science sat just below the Peak of Inflated Expectations and Big Data just past it, sliding toward the Trough — both marked "plateau in 5–10 years". Tap each phase to learn what happens there.

Data Science Big Data InnovationTrigger Peak of InflatedExpectations Trough ofDisillusionment Slope ofEnlightenment Plateau ofProductivity expectations ↑ time →
09

Real-life examples — the chapter's two case studies

The slide introduces this section with a warning worth quoting: "Companies learn your secrets, shopping patterns, and preferences." Both case studies prove it — one from retail, one from politics. These are the most likely sources of long descriptive case questions, so learn them as stories with steps, not as one-liners.

Case A — Target: "Can we know if a woman is pregnant, even if she doesn't want us to know?"

The slide poses exactly this question. Target, the US retailer, assigned each shopper a guest ID and recorded everything bought with it. Then its analysts went hunting through the purchase histories of women who had signed up to baby registries — and found a pattern hiding in ordinary shopping.

1

Collect transactional data at scale

Every purchase by every loyalty/guest customer is logged — what was bought, when, and by whom (Chapter 1's transactional data, warehoused).

2

Find a labelled group to learn from

Women on the baby registry had voluntarily revealed they were pregnant — giving analysts labelled examples: known-pregnant shoppers and their baskets.

3

Mine the pattern (knowledge discovery)

Comparing baskets revealed tell-tale shifts — e.g. large amounts of unscented lotion early on, then supplements like calcium, magnesium and zinc. Roughly 25 products combined into a "pregnancy prediction" score, accurate enough to estimate even the due date.

4

Apply it to everyone else

The model scored all female shoppers — including those who had told nobody. Target could then mail baby-product coupons timed to each trimester.

5

The consequence — and the lesson

In the famous incident, a father complained that Target was sending his teenage daughter baby coupons — and later discovered Target knew before he did. The exam lesson: this is prediction of a missing attribute (pregnant: yes/no) from transactional data — powerful, commercial, and ethically loaded.

Exam mapping: Data type = transactional · Job = knowledge discovery / data mining · Technique family = classification-style prediction · Slide's moral = companies learn your secrets from shopping patterns.

Case B — Data Science and the elections (2008 & 2012)

The second case moves the same ideas into politics. The Obama campaigns treated voters the way Target treated shoppers: as data to be modelled, segmented and persuaded.

1

Build a door into the social graph

The campaign released a Facebook app. Installing it asked for access not just to the supporter's profile, but to information about their friends.

2

One million installs — millions more reached

The slide's number: 1 million people installed the Obama Facebook app, and each install opened a window onto that person's friend network — so the campaign's real data reach was the graph, far larger than the install count.

3

Use the network, not just the nodes

Friends of supporters are the most persuadable voters — and a message arriving via a trusted friend beats an advert. This is graph data (Section 05) turned into campaign strategy across both the 2008 and 2012 elections.

4

The transferable lesson

Retail predicts pregnancies; campaigns predict voters. Same pipeline: collect behavioural data → mine patterns → predict a hidden attribute → act on the prediction. Only the domain changed — which is why domain expertise is one of the eight ingredients.

Exam mapping: Data type = graph data (social network) · Key number = 1 million installs · Hidden multiplier = each install exposes friends' data · Theme = data science shapes decisions in politics, one of the slide's listed decision-maker fields.
10

Data scientists — who they are and what they actually do

The chapter's portrait has three parts: a one-line identity, a three-part job description, and the fields they work in. Scenario questions usually disguise one of the three and ask you to spot it.

"They find stories, extract knowledge. They are not reporters."

— The slide's identity line · A reporter retells what happened; a data scientist discovers what nobody knew was there — the story is found in the data, not handed over in a press release.

"The Sexiest Job of the 21st Century"

— The slide's title for the role (Harvard Business Review, 2012) · Remember the evidence chain: Varian's quote → McKinsey's shortage numbers → HBR's label (Section 08).

The three-part job — memorise this sentence structure

The slide: "Data scientists are the key to realizing the opportunities presented by big data." How? In three moves — and notice they climb from raw mess to boardroom decision:

MOVE 1

Bring structure to it

Big data arrives messy and formless (Variety & Complexity). The data scientist imposes structure — clean tables, defined variables, usable datasets.

MOVE 2

Find compelling patterns in it

Knowledge discovery applied by a human: not just any pattern, but compelling ones — useful, nonobvious, worth acting on (Chapter 1's two tests again).

MOVE 3

Advise executives on implications

Translate patterns into consequences for products, processes and decisions — the exact trio the slide names. The job ends in the boardroom, not the notebook.

Where do they do it? The slide's domains

"What do data scientists do?" — the slide answers with application areas ("…and more"). Learn the five named ones with an example task each, because scenarios open inside one of these domains:

National Security
Protect the state

Example task: mining communication and movement data to flag genuine threats among millions of harmless signals — anomaly detection where the anomaly matters.

Cyber Security
Protect the systems

Example task: modelling normal network traffic so an intrusion — traffic that breaks the pattern — is caught as it streams in.

Business Analytics
Grow the company

Example task: the Target pattern — segmentation, basket analysis and churn prediction that steer products, prices and marketing.

Engineering
Build things that last

Example task: sensor data from bridges, engines or production lines, mined to predict failures before they happen.

Healthcare
Treat patients better

Example task: patient records and genome data (remember the 200 TB) mined for risk factors, readmission patterns and drug responses.

…and more
Anywhere decisions meet data

The slide deliberately leaves the list open — finance, education, politics and science from Section 07's decision-maker list all employ the same three moves.

Concentration in Data Science — what to study

The chapter closes with the recipe for becoming one. Six components, straight off the slide — a "which of these belongs in a DS degree?" question is answered from this list:

Mathematics & Applied Mathematics
Foundation 1

The modeling language everything else is written in.

Applied Statistics / Data Analysis
Foundation 2

Probability and statistical modeling, aimed at real data rather than pure theory.

Solid Programming Skills
The tools — name them

R, Python, Julia, SQL — the four languages the slide lists. (Scenario trap: SQL = the relational-data language; R/Julia/Python = analysis and modeling.)

Data Mining
Core skill 1

Extracting patterns and knowledge from stored data — Section 06's top rung.

Database Storage & Management
Core skill 2

Where data lives and how it's organised — warehouses, indexes, the plumbing of Sections 05–06.

Machine Learning & Discovery
Core skill 3

Algorithms that learn patterns and surface discoveries — the engine of prediction.

11

Case scenarios — descriptive, multi-part, exam-style

These are written the way your professor writes cases: a full story first, then two or three linked questions (shown as amber tags under each story). The highlighted phrases are the clues planted in the story. Commit to answers for all parts before you reveal — in the exam, the parts build on each other.

0 of 8 cases revealed
Work through all eight before the exam.
12

Calculation lab — read time & storage cost

Chapter 2's calculations grow out of two slide facts: a disk reads at ~100 MB/s and 1 TB of disk costs $35. Every question in this family is one of two formulas. Pick a dataset from the chapter, drag the disk speed, and watch both answers — then do the two worked examples by hand.

The chapter's datasets, measured two ways

Choose a dataset straight from the slide. Cost uses decimal units (1 TB = 1,000 GB = 1,000,000 MB), exactly as the slide's "3 hours per TB" implies.

Read time = size (MB) ÷ speed (MB/s)   ·   Storage cost = size (TB) × $35
100 MB/s
—to read it once, end to end
—in raw disk storage at $35 / TB

Worked example 1 — why Google cannot use one machine (the velocity argument)

Google: 20 PB/day = 20,000,000,000 MB→ ÷ 100 MB/s = 200,000,000 s→ ÷ 86,400 s/day ≈ 2,315 days= ≈ 6.3 years

One day's data would take a single reader over six years to scan — while a new day's data arrives every 24 hours. Conclusion the examiner wants: processing must be distributed across many machines and much data must be handled as a stream, because the latency of one-machine processing can never meet the demand for interactivity. That is this chapter's Velocity definition, proved with arithmetic.

Worked example 2 — Facebook's log bill for one year

60 TB/day × 365 days= 21,900 TB per year→ 21,900 × $35= $766,500 / year

And that's logs only, on the cheapest possible disks, with no backup copies. It shows why the slide's Big Data definition says "expensive to manage" — and why companies summarise and aggregate logs instead of keeping every raw line forever. Method to copy: convert to a per-day rate, scale to the period asked, then apply $35/TB.

Unit trap: the slide's numbers only work with decimal units — 1 TB = 1,000,000 MB gives 10,000 s ≈ 2.8 hrs ≈ 3 hrs at 100 MB/s, matching the slide. If you use 1,024-based units you'll get ~3.2 hrs and confuse yourself. In the exam, show the conversion line; markers award the method even if rounding differs.
13

Rapid fire — last-check questions

Short-answer material from the chapter's corners: attributions, distinctions, and the one-line facts that examiners scatter through a paper.

State the chapter's definition of Big Data, and its three V's with this chapter's wording.
Big Data is any data that is expensive to manage and hard to extract value from. The V's: Volume — the size of the data; Velocity — the latency of data processing relative to the growing demand for interactivity; Variety and Complexity — the diversity of sources, formats, quality and structures.
Chapter 1 called velocity "data arriving fast". Chapter 2 defines it differently — what's the difference, and which should you use?
Chapter 2 frames velocity as a gap: processing latency vs. the demand for interactivity. Fast arrival is part of it, but the definition is about answers not coming back fast enough for interactive use. In a Chapter 2 question, quote the latency definition — and you can mention arrival speed as the cause of the latency.
"You can afford to scan the data once" — what is this about?
Streaming data. The flow is too fast/large to store completely, so each record is examined a single time as it passes. Algorithms must decide on the spot and keep summaries or models, not the raw stream.
Who said statisticians would be "the sexy job in the next 10 years", and who published "The Sexiest Job of the 21st Century"?
The quote is Hal Varian, Google's Chief Economist. The title is Harvard Business Review (October 2012). Two different sources, two different wordings — don't merge them in an answer.
What did McKinsey's June 2011 report actually predict?
That by 2018 the U.S. would need 140,000–190,000 predictive analysts and 1.5 million managers/analysts able to use big data analysis. Both numbers, with the "by 2018" and "U.S." qualifiers, make the complete answer.
Where did Data Science and Big Data sit on Gartner's 2014 Hype Cycle?
Data Science was just below/at the Peak of Inflated Expectations; Big Data had just passed the peak and was sliding into the Trough of Disillusionment. Both were marked as reaching the plateau in 5–10 years. The five phases in order: Innovation Trigger → Peak of Inflated Expectations → Trough of Disillusionment → Slope of Enlightenment → Plateau of Productivity.
XML and RDF appear twice in this chapter — where?
First as data types: XML is the example of semi-structured data; RDF belongs to graph data (Semantic Web). Second under what to do with data: pattern matching (XML/RDF) is the job designed for them. Same pair, two lists — examiners love asking for both.
Which four programming languages does the concentration slide name?
R, Python, Julia and SQL — under "solid programming skills". The other concentration components: mathematics & applied mathematics, applied statistics/data analysis, data mining, database storage & management, and machine learning & discovery.
A classmate says "data science only works on big data." Correct them using the slide.
The slide states: "Data science principles apply to all data — big and small." Big data is where DS became necessary and famous, but its methods — modeling, mining, visualisation — are defined for data of any size.
In one line each: what do data scientists do to big data, and what do they give executives?
They bring structure to big data and find compelling patterns in it; they give executives advice on the implications of those patterns for products, processes and decisions.