STAT5003 — Revision with Examples / 总复习 + 例子

Each point has a 💡 simple example. 每个知识点配一个简单例子。Final: 12 MCQ (2 correct each, 24) + 4 short-answer (36). 练习：真题练习。

Week 2 · Linear Regression 线性回归Regression

Statistical learning & least squares

$Y=f(X)+\epsilon$; supervised (numeric→regression / categorical→classification) vs unsupervised. Minimise $\text{RSS}=\sum(y_i-\hat y_i)^2$; $\hat\beta_1=\frac{\text{Cov}(x,y)}{\text{Var}(x)}$, $\hat\beta_0=\bar y-\hat\beta_1\bar x$.

💡 例子（手算斜率）study hours $X=\{1,2,3\}$, score $Y=\{2,4,6\}$ → each extra hour adds 2 marks, $\hat\beta_1=2,\hat\beta_0=0$, line $\hat y=2x$.学 1/2/3 小时得 2/4/6 分 → 斜率 2、截距 0、$\hat y=2x$，且正好过均值点 $(2,4)$。

Must know 必记

Line passes through $(\bar x,\bar y)$; minimises vertical residuals; closed-form coefficients.过均值点；竖直残差；有闭式解。

💡 例子（竖直距离）point $(2,5)$, line predicts $4$ → residual $=1$ measured vertically.点 (2,5)、线预测 4 → 残差是竖直的 1，不是到斜线的垂距。

SE / CI / test / $R^2$

$\text{SE}(\hat\beta_1)=\frac{\sigma}{\sqrt{\sum(x_i-\bar x)^2}}$; test $H_0:\beta_1=0$; $R^2=\frac{\text{TSS}-\text{RSS}}{\text{TSS}}$.

💡 例子$x$ all near 2 (1.9,2.0,2.1) → slope hard to pin down (big SE); $x$ from 1→100 → easy (small SE). $R^2=0.8$ = height explains 80% of weight variation.$x$ 挤一起难定斜率（SE 大），散开就好定；$R^2=0.8$ = 解释 80% 变化。

Multiple regression & confounding 多元 + 混淆

💡 例子（符号翻转）ice-cream vs drownings looks positive; add temperature and ice-cream's effect drops/flips — heat drives both. Confounding can kill significance OR flip the sign; the past-paper "simple −0.1 but multiple positive" tests the flip.冰淇淋 vs 溺水看着正相关，加"气温"后冰淇淋效应被纠正——天热才是主因。混淆变量既可能让系数变不显著，也可能让符号反转；真题考的是符号反转（遗漏相关变量）。

Smoothing 平滑

💡 例子kNN smoother $k=2$ follows every bump (overfit, high variance); $k=50$ ≈ straight line (over-smooth, high bias).$k=2$ 随波逐流（高方差），$k=50$ 拉成直线（高偏差）。

Week 3 · Density Estimation (MLE / KDE)Density

Distributions & PDF

PMF $\sum p_i=1$; PDF $\int f=1$, can exceed 1.

💡 例子（PDF>1）Uniform on $[0,0.5]$: density $=1/0.5=2>1$, but area $=2\times0.5=1$.$[0,0.5]$ 均匀分布密度 = 2 > 1，但面积仍是 1（密度≠概率）。

MLE

$L(\theta\mid x)=\prod f(x_i\mid\theta)$; max log-likelihood ≡ max likelihood; MLE can be biased.

💡 例子（似然 vs 概率）flip a coin 10× → 7 heads. Probability: "given $p=0.5$, chance of 7 heads?" Likelihood: "given 7 heads, best $p$?" → $\hat p=0.7$.抛 10 次得 7 正：概率是给定 p 问数据；似然是给定数据问 p，MLE 给 0.7。

KDE & bandwidth

$\hat f(x)=\frac{1}{nh}\sum K(\frac{X_i-x}{h})$; kernel non-negative/symmetric/integrates to 1; bandwidth > kernel.

💡 例子（带宽）student heights: tiny $h$ → a spike per person (false peaks, high variance); huge $h$ → one hump hiding the real male/female two peaks (high bias).身高密度：$h$ 太小每人一尖峰（假峰）；太大抹平"男女两峰"（高偏差）。

Bias-variance

$E[(Y-\hat f)^2]=\text{Var}+\text{Bias}^2+\text{Var}(\epsilon)$; KDE Var$=O(1/nh)$, Bias$=O(h)$.

Week 4 · Clustering / PCA / t-SNE / MDSUnsupervised

K-means

Iterate centroids ↔ nearest-assign; WSS↓ with K → elbow; local optimum only; round equal-size clusters.

💡 例子points $\{1,2,9,10\}$, $K=2$ → clusters $\{1,2\}$ & $\{9,10\}$, centroids 1.5 & 9.5. Bad initial centroids can give a wrong local optimum → run several times.{1,2,9,10} 分 2 组 → {1,2} 和 {9,10}；初始不好会卡坏的局部解，要多跑几次。

Hierarchical & PCA

💡 例子（树状图）cities: Sydney–Melbourne merge first (low height = similar), Perth joins much later (high = far).悉尼–墨尔本先合并（矮=像），珀斯很晚才并入（高=远）。

PCA: maximise variance, orthogonal scores, not scale invariant → standardise, unique/linear.

💡 例子（为何标准化）height (cm, var ~100) vs weight (kg, var ~25): without standardising, PC1 ≈ "height" and weight is ignored. Standardise so both count.身高(cm)体重(kg)：不标准化主成分几乎只看身高 → 必须标准化。

t-SNE / MDS

💡 例子t-SNE on handwritten digits → 10 clean blobs (one per digit); re-run → looks different but blobs remain (probabilistic). MDS: give only pairwise city distances → it rebuilds a 2D map (rotatable).t-SNE 把手写数字画成 10 个团（每次样子不同）；MDS 只用城市两两距离就能还原地图（可旋转）。

Traps

PCA linear/unique; t-SNE nonlinear/probabilistic; K-means local optimum, no dendrogram.

Week 5 · Classification 分类Classification

Logistic regression

$p=\frac{1}{1+e^{-X\beta}}$; link = logit, sigmoid is its inverse; linear boundary.

💡 例子tumour size → P(malignant): tiny tumour $p≈0$, huge $p≈1$, boundary $p=0.5$. Linear regression could give $\hat y=1.3$ (meaningless) — that's why we use logistic.肿瘤越大恶性概率越接近 1；线性回归会算出 1.3 没意义，所以用 logistic。

LDA / kNN

💡 例子（LDA）benign ~2cm, malignant ~5cm (same spread) → boundary ≈ 3.5cm → a 4cm tumour = malignant.良性≈2cm、恶性≈5cm → 边界≈3.5cm → 4cm 判恶性。

💡 例子（kNN）new point's 3 neighbours = 2 red, 1 blue → predict red ($P=2/3$).最近 3 个邻居 2 红 1 蓝 → 判红。

SVM

Maximal margin; soft margin with a slack budget $\sum_i\epsilon_i\le C$ (total, not per-point); kernel → nonlinear boundary.

💡 例子（核）one class is a ring around the other — no straight line works; a kernel lifts the points so a plane separates them → a circular boundary in 2D.一类是另一类外面的圆环，直线分不开；核升维后用平面分 → 原图里是圆形边界。

Week 6 · CV / Metrics / ROC 评价Evaluation

Train/test & CV

💡 例子memorising past-paper answers → 100% on those (training) but fail the real exam (test): training error underestimates test error. 5-fold CV: 100 samples → 5 folds of 20, train on 80 test on 20, rotate 5×, average.背答案在原卷 100% 但真考挂科 → 训练误差低估测试。5 折 CV：留 20 验证、80 训练、轮 5 次取平均。

Data leakage 数据泄漏

选特征/调参/插补放 CV 内、只用训练折；嵌套 CV 外估误差内选模型。

💡 例子picking "best" features using the whole dataset before CV = peeking at the exam answers before the mock → fake-high score.先用全数据挑特征 = 考前偷看答案再自测，分数虚高。

Metrics & ROC

Metric	Formula
Recall/Sensitivity	$\frac{TP}{TP+FN}$
Precision	$\frac{TP}{TP+FP}$
Specificity	$\frac{TN}{TN+FP}$
$F_1$	$\frac{2PR}{P+R}$ ($\le\max(P,R)$)

💡 例子（混淆矩阵）100 emails, 30 spam: catch 25 (TP), miss 5 (FN), wrongly flag 10 (FP) → Precision $25/35≈0.71$, Recall $25/30≈0.83$, $F_1≈0.77$.100 封含 30 垃圾：抓 25 漏 5 误判 10 → 精确率 0.71、召回 0.83、F1 0.77。

💡 例子（accuracy 陷阱）99% emails normal → "all normal" gives 99% accuracy but catches 0 spam (recall 0).99% 正常 → 全判正常也有 99% 准确率但召回 0。

Week 7 · Bootstrap / Missing / ImbalancePractical

Bootstrap

Resample with replacement, B times; SE = sd of estimates; ≈2/3 used, 1/3 OOB.

💡 例子data $[3,5,8]$ → resamples $[5,5,8]$ (mean 6), $[3,8,8]$ (6.3), $[3,3,5]$ (3.7)… the spread of these means = SE of the mean.[3,5,8] 有放回重采样得 [5,5,8]、[3,8,8]…，这些均值的离散度 = 均值的标准误。

Missing data 缺失

💡 例子（三种缺失）income field: forms randomly smudged = MCAR; young people skip it (we know their age) = MAR; high earners refuse = MNAR.收入栏：随机污损 MCAR；年轻人懒填(但知年龄) MAR；高收入故意不填 MNAR。

Imbalance 不平衡

💡 例子1% fraud → "always not-fraud" = 99% accuracy, 0% recall → useless. Use F1/AUC/Kappa, or SMOTE the training fold.1% 欺诈，全判非欺诈准确率 99% 但召回 0 → 用 F1/AUC/Kappa 或 SMOTE。

Week 8 · Feature Selection / Ridge / LassoRegularisation

Subset selection

Best subset $2^p$; forward works for $np$; $C_p$/BIC/Adj $R^2$.

💡 例子（前向逐步）3 predictors: try X1,X2,X3 alone → keep best (X2) → try X2+X1, X2+X3 → keep best → add last. Greedy.先单独试留最好（X2），再一次加一个。

Ridge vs Lasso

	Ridge ($\ell_2$)	Lasso ($\ell_1$)
Selection	No (shrink all)	Yes (β→0, sparse)
Geometry	Circle	Diamond (corners→0)

💡 例子（Lasso 选变量）10 predictors, only X3 & X7 matter → Lasso sets the other 8 to exactly 0; Ridge keeps all 10 but shrinks them.10 个变量只有 2 个有用：Lasso 把其余 8 个压成 0，Ridge 全留但缩小。

💡 例子（选 λ）$\lambda=0$ → overfit; $\lambda$ huge → all ≈0 (underfit); CV picks the middle.λ=0 过拟合、λ 很大欠拟合，CV 选中间。先标准化。

Week 9 · Trees & Ensembles 树与集成Trees

Trees

Recursive binary splitting (greedy); prune with $\alpha$ (CV); Gini/Entropy = purity.

💡 例子（读树）baseball salary: if Years < 4.5 → low; else if Hits ≥ 117.5 → high, else medium. Read top-down like a flowchart. Gini: a 10/0 node = 0 (pure), a 5/5 node = 0.5 (impure).年限<4.5→低薪；否则打击数≥117.5→高薪。Gini：10/0 节点=0 最纯，5/5=0.5 最不纯。

Bagging / RF / Boosting

💡 例子（类比）Bagging = ask 100 friends & vote (vs 1 noisy friend). Random forest = stop one loud friend dominating by letting only a random subset speak each time (decorrelate). Boosting = redo the questions you got wrong, round after round (fit residuals).Bagging=问 100 个朋友投票；RF=每次只让部分朋友发言防"大嗓门"主导；Boosting=重点练做错的题（拟合残差）。

Bagging vs Boosting

Bagging parallel/deep/can't overfit; Boosting sequential/shallow/can overfit.

Week 10 · Monte Carlo 蒙特卡洛Simulation

$E[g(X)]\approx\frac1N\sum g(X_i)$; inverse transform $X=F^{-1}(U)$; acceptance-rejection $f\le Mg$; $\text{SE}=\sigma/\sqrt N$.

💡 例子（估 π）throw 1000 darts at a 1×1 square; 785 land inside the quarter-circle → $\hat\pi=4\times785/1000=3.14$.扔 1000 个点，785 个落入 1/4 圆 → π̂ = 4×785/1000 = 3.14。

💡 例子（接受-拒绝 / 收集问题 / 精度）sample a weird curve = throw points under an easy "umbrella" and keep those under the real curve. Collect 20 blind boxes ≈ 72 buys on average ($20\sum1/i$). To halve MC error you need 4× samples.接受-拒绝=伞下扔点只留落在真曲线下的；集齐 20 盲盒平均买 72 个；误差减半需 4 倍样本。

Week 11 · MCMCMCMC

Markov chain, correlated samples; irreducible+aperiodic → unique stationary $\pi=\pi P$; MH $\alpha=\min(\frac{f(Y)q(X_t|Y)}{f(X_t)q(Y|X_t)},1)$, symmetric → $\min(f(Y)/f(X_t),1)$.

💡 例子（政客直觉）visit 4 town halls, spend time ∝ voters: if the proposed hall has more voters always go; if fewer, go with prob = ratio. Over time, time spent ∝ voters. That's Metropolis-Hastings.政客访 4 镇按选民停留：新镇人多就去，人少按比例概率去 → 长期停留正比于选民。这就是 MH。

💡 例子（burn-in / 接受率）throw away the early "warm-up" wandering steps (burn-in). Tiny steps = always accept but crawl; huge steps = often rejected → aim 20–50% acceptance.丢掉开头热身的游走（burn-in）；步子太小爬得慢、太大老被拒 → 接受率 20–50% 最好。

💡 例子（马尔可夫性）tomorrow's weather depends only on today's, not the whole week.明天天气只看今天，不看整周历史。

Reading Plots 识图速查Plots

Plot 图	Read 怎么读	💡 example 例子
Histogram 直方图	bar height = count in bin	tall bar at 20–25 = most are aged 20–25
Boxplot 箱线图	line=median, box=IQR, dots=outliers	a dot far up = an unusually tall person
Scatter 散点	up-right = positive	height vs weight trends up-right
KDE	area=prob	two humps = male/female heights
Elbow 手肘	corner = chosen K	sharp bend at K=3 → 3 clusters
Dendrogram 树状图	low merge = similar	Sydney–Melbourne low, Perth high
Biplot 双标图	long arrow = influential	"income" & "spending" arrows same direction = correlated
ROC	Sensitivity (TPR) vs FPR (1−Spec); top-left = better; AUC→1 (0.5=random)	curve hugging top-left = strong model

Connections & Traps 串联与易错Must read

3 threads 三大串联

Elbow ×3 (K-means/PCA/MDS); bias-variance everywhere (k/window, h, $\lambda$, depth/α/B); standardise ×3 (PCA/Ridge/Lasso).手肘 ×3；偏差-方差贯穿；标准化 ×3。

Common traps 常见陷阱

PDF can exceed 1 (Uniform[0,0.5]→2); least squares = vertical distance; closed-form coefficients.
Large bandwidth → high bias; K-means local optimum, no dendrogram.
PCA linear/unique; t-SNE nonlinear/probabilistic.
Forward $np$; Lasso zeros, Ridge doesn't.
Boosting can overfit; Bagging/RF don't (with more trees).
Logistic link = logit not sigmoid; ROC = Sensitivity vs FPR(1−Spec), top-left = better, AUC→1; $F_1\le\max(P,R)$; leakage → inside CV.

Exam Question Map 真题题型 (26S1)Past paper

Format 结构

12 MCQ (exactly 2 correct each, +1/correct, max 2, 3 picks = 0) + 4 short-answer (36 marks). 真题练习.

MCQ: feature selection · PDF/MLE · linear regression · bootstrap · PCA/t-SNE · trees · classification · F1/ROC · ensembles.
Regression group Q10–12: 100-week sales; add a season factor; explain the simple-vs-multiple sign flip (the ice-cream/drowning idea).
Short-answer: MLE vs KDE / tree vs RF / LDA vs kNN; Lasso CV U-shape; AdaBoost; specificity vs sensitivity; confusion matrix; data leakage; Monte Carlo (coupon / bus arrival).