Final Exam Study Guide -- CS534 -- Spring 2005
Topics to know for the final exam:
Topics from the first half of the course:
- Details of specific hypothesis spaces and learning algorithms.
- You should be able to look at a toy problem in two dimensions
and predict how each algorithm will behave (as on the midterm).
- Linear threshold units (what can they express? What can't they
express?) Ways of fitting LTUs via: Perceptrons, Logistic
regression, multi-variate gaussians, linear programming and
quadratic programming (for SVMs)
- Decision trees (including splitting rules and methods of
handling missing values)
- Neural networks (including both squared error and softmax
error, initialization of the weights)
- Nearest Neighbor (curse of dimensionality)
- Support Vector Machines (including geometric and functional
margins, kernel methods, soft margin formulation, margin slack
vector, slack vector bound on test set error)
- Naive Bayes (How to compute it for discrete and continuous
attributes; Laplace corrections)
- Gradient descent search. How to design a gradient descent search
algorithm. Difference between batch and incremental (stochastic)
Topics from the second half of the course:
- Triple tradeoff (sample size, hypothesis complexity, error rate)
and the problem of overfitting.
- Bias/variance/noise decomposition. You should be able to analyze
new and existing algorithms and predict whether they will have high
bias, high variance, or both. You should know how various ensemble
methods affect bias and variance.
- Methods for controlling overfitting
- Penalty methods: pessimistic tree pruning, cost-complexity
pruning, and MDL pruning, SVM margin maximization, weight decay
and weight elimination.
- Holdout methods: cross-validation, validation sets, early
stopping, reduce-error pruning.
- Ensemble methods: bagging, boosting, bayesian model averaging.
- Methods for Evaluating Classifiers and Learning Algorithms
- ROC curves
- Rejection curves
- Precision/recall curves
- McNemar's test and the 5x2cv F test
- Sequential Supervised Learning Problems
- Difference between sequential problems and standard
supervised learning; sliding window methods for converting one
into the other.
- Difference between what the Viterbi algorithm computes and
what the Forward-Backward algorithm computes.
- What is the form of the hypothesis computed by the averaged
perceptron, conditional random field, and hidden markov model?
I won't ask for the details of the fitting procedure or the
details of the computations performed by Viterbi and Forward-Backward.
- Miscellaneous topics:
- Methodology for making classifier design decisions (e.g.,
using a development subset of the training data)