CS534 Final Project

Due: Friday June 6th, 11:59PM (no late submission)

The goal of this class project is to give you an opportunity to work with a "realworld" machine learning problem and explore various aspects that are involved in machine learning application and/or research. This project is intended to be more open-ended than your previous experimental assignments and give you more hands-on exprience with real-world machine learning problems and learning algorithms. You are encouraged to combine machine learning with problems in your own research area. Your class project must be about new things you have done this term, you can't use results you have developed prior to this class.

What you need to do

Find a partner to work on this project. (Two person teams are encouraged, though you may work alone. No three person teams please.)
Choose your application domain and learning problem within it. Turn in a project proposal (1 page maximum) by May 12th. Each team is required to meet with the instructor at least once to discuss the project before turning in the proposal. You can come to the office hour or email me to schedule a different time for meeting.
As a guideline, you will need to go through the following questions and make your decisions on each one.

How to formulate the learning problem. Is this a classification problem or a clustering problem? Are the data iid (independent and identically-distributed) or are they sequential, spatial?
Feature design. How should the "raw" data be transformed into proper features (inputs) so that the data is suitable for machine learning? Should the data be aggregated in some way? Should the data be transformed so that it has a more gaussian distribution? Can we apply dimension reduction / feature subset selection to improve learning performance? If so, how can we go about it?
Algorithm choice. What learning algorithms would be appropriate for this problem? Factors to consider: data set size, noise level, continuous versus discrete features, missing values, supervised vs unsupervsed vs semi-supervsed.
Algorithm tuning. If the algorithm has user-set parameters, what strategy should be used for setting them?
Overfitting Avoidance. Is there a risk of overfitting? If so, what overfitting avoidance methods should be applied? How should they be tuned?
Performance criterion. How should performance be measured? Error rate? Expected misclassification cost? Cross-validation Likelihood? What loss function should be used for training the learning algorithm?

Perform the work, run the experiments!
Turn in a final report (no longer than 8 pages including references, figures and tables). Each team should turn in a single report and please email me your report before the deadline. Your report should precisely describe the following:

The application domain and formulation of your learning task(s).
The data collection process.
A precise description of your approach; What algorithm did you choose and why? what features did you use? Describe any preprocessing that's involved. What software package was used and what was programmed by you? NOTE: no restrictions on using existing software packages and no restrictions on what programming language you use if you decide you need code your own.

Describe your experiments, evaluation results, and any conclusions your draw from this.

The clarity and content of the report will have a primary impact on your grade. Again, the report should not exceed 8 pages (using fewer pages is ok), 11 point font, including figures and tables.

Grading and determining when you have done enough

A project that does a solid job building the base learning system and carefully evaluating and describing it might get 75–80% credit. A project that includes additional pursuit of interesting extensions/alternatives or investigations into important issues (such as overfitting, noise tolerance, feature selection etc.), or achieves very impressive results might get 90–100% credit. Weight will also be given to the interestingness and novelty of the learning task considered.

Be creative! Exploring your own interesting ideas and comparing them with the baseline approaches will receive credit whether they beat the baseline or not.

Some Possible Learning Problems

Computer Vision Recognition Tasks

Optical Character Recognition
Face Recognition Tasks
Scene recognition (e.g. house vs. no house)
Object recognition in aerial/satellite images
Visual dictionary learning
etc

Audio Recognition Tasks

Speaker identity
Speaker sentiment
Music genre
Bird species recognition by songs (if you are interested in this, speak to the instructor)
etc

Text Classification and Clustering

Spam
Email Folder Predictor
Newgroup document classifier
Sentiment
Author classifier (i.e. take latex files from different authors and try to classify according to author)

Bio-informatics

gene clustering/expression profile analysis
gene sequence analysis
etc