MyOOPS開放式課程

# 9.520 2003春季課程：統計學習理論及應用(Statistical Learning Theory and Applications, Spring 2003)

Designing and building a system that will function the same way as a human visual system, but without getting bored, and with a greater degree of accuracy. (Image courtesy of Poggio Laboratory, MIT Department of Brain and Cognitive Sciences.)

## 課程重點

Support vector machines have proven to be very useful in classification networks. These SVMs are now being used by drivers for pedestrian avoidance. This is one of the first truly universal applications of this technology.

This course is for upper-level graduate students who are planning careers in computational neuroscience. The assignments focus on some of the functions needed to make problem-solving more efficient for computer systems. The project topics students can choose from are based on unsolved problems in the field today. By the conclusion of this course, students should be able to solve one or two of these problems, and should be able to frame an approach to the rest of them.

## 課程描述

Focuses on the problem of supervised learning from the perspective of modern statistical learning theory starting with the theory of multivariate function approximation from sparse data. Develops basic tools such as Regularization including Support Vector Machines for regression and classification. Derives generalization bounds using both stability and VC theory. Discusses topics such as boosting and feature selection. Examines applications in several areas: computer vision, computer graphics, text classification and bioinformatics. Final projects and hands-on applications and exercises are planned, paralleling the rapidly increasing practical uses of the techniques described in the subject.

### 師資

Tomaso Poggio 教授
Sayan Mukherjee 博士
Ryan Rifkin 博士
Alex Rakhlin

### 聲明

18.02，9.641，6.893〔課程〕或教師許可。實際上，較高水準的數學技巧是必需的。熟悉機率和泛函分析有助學習本課程。我們儘量把數學先修部份減至最低，但是介紹複雜內容時步伐較快。

MED和正則化的聯繫

MATLAB®是MathWorks公司的商標。

Course Description
We introduce and motivate the main theme of the course, setting the problem of learning from examples as the problem of approximating a multivariate function from sparse data. We present an overview of the theoretical part of the course and sketch the connection between classical Regularization Theory and its algorithms -- including Support Vector Machines -- and Learning Theory, the two cornerstones of the course. We mention theoretical developments during the last few months that provide a new perspective on the foundations of the theory. We briefly describe several different applications ranging from vision to computer graphics, to finance and neuroscience.
Prerequisites
18.02, 9.641, 6.893 or permission of instructor. In practice, a substantial level of mathematical maturity is necessary. Familiarity with probability and functional analysis will be very helpful. We try to keep the mathematical prerequisites to a minimum, but we will introduce complicated material at a fast pace.
There will be two problem sets, a MATLAB® assignment, and a final project. To receive credit, you must attend regularly, and put in effort on all problem sets and the project.
Problem Sets

See the assignments page for the two problem sets.

Projects

Some of the most promising projects:

Hypothesis testing with small sets
Connection between MED and regularization
Feature selection for SVMs theory and experiments
Bayes classification rule and SVMs
IOHMMs evaluation of HMMs for classification vs. direct classification
Reusing the test set datamining bounds
Large-scale nonlinear least square regularization
Viewbased classification
Local vs. global classifiers experiments and theory
RKHS invariance to measure historical math
Concentration experiments (dot product vs. square distance)
Decorrelating classifiers: experiments about generalization using a tree of stumps
Kernel synthesis and selection
Bayesian interpretation of regularization and in particular of SVMs
History of induction from Kant to Popper and current state
Bayesian Priorhood

Resources

The Center for Biological and Computational Learning (CBCL) at MIT was founded with the belief that learning is at the very core of the problem of intelligence, both biological and artificial, and is the gateway to understanding how the human brain works and to making intelligent machines. CBCL studies the problem of learning within a multidisciplinary approach. Its main goal is to nurture serious research on the mathematics, the engineering and the neuroscience of learning. CBCL is based in the Department of Brain and Cognitive Sciences at MIT and is associated with the McGovern Institute for Brain Research and with the Artificial Intelligence Laboratory at MIT.

MATLAB® is a trademark of The MathWorks, Inc.

There are three sessions, two Math Camps and an extra topic, at the bottom of the calendar. These will be given when students require the background needed to understand the next series of lectures and problems.
課       課程單元

1       課程概覽
The Course at a Glance

2       學習問題觀點
The Learning Problem in Perspective

3       正則化和再生核希爾伯特空間
Regularization and Reproducing Kernel Hilbert Spaces

4       回歸和最小二乘方分類
Regression and Least-Squares Classification

5       支援向量機分類
Support Vector Machines for Classification

6       泛化邊界；穩定性簡介
Generalization Bounds, Intro to Stability

7       Tikhonov正則化的穩定性
Stability of Tikhonov Regularization

8       函數類一致性和一致收斂
Consistency and Uniform Convergence over Function Classes

9       一致收斂的必要和充分條件
Necessary and Sufficient Conditions for Uniform Convergence

10       裝袋和增強
Bagging and Boosting

11       電腦視覺，物體檢測
Computer Vision, Object Detection

12       整理零散
Loose Ends

13       逼近論
Approximation Theory

14       再生核希爾伯特空間，Mercer定理，無界領域，小波
RKHS, Mercer Thm, Unbounded Domains, Frames and Wavelets

15       生物資訊學
Bioinformatics

16       文本
Text

17       正則化網路
Regularization Networks

18       視頻的形變模型
Morphable Models for Video

19       留一逼近
Leave-One-Out Approximations

20       貝葉斯解釋
Bayesian Interpretations

21       多類分類
Multiclass Classification

22       穩定性和Glivenko-Cantelli類
Stability and Glivenko-Cantelli Classes

24       專題報告
Project Presentations

25       專題報告
Project Presentations

數學營
Math
Camp
Lagrange Multipliers/Convex Optimization

數學營
Math
Camp
Functional Analysis

額外題目
Extra
Topic
支援向量機基本原則
SVM Rules of Thumb

There is no textbook for this course. All the required information will be presented in the slides associated with each class. The books and articles listed below are useful general reference reading, especially from the theoretical viewpoint. Additional readings are listed in the lecture note PDF files.

Cristianini, N.和J. Shawe-Taylor. 《支持向量機導論》 Cambridge, 2000
Cristianini, N., and J. Shawe-Taylor. Introduction To Support Vector Machines. Cambridge, 2000.

Cucker, F.和S. Smale.〈關於學習的數學基礎〉 《美國數學學會通訊》 2002
Cucker, F., and S. Smale. "On The Mathematical Foundations of Learning." Bulletin of the American Mathematical Society. 2002.

Devroye, L., L. Gyorfi和G. Lugosi.《模式識別的機率理論》Springer, 1997.
Devroye, L., L. Gyorfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, 1997.

Evgeniou, T., M. Pontil和T. Poggio.〈正則化網路和支援向量機〉《計算數學的進展》2000.
Evgeniou, T., M. Pontil, and T. Poggio. "Regularization Networks and Support Vector Machines." Advances in Computational Mathematics. 2000.

Poggio, T.和S. Smale.〈學習的數學：處理數據〉《美國數學協會公告》2003
Poggio, T., and S. Smale. "The Mathematics of Learning: Dealing with Data." Notices of the AMS. 2003.

Vapnik, V. N.《統計學習理論的本質》Springer,1995
Vapnik, V. N. The Nature of Statistical Learning Theory. Springer, 1995.

———.《統計學習理論》Wiley，1998
———. Statistical Learning Theory. Wiley, 1998.

MED和正則化的聯繫

These are examples of problem sets and a project for the class.
Problem Sets

Problem Set 1: Kernel Hilbert Spaces (PDF)
Problem Set 2: RBF Interpolation Schemes (PDF)

Project (PDF)

For the final project, students select from one of the following suggested topics, and solve the problem that is described. If students prefer, they can bring their own project ideas to the professor or TAs for approval.

Topics:

Hypothesis testing with small sets
Connection between MED and regularization
Feature selection for SVMs theory and experiments
Bayes classification rule and SVMs
IOHMMs evaluation of HMMs for classification vs. direct classification
Reusing the test set datamining bounds
Large-scale nonlinear least square regularization
Viewbased classification
Local vs. global classifiers experiments and theory
RKHS invariance to measure historical math
Concentration experiments (dot product vs. square distance)
Decorrelating classifiers: experiments about generalization using a tree of stumps
Kernel synthesis and selection
Bayesian interpretation of regularization and in particular of SVMs
History of induction from Kant to Popper and current state
Bayesian Priorhood

 回應 告訴我們這些 .ZIP檔如何協助你進行教導與學習。 .ZIP檔案中的課程內容與「麻省理工開放式課程」所出版的材料一樣，必需依照創作共享理念授權同意書規範。

## 課程下載

9-520Spring-2003.zip

## 尋找並使用課程內容

.ZIP檔案中的課程內容與「麻省理工開放式課程」所出版的材料一樣，必需依照創作共享理念授權同意書規範。

## 常見問答集

 留下您對本課程的評論 標題： 您目前為非會員，留言名稱將顯示「匿名非會員」只能進行20字留言 留言內容： 驗證碼請輸入0 + 7 = 標籤 現有標籤：1 新增標籤：

Anonymous, 2015-07-22 10:21:13

Anonymous, 2012-06-19 17:49:10