yawiki.org entry for Supervised learning

	Navigation Home Recent Most Active Popular Blog Credits RSS
	Interaction Register Statistics
	Help Suggestions Contact Us How to Edit Help

[Edit]

Supervised learning is a machine learning technique for creating a function from training data. The training data consist of pairs of input objects (typically vectors), and desired outputs. The output of the functioncan be a continuous value (called regression), or can predict a class label of the input object (called classification). The task of the supervised learner is to predict the value of the function for any valid input object after having seen a number of training examples (i.e. pairs of input and target output). To achieve this, the learner has to generalize from the presented data to unseen situations in a "reasonable" way (see inductive bias).
(Compare with unsupervised learning.) The parallel task in human and animal psychology is often refered to as concept learning.

Supervised learning can generate models of two types. Most commonly, supervised learning generates a global model that maps input objects to desired outputs. In some cases, however, the map is implemented as a set of local models (such as in case-based reasoning or the nearest neighbor algorithm).

In order to solve a given problem of supervised learning (e.g. learning to recognize handwriting) one has to consider various steps:

Determine the type of training examples. Before doing anything else, the engineer should decide what kind of data is to be used as an example. For instance, this might be a single handwritten character, an entire handwritten word, or an entire line of handwriting.

Gathering a training set. The training set needs to be characteristic of the real-world use of the function. Thus, a set of input objects is gathered and corresponding outputs are also gathered, either from human experts or from measurements.

Determine the input feature representation of the learned function. The accuracy of the learned function depends strongly on how the input object is represented. Typically, the input object is transformed into a feature vector, which contains a number of features that are descriptive of the object. The number of features should not be too large, because of the curse of dimensionality; but should be large enough to accurately predict the output.

Determine the structure of the learned function and corresponding learning algorithm. For example, the engineer may choose to use artificial neural networks or decision trees.

Complete the design. The engineer then runs the learning algorithm on the gathered training set. Parameters of the learning algorithm may be adjusted by optimizing performance on a subset (called a validation set) of the training set, or via cross-validation. After parameter adjustment and learning, the performance of the algorithm may be measured on a test set that is separate from the training set.

    Supervised learning
        Empirical risk minimization
        Approaches and algorithms
        Applications
        General issues

top

Empirical risk minimization

The goal of supervised learning of a global model is to find a function g, given a set of points of the form (x, g(x)).

It is assumed that the set of points for which the behavior of g is known is an i.i.d. sample drawn according to an unknown probability distribution p of a larger, possibly infinite, population. Furthermore, one assumes the existence of a task-specific loss function L of type

$L: Y imes Y o Bbb^+$

where Y is the codomain of g and L maps into the nonnegative real numbers (further restrictions may be placed on L). The quantity L(z, y) is the loss incurred by predicting z as the value of g at a given point when the true value is y.

The risk associated with a function f is then defined as the expectation of the loss function, as follows:

$R(f) = sum_i L(f(x_i), g(x_i))$
p(x_i)

if the probability distribution p is discrete (the analogous continuous case employs a definite integral and a probability density function).

The goal is now to find a function f^{among a fixed subclass of functions for which the risk R(f^{) is minimal.}

However, since the behavior of g is generally only known for a finite sequence of points (x₁, y₁), ..., (x_n, y_n), one can only approximate the true risk, for example with the empirical risk:

$ilde_n(f) = rac sum_^n L(f(x_i), y_i)$

Selecting the function f^{that minimizes the empirical risk is known as the principle of empirical risk minimization. Statistical learning theory investigates under what conditions empirical risk minimization is admissible and how good the approximations can be expected to be.

top
Approaches and algorithms

Analytical learning
Artificial neural network
Backpropagation
Boosting
Bayesian statistics
Case-based reasoning
Decision tree learning
Inductive logic programming
Gaussian process regression
Learning automata theory
Minimum message length (decision trees, decision graphs, etc.)
Naive bayes classifier
nearest neighbour algorithm
Probably approximately correct learning (PAC) learning
Symbolic machine learning algorithms
Subsymbolic machine learning algorithms
Support vector machines
Random Forests

top
Applications

Bioinformatics
Cheminformatics
Quantitative structure-activity relationship

Handwriting recognition
Information retrieval
Object recognition in computer vision
Optical character recognition
Spam detection
Pattern recognition
Speech recognition

top
General issues

Computational learning theory
Inductive bias
Overfitting
Version spaces}}

	Source Privacy License Download Contact Us Atlas Scientus.org Dictionary (Yet Another Wiki) RC : 1.39 This article is licensed under the GNU Free Documentation License [copyleft]. It uses material from the Wikipedia article "Supervised learning". link