|
Radial basis functions (RBF) are a means for interpolation in a stream of data. They differ from statistical approaches in that approximations must be performed on streams of data rather than on complete data sets. RBFs use supervised learning and sometimes unsupervised learning to minimize approximation error in a stream of data. They are used in function approximation, time series prediction, and control. Overview Radial basis functions (RBF) are powerful techniques for interpolation in multidimensional space. An RBF is a function which has built into it a distance criterion with respect to a center. Such functions can be used very efficiently for interpolation and for smoothing of data. Radial basis functions have been applied in the area of neural networks where they are used as a replacement for the sigmoidal transfer function. Such networks have 3 layers, the input layer, the hidden layer with the RBF non-linearity and a linear output layer. The most popular choice for the non-linearity is the Gaussian. RBF networks have the advantage of not being locked into local minima as do the feedforward networks. The problem The problem solved by RBF's is the development of an analytic approximation for the input/output mappings described by a deterministic, noisy, or stochastic data stream where is the input vector at time t, is the output at time t, and is the dimension of the input space. In the deterministic case the data is drawn from the set . In the noisy case data is drawn from the set where is a partially known random process. In the stochastic case, data is drawn from the joint probability distribution ight ) . Architecture RBF architectures come in two forms, normalized and unnormalized. The forms can be expanded into a superposition of local linear models. Unnormalized The unnormalized radial basis function architecture, , is ho ig ( left Vert mathbf - mathbf_i ight Vert ig ) where is the approximation to the data, , known as a "radial basis function," is a local function of the distance between the input vector and a "basis function center" , and are weights to be determined by data. Typically the distance is taken to be the Euclidean distance and the basis function is taken to be Gaussian ho ig ( left Vert mathbf - mathbf_i ight Vert ig ) propto exp left -eta left Vert mathbf{x} - mathbf{c}_i ight Vert ^2 ight . The weights , , and are determined in a manner that optimizes the fit between and the data. Normalized architecture The normalized RBF architecture is ight Vert ig ) where ight Vert ig ) equiv rac is known as a "normalized radial basis function." Theoretical motivation for normalization There is theoretical justification for this architecture in the case of stochastic data flow. Assume a Stochastic kernel approximation for the joint probability density ight ) = sum_^N , ho ig ( left Vert mathbf - mathbf_i ight Vert ig ) , sigma ig ( left vert y - e_i ight vert ig ) where the weights and are exemplars from the data and we require the kernels to be normalized ho ig ( left Vert mathbf - mathbf_i ight Vert ig ) , d^nmathbf =1 and ight vert ig ) , dy =1. The probability densities in the input and output spaces are ight ) = int P left ( mathbf land y ight ) , dy = sum_^N , ho ig ( left Vert mathbf - mathbf_i ight Vert ig ) and ight ) = int P left ( mathbf land y ight ) , d^n mathbf = sum_^N , sigma ig ( left vert y - e_i ight vert ig ) The expectation of y given an input is ight ) equiv Eleft ( y mid mathbf ight ) = int y , Pleft ( y mid mathbf ight ) dy where ight ) is the conditional probability of y given . The conditional probability is related to the joint probability through Bayes theorem ight ) = rac which yields ight ) = int y , rac , dy . This becomes ight ) = rac = sum_^N a_i u ig ( left Vert mathbf - mathbf_i ight Vert ig ) when the integrations are performed. Local linear models It is sometimes convenient to expand the architecture to include local linear models. In that case the architectures become, to first order, ight ) = sum_^N left ( a_i + mathbf_i cdot left ( mathbf - mathbf_i ight ) ight ) ho ig ( left Vert mathbf - mathbf_i ight Vert ig ) and ight ) = sum_^N left ( a_i + mathbf_i cdot left ( mathbf - mathbf_i ight ) ight )u ig ( left Vert mathbf - mathbf_i ight Vert ig ) in the unnormalized and normalized cases, respectively. Here are weights to be determined. Higher order linear terms are also possible. This result can be written ight ) = sum_^ sum_^n e_ v_ ig ( mathbf - mathbf_i ig ) where and ho ig ( left Vert mathbf - mathbf_i ight Vert ig ) , & mbox i in 1,N \ left ( x_ - c_ ight ) ho ig ( left Vert mathbf - mathbf_i ight Vert ig ) , & mboxi in N+1,2N end in the unnormalized case and ight Vert ig ) , & mbox i in 1,N \ left ( x_ - c_ ight ) u ig ( left Vert mathbf - mathbf_i ight Vert ig ) , & mboxi in N+1,2N end in the normalized case. Here is a Kronecker delta function defined as e j end . Objective functions The weights, which we signify by , in the RBF architecture are found through optimization of an objective function. The most common objective function is the least squares function where . We have explicitly included the dependence on the weights. Minimization of the least squares objective function by optimal choice of weights optimizes accuracy of fit. There are occasions in which multiple objectives, such as smoothness as well as accuracy, must be optimized. In that case it is useful to optimize a regularized objective function such as where and where optimization of S maximizes smoothness and is known as a regularization parameter. Training Choosing weights that optimize the objective function is known as "training" or "learning." Training is performed at each time step as data streams in. Gradient descent training of the linear weights The simplest training algorithm is Gradient descent. In gradient descent training the weights are adjusted at each time step by moving them in a direction opposite from the gradient of the objective function u rac H_t(mathbf) where is a "learning parameter." For the case of training the linear weights, , the algorithm becomes u ig y(t) - varphi ig ( mathbf{x}(t), mathbf{w} ig ) ig ho ig ( left Vert mathbf(t) - mathbf_i ight Vert ig ) in the unnormalized case and u ig y(t) - varphi ig ( mathbf{x}(t), mathbf{w} ig ) ig u ig ( left Vert mathbf(t) - mathbf_i ight Vert ig ) in the normalized case. For local-linear-architectures gradient-descent training is u ig y(t) - varphi ig ( mathbf{x}(t), mathbf{w} ig ) ig v_ ig ( mathbf(t) - mathbf_i ig ) Projection operator training of the linear weights For the case of training the linear weights, and , the algorithm becomes u ig y(t) - varphi ig ( mathbf{x}(t), mathbf{w} ig ) ig rac in the unnormalized case and u ig y(t) - varphi ig ( mathbf{x}(t), mathbf{w} ig ) ig rac in the normalized case and u ig y(t) - varphi ig ( mathbf{x}(t), mathbf{w} ig ) ig rac in the local-linear case. For one basis function, projection operator training reduces to Newton's method. Training the basis function centers Basis function centers can be chosen from exemplars of the input data or they can be trained on the input data using self-organizing maps or unsupervised learning. Logistic map The basic properties of radial basis functions can be illustrated with a simple mathematical map, the logistic map, which maps the unit interval onto itself. It can be used to generate a convenient prototype data stream. The logistic map can be used to explore function approximation, time series prediction, and control theory. The map originated from the field of population dynamics and became the prototype chaotic time series. The map, in the fully chaotic regime, is given by where t is a time index. The value of x at time t+1 is a parabolic function of x at time t. This equation represents the underlying geometry of the chaotic time series generated by the logistic map. Generation of the time series from this equation is the forward problem. The examples here illustrate the inverse problem; identification of the the underlying dynamics, or fundamental equation, of the logistic map from exemplars of the time series. The goal is to find an estimate for f. Unnormalized radial basis functions The architecture is ho ig ( left Vert mathbf - mathbf_i ight Vert ig ) where ho ig ( left Vert mathbf - mathbf_i ight Vert ig ) = exp left -eta left Vert mathbf{x} - mathbf{c}_i ight Vert ^2 ight = exp left -eta left ( x(t) - c_i ight ) ^2 ight . Since the input is a scalar rather than a vector, the input dimension is one. We choose the number of basis functions as N=5 and the size of the training set to be 100 exemplars generated by the chaotic time series. The weight is taken to be a constant equal to 5. The weights are five exemplars from the time series. The weights are trained with projection operator training: u ig x(t+1) - varphi ig ( mathbf{x}(t), mathbf{w} ig ) ig rac where the learning rate is taken to be 0.3. The training is performed with one pass through the 100 training points. The rms error is 0.15. Normalized radial basis functions The normalized RBF architecture is ight Vert ig ) where ight Vert ig ) equiv rac . Again: ho ig ( left Vert mathbf - mathbf_i ight Vert ig ) = exp left -eta left Vert mathbf{x} - mathbf{c}_i ight Vert ^2 ight = exp left -eta left ( x(t) - c_i ight ) ^2 ight . Again, we choose the number of basis functions as five and the size of the training set to be 100 exemplars generated by the chaotic time series. The weight is taken to be a constant equal to 6. The weights are five exemplars from the time series. The weights are trained with projection operator training: u ig x(t+1) - varphi ig ( mathbf{x}(t), mathbf{w} ig ) ig rac where the learning rate is again taken to be 0.3. The training is performed with one pass through the 100 training points. The rms error on a test set of 100 exemplars is 0.084, smaller than the unnormalized error. Normalization yields accuracy improvement. Typically accuracy with normalized basis functions increases even more over unnormalized functions as input dimensionality increases. Time series prediction Once the underlyting geometry of the time series is estimated as in the previous examples, a prediction for the time series can be made by iteration: . A comparison of the actual and estimated time series is displayed in the figure. The estimated times series starts out at time zero with an exact knowledge of x(0). It then uses the estimate of the dynamics to update the the time series estimate for several time steps. Note that the estimate is accurate for only a few time steps. This is a general characteristic of chaotic time series. This is a property of the sensitive dependence on initial conditions common to chaotic time series. A small initial error is amplified with time. A measure of the divergence of time series with nearly identical initial conditions is known as the Lyapunov exponent. Control of a chaotic time series We assume the output of the logistic map can be manipulated through a control parameter such that . The goal is to choose the control parameter in such a way as to drive the time series to a desired output . This can be done if we choose the control paramer to be where is an approximation to the underlying natural dynamics of the system. The learning algorithm is given by u varepsilon rac where . See also | |||||||
|
| ||||||||
 |
|
| |