I have forgotten

• http://facebook.com/
• https://www.google.com/accounts/o8/id
• https://me.yahoo.com

Least Squares

Curve fitting, least squares, optimization View version details

Introduction

Consider a series of given data points in -dimensional space, , where is a vector, for all . Define a function through where and are given functions, for all . Also, define the function as the sum of squared residuals: The method of least squares fitting refers to finding the parameters which are the solution to the following unconstrained optimization problem: After solving this problem, the function which provides the best fit, in the least-squares sense, is given by: This general framework allows us to classify the different types of regression, as follows:

• When the method is known as univariate regression, while if we have multivariate regression.
• Provided that all the functions are linear, the method is called linear regression, otherwise it is known as nonlinear regression.
• Also, based on the type of the functions we may have polynomial regression, regression by orthogonal polynomials, and so on.

Solving The Problem

Since the sum of residuals function is convex on its entire domain, a necessary and sufficient condition for a tuple of parameters to be a solution to the above optimization problem is that which can also be written as the system of (possibly nonlinear) equations: Let us fix some value of and calculate . We have: Therefore, the system of equations whose solution is the tuple of optimal parameters can explicitly be written as: In the case of linear regression in small dimensions, it is possible to solve this system directly, using algebra. Generally, however, we should use numerical root-finding techniques to find the optimal parameters.
Lucian Bentea (September 2008)