I have forgotten

• https://me.yahoo.com

# Least Squares

Curve fitting, least squares, optimization
View version details

## Introduction

Consider a series of given data points in -dimensional space, , where is a vector, for all . Define a function through

where and are given functions, for all . Also, define the function as the sum of squared residuals:

The method of least squares fitting refers to finding the parameters which are the solution to the following unconstrained optimization problem:

After solving this problem, the function which provides the best fit, in the least-squares sense, is given by:

This general framework allows us to classify the different types of regression, as follows:

• When the method is known as univariate regression, while if we have multivariate regression.
• Provided that all the functions are linear, the method is called linear regression, otherwise it is known as nonlinear regression.
• Also, based on the type of the functions we may have polynomial regression, regression by orthogonal polynomials, and so on.

## Solving The Problem

Since the sum of residuals function is convex on its entire domain, a necessary and sufficient condition for a tuple of parameters to be a solution to the above optimization problem is that

which can also be written as the system of (possibly nonlinear) equations:

Let us fix some value of and calculate . We have:

Therefore, the system of equations whose solution is the tuple of optimal parameters can explicitly be written as:

In the case of linear regression in small dimensions, it is possible to solve this system directly, using algebra. Generally, however, we should use numerical root-finding techniques to find the optimal parameters.
Lucian Bentea (September 2008)