Consider a series of
given data points in
is a vector, for all
. Define a function
are given functions, for all
. Also, define the function
as the sum of squared residuals:
The method of least squares fitting
refers to finding the parameters
which are the solution to the following unconstrained optimization problem:
After solving this problem, the function
which provides the best fit, in the least-squares sense, is given by:
This general framework allows us to classify the different types of regression, as follows:
- When the method is known as univariate regression, while if we have multivariate regression.
- Provided that all the functions are linear, the method is called linear regression, otherwise it is known as nonlinear regression.
- Also, based on the type of the functions we may have polynomial regression, regression by orthogonal polynomials, and so on.
Solving The Problem
Since the sum of residuals function
is convex on its entire domain, a necessary and sufficient condition for a tuple of parameters
to be a solution to the above optimization problem is that
which can also be written as the system of (possibly nonlinear) equations:
Let us fix some value of
. We have:
Therefore, the system of equations whose solution is the tuple of optimal parameters
can explicitly be written as:
In the case of linear regression in small dimensions, it is possible to solve this system directly, using algebra. Generally, however, we should use numerical root-finding techniques to find the optimal parameters.