Maths › Data Fitting ›

# Least Squares

Curve fitting, least squares, optimization

## Introduction

Consider a series of given data points in -dimensional space, , where is a vector, for all . Define a function through where and are given functions, for all . Also, define the function as the sum of squared residuals: The method of*least*

*squares*

*fitting*refers to finding the parameters which are the solution to the following unconstrained optimization problem: After solving this problem, the function which provides the best fit, in the least-squares sense, is given by: This general framework allows us to classify the different types of regression, as follows:

- When the method is known as
*univariate**regression*, while if we have*multivariate**regression*. - Provided that all the functions are linear, the method is called
*linear**regression*, otherwise it is known as*nonlinear**regression*. - Also, based on the type of the functions we may have
*polynomial**regression*, regression by*orthogonal**polynomials*, and so on.

## Solving The Problem

Since the sum of residuals function is convex on its entire domain, a necessary and sufficient condition for a tuple of parameters to be a solution to the above optimization problem is that which can also be written as the system of (possibly nonlinear) equations: Let us fix some value of and calculate . We have: Therefore, the system of equations whose solution is the tuple of optimal parameters can explicitly be written as: In the case of linear regression in small dimensions, it is possible to solve this system directly, using algebra. Generally, however, we should use numerical root-finding techniques to find the optimal parameters.## References

- http://en.wikipedia.org/wiki/Curve_fitting
- http://en.wikipedia.org/wiki/Regression_analysis
- http://en.wikipedia.org/wiki/Least_squares
- Franklin A. Graybill, Hariharan K. Iyer, Regression Analysis. Concepts and Applications, Duxbury Press, Belmont, California.