An inner product space is a vector space equipped with a positive definite symmetric bilinear form, and over the real numbers the standard example is the Euclidean dot product. Positive definiteness is the whole content of the geometry, because it makes orthogonality meaningful, forces a unique best approximation in a subspace, and supplies a coordinate free definition of length, angle, and projection. Two vectors are orthogonal when their inner product vanishes, and a set of pairwise orthogonal nonzero vectors is automatically linearly independent, since a nontrivial combination with coefficients whose squared magnitudes sum to zero is impossible. The dimension bound then shows that an orthogonal set can be extended to an orthonormal basis, and in finite dimensions every inner product space admits one, which means every inner product space of the same dimension is isometric to Euclidean space. That conclusion is reassuring but often less useful than it appears, since the decomposition of a vector into orthogonal components depends on the inner product, and a different geometry produces a different notion of nearest neighbor. Orthogonal projection follows directly. If a subspace is spanned by an orthonormal set, the coefficients of the projection are just the inner products with that set, and the residual is automatically orthogonal to the subspace. The geometric meaning is that the error is minimized in length, so the distance from a point to a subspace is the length of its orthogonal residual. This gives the least squares solution in a single stroke. Minimizing the sum of squared residuals is exactly the problem of finding the point in a column space closest to the observed data, and when the data vectors are not linearly independent the minimum need not be unique, which is where the pseudoinverse enters. The normal equations express the first order conditions as a linear system in the unknowns, and the system is singular precisely when the design matrix fails to have full column rank. Solving the normal equations directly is tempting and inadvisable, because forming the product of the design matrix with its transpose squares the condition number and can destroy accuracy even when the original problem is well behaved. The numerically sound alternatives are the orthogonal factorization, whose triangular factor can be used directly, and the singular value decomposition, which reveals rank deficiency, exposes the effective dimension, and provides the pseudoinverse solution with a natural minimum norm interpretation. Applications are everywhere. In surveying, photogrammetry, and computer vision, one estimates a camera pose or a rigid transformation by minimizing reprojection error, which is a least squares problem in disguise. In signal processing, matched filtering and linear prediction are solved by orthogonality conditions. In statistics, the normal equations define generalized least squares, and the trace of the inverse covariance times the residual outer product estimates the residual variance. Regularization adds a penalty proportional to a matrix or the identity, which pulls the solution away from directions of large variance and stabilizes an otherwise ill posed problem, and ridge regression and the Tikhonov framework express exactly that idea. Numerical stability deserves emphasis: the loss of orthogonality in modified Gram Schmidt grows with the number of steps, while Householder reflections produce orthogonal bases with excellent backward stability, which is why production codes prefer Householder. The concept carries a striking generalization to Hilbert spaces, where the space of square summable sequences supports Fourier series, and Parseval identity expresses that orthogonality of the trigonometric basis is what makes coefficient computation efficient. Anyone who has internalized that best approximation means orthogonal residual will find a surprising number of otherwise mysterious results to be routine. The remaining members of the family deserve mention because they are simple and useful. The angle between two vectors is defined through the inner product and the product of their norms, and Cauchy Schwarz asserts that this cosine like quantity never exceeds one in magnitude, an inequality that underpins the geometric series manipulations, the convergence of the inner product series in Hilbert space, and the derivation of the adjoint of a linear operator. The orthogonal complement of a subspace is a subspace, the projection onto it exists and is unique, and the decomposition into a component in the subspace plus a component in its orthogonal complement is the finite dimensional form of a decomposition that survives in infinite dimensions. Bessel inequality says that the squared lengths of the projections of a vector onto an orthonormal set are bounded by the squared length of the vector, and it is the mechanism behind Parseval identity and behind the convergence of Fourier and wavelet expansions. The Riesz representation theorem states that a continuous linear functional on a finite dimensional inner product space is given by inner product with a unique vector, which is a deceptively deep statement: it means that once an inner product is fixed, every linear quantity of interest can be realized as a projection. Weighted inner products are worth mentioning because they immediately generalize the projection formulas. Giving observations different weights produces weighted least squares, and choosing weights as the inverse of a covariance matrix produces generalized least squares, which is the correct treatment when errors are correlated, and the choice of weights as the inverse covariance again gives the Mahalanobis distance, the natural notion of distance in a space where units differ across coordinates. This is the same idea that the kernel trick exploits, since a positive definite kernel implicitly defines a richer inner product on a space of functions, and support vector machines are then simply a hard margin optimization in that implicit geometry, with orthogonality conditions arising as Lagrange multipliers. Total least squares makes the geometry symmetric by correcting both sides of a model equation rather than only the response. Regularization can likewise be read as changing the geometry, since a penalty in the coefficients amounts to shrinking the space of candidate solutions, and the resulting estimator is no longer a pure projection but a compromise between data fidelity and prior structure. Once that view is available, ridge regression, early stopping, and Bayesian estimators with Gaussian priors all look like the same idea wearing different clothes, and the beginner who sees this will find the later theory of function spaces much less forbidding.
