A closed form solution occurs when the optimization objective function can be derived by hand and written in an explicit form consisting of a finite number of mathematical operations. Examples of closed form solutions include the quadratic formula and normal equations in least squares regression.
Problems that have a closed form solution can be optimized at a much faster speed than those that require numeric approximation. It is also advantageous for documentation and interpretation to be able to clearly derive the form of the solution. Linear regression (without regularization) has a closed form solution, and among classification algorithms, Discriminant Analysis and Naive Bayes are known for having solutions of such a form.
