Welcome to OptiViz

An interactive tool to visualize optimization algorithms in both 1D and 2D!

Bauhaus University Weimar

What is Optimization

Optimization is the process of finding the best solution from set of all possible solutions. For example, finding the best route to deliver goods that minimizes the total travel time and/or cost, scheduling machines in a factory to maximize the production quantity. In engineering, it involves finding the optimal shape of a dam to maximize its strength against the water pressure. Most recently, adjusting the parameters of a machine learning model to achieve better accuracy.

In order to achieve above mentioned goals, It is essential to describe the problem as a mathematical function (also called objective or cost function) with all the influencing factors as inputs to the function. The mathematical modelling of the problem requires domain knowledge. Systematic methods are then used to find the input values that minimize or maximize the function value.

Lets assume we have a function \( f(x) \) that we want to minimize. The optimization process starts with an initial guess \( x = x_0 \) and iteratively update the \( x\) to \( x_i \) until certain number of iterations or the solution met the convergence criteria (more on this in optimization methods section). All the optimization methods vary in how they update the \( x \) value in each iteration.

Optimization Method Categories

Optimization algorithms can be broadly categorized into three main types based on their approach to finding the minimum or maximum of a function:

First-Order (Gradient) Methods
Uses only the first derivative of the function to find new optimal point.
  • Steepest Descent (1D/2D)
  • Conjugate Gradient (1D/2D)
Second-Order (Hessian) Methods
Uses the first and second detivates of the function to find new optimal point.
  • Newton method (1D/2D)
  • Levenberg-Marquardt Method (1D/2D)
Zeroth-Order (derivative-free) Methods
Doesn't use the derivatives of the function. Evaluates the function at various points to find the minimum.
  • Golden cut (1D)
  • Powell (2D)
  • Nelder-Mead (2D)