Rectifier (neural networks)

In the context of artificial neural networks, the rectifier is an activation function defined as:

$f(x)=x^{+}=\max(0,x)$ ,

where x is the input to a neuron. This is also known as a ramp function and is analogous to half-wave rectification in electrical engineering. This activation function was first introduced to a dynamical network by Hahnloser et al. in a 2000 paper in Nature with strong biological motivations and mathematical justifications. It has been used in convolutional networks more effectively than the widely used logistic sigmoid (which is inspired by probability theory; see logistic regression) and its more practical counterpart, the hyperbolic tangent. The rectifier is, as of 2015^[update], the most popular activation function for deep neural networks.

A unit employing the rectifier is also called a rectified linear unit (ReLU).