What ReLU does to a response AI

Why some image-network responses are zero and why that differs from a GRU state.

The rectified linear unit, or ReLU, keeps positive values and replaces negative values with zero:

$$\operatorname{ReLU}(x)=\max(0,x).$$

For example, the vector $[-2,0.5,3]$ becomes $[0,0.5,3]$. This simple nonlinear operation prevents a stack of linear transformations from collapsing into just one linear transformation. The PyTorch definition specifies its elementwise behavior.

The image report records stage outputs after ReLU, so those saved responses are nonnegative. A dark cell can mean a zero or small displayed response; it does not establish that the corresponding image region is unimportant.

The GRU’s hidden states instead combine sigmoid gates and a hyperbolic-tangent candidate. They can be negative or positive, and the activity report uses a blue–white–red scale. The two displays should not be interpreted as if they use the same activation function.

Explore the neural-network demo.