How a GRU carries information through time AI

How a recurrent update turns nine sensor values into a changing 128-value state.

A gated recurrent unit (GRU) reads a sequence while updating a hidden state. In the activity inspector, each step receives nine sensor values and the preceding 128-value state. The same parameters are reused at every step.

The update gate blends retained state with a candidate state:

$$h_t=(1-z_t)\odot n_t+z_t\odot h_{t-1}.$$

Here $z_t$ is a vector of gate values between zero and one, $n_t$ is the candidate, and $\odot$ means elementwise multiplication. The reset gate controls how the preceding state contributes to the candidate. PyTorch’s particular candidate calculation is specified in its GRU documentation.

In this pilot, the state starts at zero for every 128-sample window. The diagram’s boxes are successive uses of one GRU, not separate layers. Moving the time cursor changes which saved state is inspected; it does not retrain the network. The classifier receives only the final state. Gate values were not exported, so the display cannot establish which gate caused a particular response.

Explore the neural-network demo.