129.129. RNN Cell

RNN Cell (Recurrent Neural Network)

Implement a basic RNN (Recurrent Neural Network) cell. An RNN cell processes sequential data by maintaining a hidden state that captures information from previous time steps. At each step, the cell combines the current input with the previous hidden state to produce a new hidden state.

Formula:

h_t = tanh(W @ [x_t ; h_{t-1}])

Where:

x_t = input at current time step (shape: input_size)

h_{t-1} = hidden state from previous time step (shape: hidden_size)

[x_t ; h_{t-1}] = concatenation of input and previous hidden state

W = weight matrix (shape: hidden_size x (input_size + hidden_size))

tanh = hyperbolic tangent activation function

h_t = new hidden state (shape: hidden_size)

Example:

input = [1.0, 0.5]

prev_hidden = [0.0, 0.0]

weights = [[0.5, 0.3, 0.1, 0.2],

[0.4, 0.6, 0.3, 0.1]]

Step 1: Concatenate input and hidden state

combined = [1.0, 0.5, 0.0, 0.0]

Step 2: Matrix multiply

z = weights @ combined = [0.5*1.0 + 0.3*0.5 + 0.1*0.0 + 0.2*0.0,

0.4*1.0 + 0.6*0.5 + 0.3*0.0 + 0.1*0.0]

= [0.65, 0.70]

Step 3: Apply tanh activation

h_t = tanh([0.65, 0.70]) = [0.5717, 0.6044]

**Explanation:** The RNN cell is the fundamental building block of recurrent neural networks. Unlike feedforward networks that process each input independently, RNNs maintain a hidden state that acts as a memory, allowing the network to capture temporal dependencies in sequential data like text, time series, and speech. The tanh activation keeps the hidden state values bounded in (-1, 1), preventing values from growing unboundedly across time steps.

Constraints: