SamTech
Table of contents

Recurrent Neural Networks (RNN)

Recurrent Neural Networks (RNN) are a type of artificial neural network designed to recognize patterns in sequences of data, such as text, genomes, handwriting, or spoken words. Unlike traditional neural networks, RNNs have loops, allowing information to be passed from one step in the sequence to the next.

Characteristics

RNNs are particularly known for their characteristic ability to “remember” previous inputs in the sequence using their hidden state, which makes them very effective for tasks that involve sequential data. Here are some defining characteristics of RNNs:

  • Sequential processing: While other types of neural networks process each input independently, RNNs process inputs in a sequential manner. This characteristic allows them to effectively handle sequence prediction problems.

  • Shared parameters across time steps: In RNNs, the weights of the recurrent hidden layers are shared across time steps. This characteristic significantly reduces the number of parameters in the model and provides the model the ability to generalize across sequences of different lengths.

Basic Concept of RNN

An RNN processes sequences by iterating through the sequence elements and maintaining a ‘state’ containing information relative to what it has seen so far. In essence, RNNs have a ‘memory’ which captures information about what has been calculated so far. The key aspect of RNNs is their hidden state, which captures some information about a sequence. The formula for an RNN is given by:

$h_t = f(W_{hh}h_{t-1} + W_{xh}x_t)$

where $h_t$ is the hidden state at time $t$, $f$ is an activation function (usually tanh or ReLU), $W_{hh}$ is the hidden-to-hidden weight matrix, $W_{xh}$ is the input-to-hidden weight matrix and $x_t$ is the input at time $t$.

Types of RNNs

Different types of RNNs have been proposed to address various issues. The major ones are:

  1. Simple RNN (SRNN): This is the most basic form of an RNN. At each time step, it computes the new hidden state using the current input and the previous hidden state. However, such simple RNNs have a problem known as “long-term dependencies,” where information from earlier in the sequence gets diluted as time progresses.

  2. Long Short-Term Memory (LSTM): LSTMs were designed to combat the long-term dependency problem of SRNNs. They introduce the concept of a cell state and gates to better retain and manage information. The update formulas for an LSTM are as follows:

    • Forget Gate: $f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$
    • Input Gate: $i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$
    • Cell State: $\tilde{C}t = \tanh(W_C \cdot [h{t-1}, x_t] + b_C)$
    • Final Cell State: $C_t = f_t * C_{t-1} + i_t * \tilde{C}_t$
    • Output Gate: $o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$
    • Hidden State: $h_t = o_t * \tanh(C_t)$

  3. Gated Recurrent Unit (GRU): GRUs are a simplified version of LSTMs that combine the cell state and hidden state into one. They also reduce the three gates of an LSTM to two, thereby reducing computational requirements. The update formulas for a GRU are as follows:

    • Update Gate: $z_t = \sigma(W_z \cdot [h_{t-1}, x_t] + b_z)$
    • Reset Gate: $r_t = \sigma(W_r \cdot [h_{t-1}, x_t] + b_r)$
    • Candidate Hidden State: $\tilde{h}t = \tanh(W \cdot [r_t * h{t-1}, x_t] + b)$
    • Hidden State: $h_t = (1 - z_t) * h_{t-1} + z_t * \tilde{h}_t$

In these equations, $\sigma$ represents the sigmoid function, $\tanh$ is the hyperbolic tangent function, and $*$ denotes element-wise multiplication.

Loss Function and Optimization

Commonly, the Cross-Entropy loss function is used for training RNNs for classification tasks, and Mean Squared Error (MSE) for regression tasks. As for optimization, Stochastic Gradient Descent (SGD) and its variations such as RMSprop and Adam are typically used. The update rule of SGD is defined as:

$$ W = W - \eta \frac{\partial L}{\partial W} $$

where $W$ represents the weights of the network, $\eta$ is the learning rate, $L$ is the loss function, and $\frac{\partial L}{\partial W}$ is the gradient of the loss function with respect to the weights.

RNNs in Real-world Applications

RNNs have a wide variety of applications. They’re used in language modeling and generation, machine translation, speech recognition, and more.

Considerations

  • Vanishing & Exploding Gradient Problem: RNNs are prone to vanishing and exploding gradients, making them hard to train for long sequences.
  • Long-Term Dependencies: RNNs have difficulties in learning to connect the information between older steps and the current step if they are too far apart, known as long-term dependencies.
  • Training Time: RNNs can be slow to train due to their recurrent nature, which prevents parallelization across time-steps.

Recurrent Neural Networks, with their ability to process sequential data and their applicability to tasks such as language modeling, machine translation, and speech recognition, are a crucial tool in the field of deep learning. While there are some challenges to be aware of, their potential makes them a powerful tool to have in your AI toolbox.