Bidirectional RNN and Deep RNN

Original Source: https://www.coursera.org/specializations/deep-learning

Bidirectional RNN

Let’s say that we want to do name entity recognition. Following two examples are included in the training set; ‘He said, “Teddy bears are on sale!”’, and ‘He said, “Teddy Roosevelt was a great President!”’.

If the information flows only from left to right, our model has no way to learn that “Teddy” in the second example is a name.

To solve this kind of problem, bidirectional RNN (BRNN) is introduced.

In BRNN, there are two forward propagations; left to right and right to left. For each layer we combine activations from these two forward props to output prediction.

For each sequential layer, output is computed as following;

\[\hat{y}^{<t>} = g(W_y[\overrightarrow{a}^{<t>}, \overleftarrow{a}^{<t>}] + b_y)\]

In natural language processing, BRNN with LSTM is the most standard model.

Deep RNN

We can make the RNN model more complex by adding layers vertically.