Bidirectional LSTM model state output data format

The document shows the bidirectional LSTM model output hidden state and cell of memory in the formation of [2*num_layers, batch_size, num_hidden]. I am wondering what the sequence in the first dimension.

Suppose the num_layers = 2, is the sequence of hidden state in the following way:

  1. the forward hidden state of the first layer,
  2. the forward hidden state of the second layer,
  3. the backward hidden state of the first layer
  4. the backward hidden state of the second layer?