Modify ELMo tutorial
@@ -37,7 +37,7 @@ To add ELMo, there are three relevant changes. First, modify the `text_field_em
"type": "elmo_token_embedder",
"options_file": "https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_options.json",
"weight_file": "https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_weights.hdf5",
- "do_layer_norm": true,
+ "do_layer_norm": false,
"dropout": 0.5
}
```
@@ -86,8 +86,6 @@ from allennlp.commands.elmo import ElmoEmbedder
ee = ElmoEmbedder()
-ee.embed_sentence("Bitcoin alone has a sixty percent share of global search.")
-
embeddings = ee.embed_sentence("Bitcoin alone has a sixty percent share of global search .".split())
# embeddings has shape (3, 11, 1024)
@@ -96,6 +94,31 @@ embeddings = ee.embed_sentence("Bitcoin alone has a sixty percent share of globa
# 1024 - the length of each ELMo vector
```
+For larger datasets, batching the sentences by using the `batch_to_embeddings` method
+will speed up the computation significantly.
+
+Also note that `ElmoEmbedder` is a utility class that bundles together several
+tasks related to computing ELMo representations including mapping strings to character ids and
+running the pre-trained biLM. It is not designed to be used when training a model and
+is not a subclass of `torch.nn.Module`. To train a model with ELMo, we recommend using
+the `allennlp.modules.elmo.Elmo` class, which does subclass `torch.nn.Module` and implements
+`forward`.
+
+## Recommended hyper-parameter settings for `Elmo` class
+
+When using ELMo, there are several hyper-parameters to set. As a general rule, we have found
+training to be relatively insensitive to the hyper-parameters, but nevertheless here are some
+general guidelines for an initial training run.
+
+* Include one layer of ELMo representations at the same location as pre-trained word representations.
+* Set `do_layer_norm=False` when constructing the `Elmo` class.
+* Add some dropout (0.5 is a good default value), either in the `Elmo` class directly, or in the next layer of your network. If the next layer of the network includes dropout then set `dropout=0` when constructing the `Elmo` class.
+* Add a small amount of L2 regularization to the scalar weighting parameters (`lambda=0.001` in the paper). These are the parameters named `scalar_mix_L.scalar_parameters.X` where `X=[0, 1, 2]` indexes the biLM layer and `L` indexes the number of ELMo representations included in the downstream model. Often performance is slightly higher for larger datasets without regularizing these parameters, but it can sometimes cause training to be unstable.
+
+Finally, we have found that including pre-trained GloVe or other word vectors in addition to ELMo
+provides little to no improvement over just using ELMo and slows down training.
+
+
## Writing contextual representations to disk
You can write ELMo representations to disk with the `elmo` command. The `elmo`
