Improve vocabulary documentation
Clarify the different ways of generating the vocabularies.
@@ -12,23 +12,31 @@ For text inputs, vocabulary files should be provided in the data configuration.
The `onmt-build-vocab` script can be used to generate vocabulary files in multiple ways:
-### from tokenized files
+### Generate a vocabulary from tokenized training files
-The script can be run directly on tokenized files. For example the command:
+If your training data is already tokenized, you can build a vocabulary with the most frequent tokens. For example, the command below extracts the 50,000 most frequent tokens from the files `train.txt.tok` and `other.txt.tok` and saves them to `vocab.txt`:
```bash
-onmt-build-vocab --size 50000 --save_vocab vocab.txt train.txt.tok
+onmt-build-vocab --save_vocab vocab.txt --size 50000 train.txt.tok other.txt.tok
```
-extracts the 50,000 most frequent tokens from `train.txt.tok` and saves them to `vocab.txt`.
+Instead of defining a fixed size, you can also prune tokens that appear below a minimum frequency. See the `--min_frequency` option.
-### from raw files
+### Generate a vocabulary from raw training files with on-the-fly tokenization
By default, `onmt-build-vocab` splits each line on spaces. It is possible to define a custom tokenization with the `--tokenizer_config` option. See [Tokenization](tokenization.md) for more information.
-### from SentencePiece training
+### Convert a SentencePiece vocabulary to OpenNMT-tf
-`onmt-build-vocab` can also prepare a [SentencePiece](https://github.com/google/sentencepiece) vocabulary and model from raw data. For example the command:
+If you trained a [SentencePiece](https://github.com/google/sentencepiece) model, a vocabulary file was generated in the process. You can convert this vocabulary to work with OpenNMT-tf:
+
+```bash
+onmt-build-vocab --from_vocab sp.vocab --from_format sentencepiece --save_vocab vocab.txt
+```
+
+### Train a SentencePiece model and vocabulary with OpenNMT-tf
+
+The `onmt-build-vocab` script can also train a new [SentencePiece](https://github.com/google/sentencepiece) vocabulary and model from raw data. For example the command:
```bash
onmt-build-vocab --sentencepiece --size 32000 --save_vocab sp train.txt.raw
@@ -75,6 +83,7 @@ data:
data:
source_1_1_vocabulary: src_1_1_vocab.txt
source_1_2_vocabulary: src_1_2_vocab.txt
+ source_2_vocabulary: src_2_vocab.txt
```
**Note:** If you train a model with shared embeddings, you should still configure all vocabulary parameters but in this case they should simply point to the same vocabulary file.
